Deep Reinforcement Learning-Based Battery Management Algorithm for Retired Electric Vehicle Batteries with a Heterogeneous State of Health in BESSs

Abstract

In this paper, we propose a battery management algorithm to optimize the lifetimes of retired lithium batteries with heterogeneous states of health in a battery energy storage system under dynamic power demand. A battery energy storage system allows for the use of retired lithium batteries for applications such as backup power in homes, data centers, etc. In a battery energy storage system, a battery pack consists of several retired batteries connected in parallel or in series to fulfill the required power demand. Owing to the retired batteries’ different capacity levels, i.e., states of health, a scheduling strategy is required to turn battery cells inside the battery pack on and off such that the secondary lifetimes of the retired batteries are extended. To establish the optimal scheduling policy, it is necessary to determine the correct states of each battery cell, including the state of charge and the state of health. To that end, the proposed algorithm first estimates the state of charge and state of health for all cells based on data measured using an extended Kalman filter. Then, a deep reinforcement learning scheduling algorithm is implemented to connect/disconnect the battery cells to/from the battery pack based on their states. Via simulation, we show that the proposed algorithm estimates the state of charge and state of health of each battery cell with low error and extends the lifetime of battery packs by 20.6%, compared to methods proposed in previous works.

1. Introduction

Lithium-ion batteries have become an essential component of modern life, powering everything from smartphones to electric vehicles (EVs) [1]. Their advantages include high energy efficiency, minimal memory effects, extended lifespans, and low self-discharge rates compared to other battery types, and they are now widely used [2]. However, lithium-ion batteries for EVs have a limited lifespan and (eventually, for safety) require replacement when their capacity drops to 80% or lower [3]. The total amount of retired battery-pack power is forecast to reach 120 GWh globally by 2030 [4]. This will create a significant amount of waste and financial burden, particularly as the demand for lithium-ion batteries continues to grow. As a result, it is becoming increasingly urgent and necessary to identify solutions to reuse retired lithium-ion batteries.

One promising application for retired lithium-ion batteries is in battery energy storage systems (BESSs) that can then be used for backup power in homes, EV charging stations, or telecommunication and data center systems [5]. BESSs have the potential to significantly reduce the demand for new batteries and can help reduce the environmental impact of battery production. A battery energy storage system (BESS) has a battery pack in which multiple batteries are connected in parallel or in series to increase the capacity or voltage of the battery pack. A switch is added to each battery cell to connect it to or disconnect it from the battery pack [6]. Cells in a battery pack have different capacity levels (i.e., heterogeneous states of health), which hinders the effective utilization of the batteries and, consequently, affects the performance of the battery pack. A scheduling policy is required to control the switches in battery cells to prolong the lifetime of the battery pack in the BESS and reduce the imbalanced capacities of battery cells.

For an optimal scheduling policy in a BESS, correct identification of battery characteristics, including the state of charge (SOC) and state of health (SOH), is important. The SOC of a battery is the level of charge relative to the battery’s capacity, whereas the SOH is the ratio of the maximum battery charge to its rated capacity. The relationships between SOC and SOH are illustrated in Figure 1. Information on SOC and SOH in a scheduling policy protects the battery cells by preventing them from overcharging or discharging excessively and increases the capacity of the BESS. SOC and SOH parameters cannot be measured directly from a battery cell. Instead, they are estimated through measurable parameters such as voltage, current, and cell temperature. Based on the states of the battery cells in a pack (including the SOC and SOH), the ON/OFF switches for the batteries in the pack are scheduled so that the states of health of all batteries are balanced. As a result, the lifetime of the battery pack is extended. Therefore, the correct estimation of SOC and SOH, along with the scheduling of battery cell switches, is necessary to optimize the performance of a BESS.

Battery state estimation approaches have been explored in the literature. The Coulomb counting method [7,8] calculates the SOC of cells by counting the amount of charge that enters or exits a cell. However, the Coulomb counting method was unable to measure the SOC of cells in an online parallel-connected battery pack following a sharp drop in SOH. An algorithm was proposed for the online estimation of SOC using deep learning [9], but this algorithm ignores the estimation of SOH. A neural network was used to estimate SOH using experimental datasets [10]. However, the authors did not consider SOC estimation. Information on both SOCs and SOHs of battery cells is required for efficient scheduling of a battery pack to optimize BESS performance. The authors of [11] proposed joint lithium battery SOC and SOH estimation using a data-driven method. This approach required a large dataset to train the model before operation on site. Kalman filter-based approaches can estimate SOC and SOH levels [12] but depend on the correctness of electrochemical impedance spectroscopy (EIS) parameters, including a resistor and one or more resistor–capacitor (RC) pairs. However, the effect of SOH reduction on EIS parameters is ignored in Kalman filter approaches [13]. In [14], SOH reduction was considered to reidentify EIS parameters, but the SOH was updated only offline. Several researchers have studied the problem of cell scheduling in a parallel-connected battery pack. The authors of [15] utilized a fuzzy logic control strategy to adjust the number of cells in a circuit in accordance with the load demand for the purpose of reducing loop current, which leads to battery inconsistency. In [16], battery resistance degradation was monitored to detect weak cells and disconnect them from the battery pack. This approach solved the issue of mismatched characteristics but requires a complex measuring system or incurs a high computational burden. In [17], the weighted-k round-robin (kRR) scheduling framework was proposed to extend the lifetime of a battery pack by considering load demand and SOH reduction. However, kRR-based scheduling can be implemented only for a fixed model, i.e., the number of cells in the battery pack or the battery models inside the pack cannot change. In [6], a multiactor–critic method was proposed to solve battery scheduling problems. This approach prolonged the lifetime of the battery pack and reduced the imbalance between the batteries but ignored dynamic power demand. In [18], a  strategy for a battery management system was proposed, including SOC estimation using an extended Kalman filter algorithm and a scheduler to reduce the difference between the SOCs of battery cells. However, SOH and power demand were not considered in that approach. The main challenge is to determine the accurate state (i.e., SOC and SOH) of a battery cell in a battery pack, then schedule the turning ON/OFF of battery cells based on their current states such that the imbalance in SOHs of cells is reduced.

The main contributions of our work are as follows:

• A scheduling algorithm is proposed to maximize the lifetime of a battery pack consisting of parallel-connected battery cells with heterogeneous states of health in a BESS.
• We define the battery lifetime maximization problem as the reduction in the SOH of a battery pack that can be achieved by reducing the imbalance in the SOHs of battery cells in a battery pack.
• A deep reinforcement learning (DRL) framework is implemented in the scheduling algorithm that uses battery cells’ states to set their ON/OFF status and balance the SOHs.
• To measure the battery cells’ states to schedule their ON/OFF status, an extended Kalman filter (EKF)-based algorithm is proposed to estimate SOC and SOH.
• A dataset of real measurements is used to determine the accuracy of the proposed estimation algorithm. The proposed algorithm achieves minimal error compared to methods proposed in other works. Simulation results show that the proposed algorithm outperforms previous studies by extending the lifetime of a battery pack under constant and dynamic power demands.

The remainder of this paper is organized as follows. Section 2 discusses the proposed parallel-connected battery model and the scheduling issues. Section 3 presents the framework of the proposed combined algorithm, which includes EKF-based and DRL-based algorithms. Section 4 describes the simulation and presents the results and impacts of the algorithm. Finally, we conclude this work in Section 5.

For ease of presentation, the key notations listed in

Table 1. Summary of notations.

Notation	Definition
$\mathcal{W}$	Operational time
$\mathcal{V}\left(t_k\right)$	Set of measured cell voltages at time $t_k$
$\mathcal{I}\left(t_k\right)$	Set of measured cell currents at time $t_k$
$\mathcal{T}\left(t_k\right)$	Set of measured cell temperatures at time $t_k$
$\mathcal{C}\left(t_k\right)$	Set of cells’ SOC values at time $t_k$
$\mathcal{H}\left(t_k\right)$	Set of cells’ SOH values at time $t_k$
$SO{H}_{\mathcal{P}}\left(t_k\right)$	SOH of the battery pack at time $t_k$
$V_i\left(t_k\right)$	Measured terminal voltage of cell i at time $t_k$
$I_i\left(t_k\right)$	Measured current of cell i at time $t_k$
$T_i\left(t_k\right)$	Measured cell temperature of cell i at time $t_k$
${\mathcal{X}}_i\left(t_k\right)$	ON/OFF switch of cell i
$V_{Oi}$	Open-circuit voltage of cell i
$R_{si}$	Internal resistance of cell i
$R_{pi}$, $C_{pi}$	Resistor–capacitor pair of cell i
$l_D\left(t_k^j\right)$	Discharging power load in cycle j up to time $t_k$
$l_C\left(t_k^j\right)$	Charging power load in cycle j up to time $t_k$
$\eta$	Efficiencies of the discharging/charging process

are used throughout this paper.

2. System Model

2.1. Overall System

In this paper, we consider a parallel-connected BESS [19,20] with a power supply and a load, as shown in Figure 2. The BESS comprises a battery pack and a battery management system (BMS) connected to a power supply and a load. We consider a discrete-time model, where the working time ($\mathcal{W}$) is divided into w time slots such that $\mathcal{W}=\{t_k\mid k=1,2,...,w\}$ with durations of $\mathsf{\Delta }t=t_k-t_{k-1}$.

The battery pack consists of N lithium battery cells connected in parallel. A first-order Thévenin equivalent model is considered for the cells [21]. Cell $i\in \mathcal{N}=\{1,2,...,N\}$ has EIS parameters including an open-circuit voltage ($V_{Oi}$); internal resistance ($R_{si}$); and an RC pair, which includes a resistor ($R_{pi}$) and capacitor ($C_{pi}$) connected in parallel. The terminal voltage of cell i at time t ($V_i\left(t\right)$) is computed as

$$V_i\left(t_k\right)=V_{Oi}\left(t_k\right)-V_{pi}\left(t_k\right)-R_{si}\left(t_k\right)I_i\left(t_k\right),$$

(1)

where $V_{pi}\left(t_k\right)$ is the polarization voltage applied to the parallel RC network, calculated as [14]

$$V_{pi}\left(t_k\right)=e^{-\frac{\mathsf{\Delta }t}{R_{pi}\left(t_{k-1}\right)C_{pi}\left(t_{k-1}\right)}}{V}_{pi}\left(t_{k-1}\right)+R_{pi}\left(t_{k-1}\right)(1-e^{-\frac{\mathsf{\Delta }t}{R_{pi}\left(t_{k-1}\right)C_{pi}\left(t_{k-1}\right)}})I_i\left(t_{k-1}\right).$$

(2)

There are N switches corresponding to N cells linking them to the battery circuit. ${\mathcal{X}}_i\left(t_k\right)$ shows whether a switch of cell i is connected to a battery circuit or not, such that

$$\begin{array}{c}\hfill {\mathcal{X}}_i\left(t_k\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{cell}i\mathrm{is}\mathrm{ON}\hfill \\ 0,\hfill & \mathrm{if}\mathrm{cell}i\mathrm{is}\mathrm{OFF}\hfill \end{array}\right.\end{array}$$

(3)

Similarly, sets $\mathcal{V}\left(t_k\right)$, $\mathcal{I}\left(t_k\right)$, and $\mathcal{T}\left(t_k\right)$ consist of terminal voltages, currents, and temperatures of all cells at time $t_k$, respectively.

A BMS monitors the states of the battery pack and estimates both the SOC and the SOH of cells in order to schedule the switches in the battery pack. We mathematically define the SOC of cell i at time $t_k$ as

$$\begin{array}{c}\hfill SO{C}_i\left(t_k\right)=SO{C}_i\left(t_{k-1}\right)-\frac{\eta \mathsf{\Delta }t{I}_i\left(t_{k-1}\right)}{M_i\left(t_k\right)},\end{array}$$

(4)

where $M_i\left(t_k\right)$ is the capacity level of cell i at time $\left(t_k\right)$, and $\eta$ is the Coulombic efficiencies of the discharging or charging process. Similarly, the SOH of cell i at time $t_k$ is defined as

$$\begin{array}{c}\hfill SO{H}_i\left(t_k\right)=\frac{M_i\left(t_k\right)}{M_{new}},\end{array}$$

(5)

where $M_{new}$ is the initial capacity of new cell i. Sets $\mathcal{C}\left(t_k\right)$ and $\mathcal{H}\left(t_k\right)$ consist of the SOCs and SOHs of all the cells at time $t_k$, respectively. We define the SOH of the battery pack ($SO{H}_{\mathcal{P}}\left(t_k\right)$) as 

$$\begin{array}{c}\hfill SO{H}_{\mathcal{P}}\left(t_k\right)=\underset{i\in \mathcal{N}}{min}\left(\mathcal{H}\left(t_k\right)\right)\end{array}$$

(6)

Power supply and load are used for charging and discharging of the battery pack. The battery pack current at time $t_k$ ($I_{\mathcal{P}}\left(t_k\right)$) has a positive value when discharging and a negative value when charging. The battery pack fulfills the load demand when discharging, then recharges to recover the corresponding amount of power. The process of complete charging and discharging of a battery pack is referred to as a cycle. During the working time ($\mathcal{W}$), an arbitrary cycle (j) has multiple time slots based on the power demand. If time slot $t_k$ belongs to cycle j, we consider $l_D\left(t_k^j\right)$ and $l_C\left(t_k^j\right)$ to be the amount of power load when discharging and charging in cycle j, respectively, up to time slot $t_k$, which are calculated as

$$\begin{array}{c}\hfill l_D\left(t_k^j\right)=\sum _{\tau =\varkappa }^k\sum _{\forall i\in \mathcal{N}}\eta V_i\left(t_{\tau }\right)I_i\left(t_{\tau }\right)\mathsf{\Delta }t,\end{array}$$

(7)

and

$$\begin{array}{c}\hfill l_C\left(t_k^j\right)=\sum _{\tau =\varkappa }^k\sum _{\forall i\in \mathcal{N}}\eta V_i\left(t_{\tau }\right)I_i\left(t_{\tau }\right)\mathsf{\Delta }t.\end{array}$$

(8)

where $\varkappa$ represents the slot number when cycle j starts, i.e., cycle j starts at time $t_{\varkappa }$.

2.2. Problem Formulation

The objective of this paper is to prolong the lifetime of a battery pack by reducing the rate of aging in cells. To that end, the problem is formulated to minimize the SOH reduction of the battery pack during working time ($\mathcal{W}$), which is mathematically expressed as

$$\min \hspace{1em}\sum _{k=1}^w\mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$$

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.\hfill & \mathsf{\Delta }SOH\mathcal{P}\left(t_k\right)\ge 0,\hfill \\ & I_{min}^{-}\le I_i\left(t_k\right)\le I_{max}^{+},\hfill \\ & SO{C}_{min}\le SO{C}_i\left(t_k\right)\le SO{C}_{max},\hfill \\ & l_D\left(t_k^j\right)\ge d\left(t_k^j\right),\hfill \\ & l_C\left(t_k^j\right)\ge d\left(t_k^j\right),\hfill \end{array}$$

(9)

where $\mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)$ represents the SOH reduction of the battery pack at time slot $t_k$; $I_{max}^{+}$ and $I_{min}^{-}$ represent the discharge current and charge current thresholds, respectively; $SO{C}_{min}$ and $SO{C}_{max}$ indicate the lower and upper bounds of the SOC, respectively, which are required to prevent excessive discharging and charging; $l_D\left(t_k^j\right)$ and $l_C\left(t_k^j\right)$ represent the power load in cycle j up until time slot $t_k$ when discharging and charging, respectively; and $d\left(t_k^j\right)$ indicates the power demand at time slot $t_k$ in cycle j. The SOH reduction of the battery pack at time $t_k$ ($\mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)$) is defined as

$$\begin{array}{c}\hfill \mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)=SO{H}_{\mathcal{P}}\left(t_{k-1}\right)-SO{H}_{\mathcal{P}}\left(t_k\right),\end{array}$$

(10)

where $SO{H}_{\mathcal{P}}\left(t_{k-1}\right)$ and $SO{H}_{\mathcal{P}}\left(t_k\right)$ denote the SOH of the battery pack at time slots $t_{k-1}$ and $t_k$, respectively. Since $\mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)$ is a non-increasing function, we constrain it with

$$\begin{array}{c}\hfill \mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)\ge 0.\end{array}$$

(11)

3. The Proposed Algorithm

To tackle the optimization problem (9), we propose a battery-scheduling algorithm that is run by the BMS. In each time slot, the algorithm first collects measurement data that include the terminal voltage, current, and temperature of each cell, then estimates the SOC and the SOH (Algorithm 1) and controls the charging or discharging process of the BESS based on the load demand (Algorithm 2). Algorithms 1 and 2 return a state vector consisting of a set of SOC values of cells ($\mathcal{C}\left(t_k\right)$) a set of SOH values of cells ($\mathcal{H}\left(t_k\right)$), as well as the battery pack current ($I_{\mathcal{P}}\left(t_k\right)$) and power demand ($d\left(t_k\right)$), triggering the DRL-based battery-scheduling algorithm (Algorithm 3). The overall flow of the proposed algorithm is shown in Figure 3. Each part of the proposed algorithm is discussed in detail in the subsections below.

Algorithm 1 EKF-based SOC and SOH estimation

1: Input: Measurement data $\mathcal{V}\left(t_k\right)$, $\mathcal{I}\left(t_k\right)$,$\mathcal{T}\left(t_k\right)$; Data tables 2: Output: $\mathcal{C}\left(t_k\right)$,$\mathcal{H}\left(t_k\right)$ 3: Estimate state vector $\widehat{x_i}\left(t_k\right)$ and error covariance $\widehat{P_i}\left(t_k\right)$ using (12) and (13) 4: Estimate terminal voltage $\widehat{V_i}\left(t_k\right)$ and compute Kalman gain $G_i\left(t_k\right)$ using (17) and (20) 5: Update $x_i\left(t_k\right)$ and $P_i\left(t_k\right)$ using (21) and (22) 6: Update $SO{C}_i\left(t_k\right)$ and $M_i\left(t_k\right)$ 7: if cycle is completed then 8:     Update $SO{H}_i\left(t_k\right)$ using (23) 9: else 10:     $SO{H}_i\left(t_k\right)\leftarrow SO{H}_i\left(t_{k-1}\right)$ 11: end if

Algorithm 2 The Charge/Discharge Control Algorithm

1: Input: $I_{\mathcal{P}}\left(t_k\right)$, $l_D\left(t_k^j\right)$, $l_C\left(t_k^j\right)$, $d\left(t_k^j\right)$ 2: Output: Discharge or Charge 3: if $I_{\mathcal{P}}\left(t_k\right)>0\mathrm{and}{t}_k\in \mathrm{cycle}j$ then    ▹ Discharging 4:     if $l_D\left(t_k^j\right)\ge d\left(t_k^j\right)$ then 5:         Convert discharge to charge 6:     else 7:         Continue to discharge 8:     end if 9: else if $I_{\mathcal{P}}\left(t_k\right)<0\mathrm{and}{t}_k\in \mathrm{cycle}j$ then   ▹ Charging 10:     if $l_C\left(t_k^j\right)\ge d\left(t_k^j\right)$ then 11:         Convert charge to discharge       ▹ cycle $j+1$ 12:     else 13:         Continue to charge 14:     end if 15: end if

Algorithm 3 The Deep Q Network Switches Scheduling Algorithm

1: Input: state vector $s\left(t_k\right)$ 2: Output: Optimal schedule action $\mathcal{X}\left(t_k\right)$ 3: Initialize Replay experience $\mathcal{E}$ with capacity M 4: Add $\langle s\left(t_{k-1}\right),\mathcal{X}\left(t_{k-1}\right),r\left(t_{k-1}\right),s\left(t_k\right)\rangle$ into $\mathcal{E}$ 5: Construct main network $\mathcal{Q}$ and target network $\overline{\mathcal{Q}}$ 6: Initialize $\mathcal{Q}$ and $\overline{\mathcal{Q}}$ with random weights 7: Perform a gradient descent to minimize loss function $\mathcal{L}\left(\varphi \left(t_k\right)\right)$ 8: if  $s\left(t\right)\in \mathcal{K}$  then 9:      Select action $\mathcal{X}\left(t_k\right)$ using (30)      ▹ Switch ON/OFF 10: else 11:     Select action $\mathcal{X}\left(t_k\right)$ randomly 12: end if 13: Compute immediate reward $R\left(s\left(t_k\right),\mathcal{X}\left(t_k\right)\right)$ using (31) 14: Compute cumulative reward $r\left(t_k\right)$ using (32)

3.1. EKF-Based SOC and SOH Estimation

The algorithm estimates the SOC and SOH of each cell in the battery pack to observe the states of the battery cells using a third-order EKF. To obtain the SOC and SOH of battery cell i at $t_k$, the algorithm first estimates state vector $\widehat{x_i}\left(t_k\right)$ and error covariance $\widehat{P_i}\left(t_k\right)$ as

$$\begin{array}{c}\hfill \widehat{x_i}\left(t_k\right)=A_i\left(t_{k-1}\right)x_i\left(t_{k-1}\right)+B_i\left(t_{k-1}\right)I_i\left(t_{k-1}\right),\end{array}$$

(12)

and

$$\begin{array}{c}\hfill \widehat{P_i}\left(t_k\right)=A_i\left(t_{k-1}\right)P_i\left(t_{k-1}\right)A_i{\left(t_{k-1}\right)}^T,\end{array}$$

(13)

where $x_i\left(t_{k-1}\right)$ is the state vector of cell i at time ${}_{k-1}$, which is defined as

$$\begin{array}{c}\hfill x_i\left(t_{k-1}\right)={[SO{C}_i\left(t_{k-1}\right),V_{pi}\left(t_{k-1}\right),1/M_i\left(t_{k-1}\right)]}^T,\end{array}$$

(14)

and $A_i\left(t_{k-1}\right)$ and $B_i\left(t_{k-1}\right)$ denote the transition matrix and the input matrix, respectively, which are defined as follows

$$A_i\left(t_{k-1}\right)=\left[\begin{array}{ccc}1& 0& -\eta \mathsf{\Delta }t{I}_i\left(t_{k-1}\right)\\ 0& e^{-\frac{\mathsf{\Delta }t}{R_{pi}\left(t_{k-1}\right)C_{pi}\left(t_{k-1}\right)}}& 0\\ 0& 0& 1\end{array}\right],$$

(15)

$$B_i\left(t_{k-1}\right)=\left[\begin{array}{c}0\\ R_{pi}\left(t_{k-1}\right)(1-e^{-\frac{\mathsf{\Delta }t}{R_{pi}\left(t_{k-1}\right)C_{pi}\left(t_{k-1}\right)}})\\ 0\end{array}\right],$$

(16)

where $I_i\left(t_{k-1}\right)$ is the measured current of cell i at $t_{k-1}$. $R_{si}\left(t_{k-1}\right)$, $R_{pi}\left(t_{k-1}\right)$, and $C_{pi}\left(t_{k-1}\right)$ are functions of $SO{C}_i$, $SO{H}_i$, and $T_i$, respectively, which are obtained from two-dimensional look-up tables (A dataset [22] is used to construct look-up tables where $R_{si}\left(t_{k-1}\right)$, $R_{pi}\left(t_{k-1}\right)$, and $C_{pi}\left(t_{k-1}\right)$ are exponential functions of $SO{C}_i\left(t_{k-1}\right)$, such as $\left(x_{1}exp\left(x_{2}SO{C}_i\left(t_{k-1}\right)\right)+x_3\right)$, and $x_1$, $x_2$, and $x_3$ are real numbers. These real numbers change when $SO{H}_i$ decreases). Then, the algorithm estimates the terminal voltage ($\widehat{V_i}\left(t_k\right)$), using $\widehat{x_i}\left(t_k\right)$ and Jacobian matrices $C_i\left(t_k\right)$ and $D_i\left(t_k\right)$ as

$$\begin{array}{c}\hfill \widehat{V_i}\left(t_k\right)=C_i\left(t_k\right)\widehat{x_i}\left(t_k\right)+D_i\left(t_k\right)I_i\left(t_k\right)\end{array}$$

(17)

$$C_i\left(t_k\right)=\left[\begin{array}{ccc}\frac{\delta V_{Oi}\left(t_k\right)}{\delta SO{C}_i\left(t_k\right)}& -1& 0\end{array}\right]$$

(18)

$$\begin{array}{c}\hfill D_i\left(t_k\right)=-R_{si}\left(t_k\right),\end{array}$$

(19)

where the open-circuit voltage ($V_{Oi}\left(t_k\right)$) is identified by exploiting the look-up tables ($V_{Oi}\left(t_k\right)$ is the ath-order polynomial function of $SO{C}_i\left(t_k\right)$, which is defined as $({\sum }_{b=0}^a{y}_b$ ${\left(SO{C}_i\left(t_k\right)\right)}^b)$, where $y_b$ is a real number that changes when $SO{H}_i$ decreases). The algorithm calculates the Kalman gain ($G_i\left(t_k\right)$) to determine the error between the real, measured value and the estimated value using (13) as

$$\begin{array}{c}\hfill G_i\left(t_k\right)=\widehat{P_i}\left(t_k\right)C_i{\left(t_k\right)}^T{\left(C_i\left(t_k\right)\widehat{P_i}\left(t_k\right)C_i{\left(t_k\right)}^T\right)}^{-1}.\end{array}$$

(20)

Based on the estimated terminal voltage ($\widehat{V_i}\left(t_k\right)$), estimated state vector ($\widehat{x_i}\left(t_k\right)$), Kalman gain ($G_i\left(t_k\right)$), and measured terminal voltage ($V_i\left(t_k\right)$), the algorithm obtains the correct state vector ($x_i\left(t_k\right)$) as

$$\begin{array}{c}\hfill x_i\left(t_k\right)=\widehat{x_i}\left(t_k\right)+G_i\left(t_k\right)\left(V_i\left(t_k\right)-\widehat{V_i}\left(t_k\right)\right).\end{array}$$

(21)

Similarly, the algorithm corrects error covariance ($P_i\left(t_k\right)$) as

$$\begin{array}{c}\hfill P_i\left(t_k\right)=\left(1-G_i\left(t_k\right)C_i\left(t_k\right)\right)\widehat{P_i}\left(t_k\right).\end{array}$$

(22)

From corrected state vector ($x_i\left(t_k\right)$), the proposed algorithm obtains $SO{C}_i\left(t_k\right)$ and $M_i\left(t_k\right)$. The algorithm updates the SOH of cell i after one cycle (complete charging and discharging of the battery pack), since the SOH does not decrease after one or several time slots [23]. The algorithm updates the SOH of cell i at time slot $t_k$ as

$$\begin{array}{c}\hfill SO{H}_i\left(t_k\right)=\left\{\begin{array}{cc}\frac{{\stackrel{—}{M}}_i^j\left(t_k\right)}{M_{new}}\hfill & \mathrm{if}\mathrm{cycle}j\mathrm{is}\mathrm{completed};\hfill \\ SO{H}_i\left(t_{k-1}\right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(23)

where ${\stackrel{—}{M}}_i^j\left(t_k\right)$ is the effective current capacity (on average) of cell i in cycle j, which has $(k-\varkappa +1)$ time slots if cycle j is completed at time slot $t_k$. The effective current capacity (on average) of cell i is calculated as

$$\begin{array}{c}\hfill {\stackrel{—}{M}}_i^j\left(t_k\right)=\frac{{\sum }_{\tau =\varkappa }^k{M}_i\left(t_{\tau }\right)}{k-\varkappa +1},\end{array}$$

(24)

where cycle j starts at $t_{\varkappa }$ and ends at $t_k$. Algorithm 1 summarizes the EKF-based estimation for the SOH and SOC of cells.

3.2. The Charge/Discharge Control Algorithm

To control the process of charging and discharging the battery pack in the BESS, the algorithm first identifies the process that is underway. If the current of the battery pack is positive, i.e., $I_{\mathcal{P}}\left(t_k\right)>0$, we calculate the amount of electric power discharged in cycle j $l_D\left(t_k^j\right)$ using (7). If $l_D\left(t_k^j\right)$ reaches electrical demand ($d\left(t_k^j\right)$), the BMS converts the BESS process from discharging to charging. Otherwise, the discharge process continues.

If $I_{\mathcal{P}}\left(t_k\right)$ is negative, the algorithm determines $l_C\left(t_k^j\right)$ (the amount of electrical power charged in cycle j) using (8) and compares it with electrical demand ($d\left(t_k^j\right)$). If $l_C\left(t_k^j\right)$ reaches $d\left(t_k^j\right)$, the algorithm converts the BESS process from charging to discharging for a new cycle $(j+1)$; otherwise, it continues charging. The process of charging and discharging the battery pack is summarized in Algorithm 2.

3.3. Deep Reinforcement Learning-Based Scheduling Algorithm

A deep Q network (DQN) scheduling algorithm is proposed for the ON/OFF cell switches in the battery pack. The scheduling algorithm has three elements: state $s\left(t_k\right)$, which represents the current state of the BESS; action $\mathcal{X}\left(t_k\right)$, which indicates cell switches that are ON or OFF; and reward function $r\left(t_k\right)$ based on action $\mathcal{X}\left(t_k\right)$. The algorithm selects action $\mathcal{X}\left(t_k\right)$ by interacting with the environment, i.e., the BESS, to perceive the state of the battery pack ($s\left(t_k\right)$) to maximize the cumulative reward ($r\left(t_k\right)$), i.e., to minimize SOH reduction of the battery pack. To choose an optimal schedule as $\mathcal{X}\left(t_k\right)$ for state $s\left(t_k\right)$, the algorithm utilizes and updates acquired knowledge ($\mathcal{K}$) using deep reinforcement learning. That knowledge includes a switch-scheduling policy for the given battery states and the corresponding scheduling of rewards. The DQN-based scheduling algorithm is summarized in Algorithm 3.

The algorithm first observes the current environmental state of the battery pack and obtains state vector $s\left(t_k\right)$, which is defined as

$$\begin{array}{c}\hfill s\left(t_k\right)=\left[\mathcal{C}\left(t_k\right),\mathcal{H}\left(t_k\right),I_{\mathcal{P}}\left(t_k\right),d\left(t_k\right)\right],\end{array}$$

(25)

where $\mathcal{C}\left(t_k\right)$ and $\mathcal{H}\left(t_k\right)$ are sets of the SOCs and SOHs of N cells, respectively; $I_{\mathcal{P}}\left(t_k\right)$ is the load current of the battery pack; and $d\left(t_k\right)$ is the load demand. Then, the algorithm initializes knowledge ($\mathcal{K}$) that includes replay experience ($\mathcal{E}$) with samples $\langle s\left(t_{k-1}\right),\mathcal{X}\left(t_{k-1}\right),$$r\left(t_{k-1}\right),s\left(t_k\right)\rangle$, a main network ($\mathcal{Q}$), and a target network ($\overline{\mathcal{Q}}$) with random weights. Neural networks $\mathcal{Q}$ and $\overline{\mathcal{Q}}$ have the same structure. The algorithm explores actions based on past experiences to update the acquired knowledge that leads to a long-term benefit. The DQN updates acquired knowledge ($\mathcal{K}$) by minimizing loss function $\mathcal{L}\left(\varphi \left(t_k\right)\right)$ using gradient descent. The loss function is defined as

$$\begin{array}{c}\hfill \mathcal{L}\left(\varphi \left(t_k\right)\right)\leftarrow \mathbb{E}\left[{\left(\overline{Q}\left(t_{k-1}\right)-Q\left(t_{k-1}\right)\right)}^2\right],\end{array}$$

(26)

which $\varphi \left(t_k\right)$ is the DQN network parameter (weight of the main network) and is calculated as

$$\begin{array}{c}\hfill \varphi \left(t_k\right)=\varphi \left(t_{k-1}\right)+\alpha \nabla \mathcal{L}\left(\varphi \left(t_{k-1}\right)\right)\end{array}$$

(27)

where $\alpha \in (0,1]$ is the learning factor. $Q\left(t_{k-1}\right)$ shows the expected discounted cumulative reward after time slot $t_{k-1}$ in main network $\mathcal{Q}$, and $\overline{Q}\left(t_{k-1}\right)$ is the target action value of the target network ($\overline{\mathcal{Q}}$), which represents the maximum cumulative reward, i.e., the minimum SOH reduction for the battery pack. $Q\left(t_{k-1}\right)$ and $\overline{Q}\left(t_{k-1}\right)$ are calculated as

$$\begin{array}{c}Q\left(t_{k-1}\right)=Q\left(s\left(t_{k-1}\right),\mathcal{X}\left(t_{k-1}\right)\mid \varphi \right)\hfill \\ =\mathbb{E}\left[r\left(t_{k-1}\right)\mid s\left(t_{k-1}\right),\mathcal{X}\left(t_{k-1}\right)\right],\end{array}$$

(28)

and

$$\begin{array}{c}\hfill \overline{Q}\left(t_{k-1}\right)=r\left(t_{k-1}\right)+\gamma \underset{\mathcal{X}\left(t_k\right)}{max}Q\left(s\left(t_k\right),\mathcal{X}\left(t_k\right)\mid \overline{\varphi }\right),\end{array}$$

(29)

where $\gamma \in (0,1]$ is the discount cumulative factor indicating the degree of emphasis of future rewards, and $\varphi =\{\varphi \left(t_1\right),\varphi \left(t_2\right),...,\varphi \left(t_k\right)\}$ and $\overline{\varphi }=\overline{\{\varphi \left(t_1\right)},\overline{\varphi \left(t_2\right)},...,\overline{\varphi \left(t_k\right)}\}$ represent the weights of networks $\mathcal{Q}$ and $\overline{\mathcal{Q}}$, respectively. After determining the loss based on an action, the target network ($\overline{\mathcal{Q}}$) copies the weight of the main network ($\mathcal{Q}$), i.e., $\overline{\varphi }=\varphi$.

To utilize the past experience in a DQN-based scheduling algorithm, the proposed algorithm looks at the acquired knowledge ($\mathcal{K})$ to determine whether state $s\left(t_k\right)$ is in $\mathcal{K}$ or not. If state $s\left(t_k\right)$ is in $\mathcal{K}$, the algorithm chooses action $\mathcal{X}\left(t_k\right)$ based on an $ϵ$-greedy policy, i.e., it chooses a random action with probability $p=ϵ$ or the action with probability $p=1-ϵ$ that has the largest value for $Q\left(s\left(t_k\right),\mathcal{X}\left(t_k\right)\right)$. Based on the $ϵ$-greedy policy [24], action $\mathcal{X}\left(t_k\right)$ is defined as

$$\begin{array}{c}\hfill \mathcal{X}\left(t_k\right)=\left\{\begin{array}{l}\mathrm{random}\mathrm{action},\mathrm{with}p=ϵ\hfill \\ \arg {\max }_{\mathcal{X}\left(t_k\right)}Q\left(s\left(t_k\right),\mathcal{X}\left(t_k\right)\mid \varphi \right),\mathrm{with}p=1-ϵ\hfill \end{array}\right.\end{array}$$

(30)

In the case in which state $s\left(t_k\right)$ is not in $\mathcal{K}$, scheduling action $\mathcal{X}\left(t_k\right)$ is performed at random. After taking action $\mathcal{X}\left(t_k\right)$ based on observed state $s\left(t_k\right)$, the algorithm evaluates the immediate reward as

$$\begin{array}{c}\hfill R\left(s\left(t_k\right),\mathcal{X}\left(t_k\right)\right)=E\left[-\mathsf{\Delta }SO{H}_{\mathcal{P}}\left(t_k\right)\right].\end{array}$$

(31)

Then, the algorithm determines the cumulative reward ($r\left(t_k\right)$) by interacting with the environment and looks for an optimal policy to maximize $r\left(t_k\right)$. The cumulative reward ($r\left(t_k\right)$) is calculated as

$$\begin{array}{c}\hfill r\left(t_k\right)=\mathbb{E}\left[\sum _{h=k}^w{\gamma }^{h}R\left(s\left(t_h\right),\mathcal{X}\left(t_h\right)\right)\right].\end{array}$$

(32)

The algorithm minimizes loss function $\mathcal{L}\left(\varphi \left(t_k\right)\right)$ so that action value $Q\left(t_{k-1}\right)$ has the same value as target action value $\overline{Q}\left(t_{k-1}\right)$, which also means that the SOH of the battery pack is optimized. The DQN-based scheduling algorithm is summarized in Algorithm 3, and the DQN training process is shown in Figure 4.

4. Performance Evaluation

4.1. Simulation Environment

The simulation was conducted using a lithium-ion battery model and was implemented in MATLAB and Simulink R2022a. To evaluate the performance of the proposed algorithm, we consider a parallel-connected battery pack including four lithium 3.7 V/2.2 Ah batteries with heterogeneous states of health (90.01%, 86.77%, 84.13%, and 78.15% corresponding to cells 1 to 4, respectively). MOSFETs with low ON resistance and low power are installed to connect and disconnect the battery cells from the battery pack. We consider different power demand conditions to evaluate the effectiveness of the algorithm. Based on the maximum capacity of a battery pack with new battery cells, we obtain a dynamic power demand profile by generating values from a uniform distribution across 20% to 60% of the maximum energy of a battery pack (i.e., between 6.51 Wh and 19.54 Wh). For the constant power demand, we calculate the mean value of the dynamic power demand profile as

$$\begin{array}{c}\hfill D_{avg}=\frac{1}{W}\sum _{k=1}^{w}d\left(t_k\right),\end{array}$$

(33)

where $d\left(t_k\right)$ is the power demand at time slot $t_k$, and W is the number of time slots during working time ($\mathcal{W}=\{t_k\mid k=1,2,...,w\}$). Figure 5 shows dynamic and constant power demand profiles. Constant power demand is equal to 13.13 Wh (i.e., $40.32\%$ of the maximum energy of a new battery pack). We set the load current of the battery pack when discharging and charging to 8 A.

A dataset compiled by NASA [22] was used to model a first-order Thévenin equivalent battery model with a reduction in SOH. We also use the dataset to obtain actual SOC and SOH values, which are compared with the estimated values. The dataset includes 28 lithium cobalt oxide 18,650 cells with a nominal capacity of 2.2 Ah, including in-cycle measurements of terminal voltage, current, and cell temperature. The dataset also includes measurements for discharging capacity and EIS impedance readings. We identify the EIS parameters, which include $V_{Oi}$, $R_{si}$, $R_{pi}$, and $C_{pi}$, in the 90% to 60% SOH range using the dataset.

The structure of neural networks includes one 10-dimension input layer, two 256-dimension hidden layers, one 256-dimension LSTM layer, and one 16-dimension output layer. The input layer consists of 10 elements of the battery state ($s\left(t\right)$), since there are four battery cells in a battery pack. The output layer consists of 11 cases (There must be at least two batteries ON at the same time, since we consider $8A$ current during discharging and the maximum output current of one battery is 4 A) of schedule action $\mathcal{X}\left(t\right)$. We set the learning rate ($\alpha$) to $0.001$, the $ϵ$-greedy value to $0.9$, and the discount factor ($\gamma$) to $0.99$. The period of the target network update is 10 time steps. Other simulation parameters are summarized in

Table 2. Simulation parameters.

Parameter	Value
Number of battery cells	4
Battery type	Lithium 3.7 V/2.2 Ah
Total capacity (new)	32.56 Wh
Constant power demand	13.13 Wh (40.32%)
$I_{discharge}$	8 A
$I_{charge}$	−8 A
$I_{min}^{-},I_{max}^{+}$	−4 A, 4 A
$SO{C}_{min},SO{C}_{max}$	10%, 90%
$\eta$	1 (discharge)/0.98 (charge)
Total working time ($\mathcal{W}$)	1800 h
$\mathsf{\Delta }t$	10 min
Capacity M of experience $\mathcal{E}$	500 slots
Learning rate ($\alpha$)	0.001
$ϵ$-greedy	$0.9$
Discount factor ($\gamma$)	$0.99$
Period of target network update	10 time slots

.

For the performance evaluation, we first verify the accuracy of the estimation algorithm by determining the error between estimated and actual values. Then, we investigate the effect of the proposed algorithm on the lifetime of a battery pack and the SOHs of the cells under dynamic and constant loads. To validate the performance of the proposed algorithm, we compare it with methods proposed in previous works, including an enhanced Coulomb counting method [7], a hybrid statistical data-driven estimation method [11], and a multi-actor–critic scheduling algorithm [6]. For comparison, we combine the scheduling and estimation algorithms and obtain the BESS performance. We also compare the proposed estimation algorithm with the enhanced Coulomb counting method and the hybrid statistical data-driven estimation method. For the sake of simplicity, we denote the proposed third-order extended Kalman filter (EKF) estimation algorithm as EKFest, the proposed deep Q network scheduling algorithm as DQNsch, the multi-actor–critic scheduling algorithm as MACsch, the hybrid statistical data-driven estimation method as DDest, the enhanced Coulomb counting method as ECest, and simulations without any scheduling algorithm as Non Schedule.

4.2. State Estimation Verification

To evaluate the performance results of the proposed algorithm in estimating the SOC and SOH for each cell, we first show the estimated terminal voltage of each cell in a battery pack. Figure 6 shows the root mean square error (RMSE) between the measured terminal voltage and the estimated terminal voltage. The RMSE between the measured and estimated values of the terminal voltage for each cell is close to 0.01 V and remains small over time. The small difference between measured and estimated terminal voltages shows that the proposed algorithm accurately models terminal voltage, which leads to a more accurate estimation of the SOC and SOH of a cell.

The performance results of the proposed algorithm in estimating the SOC and SOH for each cell in terms of RMSE and mean absolute error (MAE), respectively, are shown in Figure 7. The proposed estimation algorithm has the lowest RMSE compared to other works in estimating the SOCs of cells, as shown in Figure 7a. The RMSE between the actual and estimated values of the SOC for each cell is close to 1% under the proposed algorithm. The error of the proposed algorithm in estimating the SOHs of the cells is shown in Figure 7b. The proposed algorithm has an error of less than 0.2% for SOH, which is 50% less than the other estimation algorithms. ECest shows the worst performance, degrading over time. Note that the performance of the proposed estimation algorithm becomes more stable over time. Estimating the SOC and SOH of the cells with low error is of great significance in order to obtain optimal ON/OFF cell scheduling that extends the lifetime of a battery pack.

4.3. Impact of the Proposed Algorithms on Battery Pack Lifetime

The impact of the proposed algorithm on battery pack lifetime in terms of SOH reduction under constant and dynamic power demands is evaluated and shown in Figure 8. The proposed algorithm achieves better performance under both constant and dynamic power demands compared to other algorithms. The proposed algorithm reduces the SOH decay in the battery pack by efficiently scheduling the ON/OFF switching of the cells based on accurate estimation of SOHs and SOCs, resulting in an increase in battery pack lifetime.

The SOH of the battery pack reaches 60% (the end of its second life (EoL)) after a working time of 1767 h under constant power demand, which represents a 13.9% increase in battery pack lifetime compared to previous work (DDest + MACsch). Under dynamic power demand, battery pack lifetime also increases by 20.6% under the proposed algorithm compared to previous work. In addition, the difference in the performance of the proposed algorithm under constant and dynamic power demand is quite small, but the performance of methods proposed in previous work degrades under dynamic power demand. Hence, the proposed algorithm can hence efficiently schedule ON/OFF switching of battery cells to adapt to dynamic power demand.

Compared to DDest + MACsh, the lifetime of the battery pack is higher under EKFest + MACsch and DDest + DQNsch. This shows that the proposed estimation algorithm, as well as the scheduling algorithm, can an impact in extending the lifetime of a battery pack. DDest + DQNsch achieves better performance than EKFest + MACsch, which means optimal scheduling is a more dominant factor in prolonging battery pack lifetime. MACsch achieves worse performance, since it does not consider SOC while scheduling the ON/OFF cell switches to meet power demand. Without scheduling (Non-Schedule), the lifetime of the battery pack reduces rapidly because the weakest cell, i.e., the cell with the lowest SOH, operates continuously.

4.4. Impact of the Proposed Algorithm on Capacity Balancing

The effectiveness of the proposed algorithm in balancing the SOH of cells under constant and dynamic load demands is shown in Figure 9 and Figure 10, respectively. Without a scheduling algorithm (Non-Schedule), all the cells in the battery pack are utilized all the time, irrespective of their SOC and SOH, resulting in imbalanced states of health and increasing SOH reduction in the battery pack, irrespective of load demand conditions, as shown in Figure 9a and Figure 10a.

All the algorithms balance the SOH of cells in the battery pack under constant and dynamic load demands, as shown in Figure 9b–e and Figure 10b–e, respectively. Even though the methods proposed in other works achieve SOH balancing among battery pack cells, battery lifetime (the SOH of each cell) decreases rapidly under the other algorithms compared to the proposed algorithm. This means that with heterogeneous states of health for cells in a battery pack, the proposed algorithm offers better performance than other algorithms by extending the second life of battery cells. All the algorithms achieve SOH standard deviations close to zero by balancing the capacity of each cell over time under constant power demand, which can be seen in Figure 9f.

Under dynamic load demand, EKFest + DQNsch achieves more even SOH balancing and reduces the standard deviation of the cells’ SOHs to zero, while other algorithms fail to balance the SOHs of cells, except for the DDest + DQNsch, which achieves the second-best performance, as shown in Figure 10b–f. The SOH of the weakest cell (cell 4, which has the lowest initial SOH) reaches 60%, while other cells have SOHs of more than 60% under algorithms proposed in other works, resulting in higher standard deviations and earlier end of second life of the battery pack. DDest + DQNsch reduces the standard deviation of SOHs and extends battery life compared to other scheduling algorithms. This shows the effectiveness of the proposed scheduling algorithm in managing a parallel-connected BESS, even with a less accurate estimation algorithm. The superior performance of the proposed algorithm under the different load demand conditions shows the robustness of the algorithm to load demands.

4.5. Impact of Numbers of Batteries on the Proposed Algorithm

We study the impact of the number of parallel-connected batteries for the BESS on the proposed algorithm under dynamic load demand according to the SOH profiles shown in

Table 3. SOHs of batteries.

Number of Batteries	SOH Profile (%)	Total Max. Capacity
3	89.59, 85.14, 79.57	5.59 Ah
4	90.01, 86.77, 84.13, 78.15	7.46 Ah
5	91.05, 87.95, 84.76, 81.95, 78.15	9.32 Ah
6	91.17, 90.05, 84.86, 82.67, 81.65, 78.21	11.19 Ah

. The SOH profiles of batteries have the same SOH average ($84.77\%$) and standard deviation ($5.02\%$).

The performance of the proposed algorithm under different battery conditions in terms of the operational working time and standard deviation in SOHs is shown in Figure 11. The proposed algorithm (EKFest + DQNsch) achieves higher operational time (i.e., extends the second life of a battery pack) compared to other algorithms, as can be seen in Figure 11a. The proposed algorithm minimizes the SOH reduction of the battery pack in each time slot by balancing the SOHs of battery cells, thereby extending the battery pack’s lifetime.

The proposed algorithm achieves the lowest standard deviation with different numbers of batteries in a battery pack, as shown in Figure 11b. The standard deviation in SOHs increases by a minimal amount under the proposed algorithm with an increase in the number of batteries compared to other algorithms. The combinations of the proposed estimation and the proposed scheduling algorithms with the algorithms proposed in previous works (EKFest + MACsch and DDest + DQNsch) increase the lifetime of a battery pack and achieve a more uniform SOH balance compared to the combination of previously proposed algorithms (i.e., DDest + MACsch). This shows the effectiveness of both parts of the proposed algorithm in the optimization of BESSs. Figure 11 shows that the proposed algorithm is robust to the number of battery cells in a battery pack in a BESS.

5. Conclusions and Future Work

In this paper, we proposed a DRL-based battery management algorithm to optimize battery lifetime for retired batteries with heterogeneous SOHs in a parallel-connected BESS. The proposed algorithm

(i)

estimated the SOCs and SOHs of all battery cells using EKF;

(ii)

used estimated SOCs and SOHs to represent the state of a BESS for DRL-based scheduling; and

(iii)

controlled the ON/OFF switches of battery cells inside the battery pack utilizing deep Q network knowledge.

Via simulation, we showed that the proposed algorithm outperformed other proposed algorithms by showing lower estimation errors for battery cell states and extending the battery pack’s second life. The proposed algorithm extended the operation time of the battery pack by 13.9% and 20.6% compared to other algorithms under constant and dynamic power demand, respectively.

Regarding future work, we will consider a BESS in which multiple battery packs are connected in series and each battery pack has parallel-connected battery cells. Such a configuration leads to high dimensions of state space. Furthermore, the deployment of smart-grid technologies that include energy storage systems [25] requires hundreds of battery cells connected in parallel or in series in a BESS. In such systems, DRL-based battery management algorithms can achieve limited performance due to high-dimensional state space. We will investigate a distributed reinforcement learning approach to counter the limitations of centralized approaches for large-scale energy storage systems. Additionally, an experimental setup will be considered to observe the impact of the battery management algorithm on real systems.