Impact Analysis of Battery Energy Storage System and Demand Response in Microgrid

Abstract

Coordinated scheduling of distributed generation (DG), battery energy storage systems (BESS) and demand response (DR) is needed to reduce grid procurement costs without compromising customer comfort or network constraints. This paper proposes a two-stage framework for a grid-connected IEEE 33-bus microgrid. In the first stage, the Red Panda Optimization (RPO) algorithm jointly determines the locations and ratings of renewable DG units and BESS by minimizing power losses and voltage deviation. In the second stage, five customer agents with different cost–comfort preferences and three BESS agents learn their hourly schedules under real-time pricing using multi-agent Q-learning. Two incentive regimes are compared within the same decentralized framework: a price-only, customer-driven regime and an incentive-coordinated regime in which the grid operator broadcasts a shadow-price signal. Compared with the DG-only case, the two regimes reduced daily demand by 2.36% and 12.21%, energy loss by 4.53% and 27.79%, and procurement cost by 3.25% and 12.99%, respectively. Customer load curtailment provides all the demand and loss reduction and most of the cost reduction, while BESS arbitrage lowers the cost by a further 0.62 and 0.61% points at the expense of slightly higher losses. The incentive design, rather than the control architecture, largely determines how these benefits are shared between the operator and customers.

1. Introduction

1.1. Motivation

Nowadays, energy management is a challenging task due to the integration of renewable energy resources and customers’ participation in buying and selling power from the grid. Along with this another issue is the weather dependency of renewable energy based DG’s which create the uncertainty in power production like solar based DG is only produce power during day time and wind based DG produce power only when the wind is available which make the system unreliable and can arise the issue of instability within the system due to this issue a reserve energy resources must be integrate to store the surplus energy for stable operation and make system more reliable [1,2].

A Battery Energy Storage System (BESS) is installed to store surplus energy. Its key applications are providing backup power to critical loads during emergencies and compensating for peak load demand [3], and it also enhances power stability [4], reduces grid demand, and minimizes power loss [5]. Complementing storage-based solutions, another prominent approach for enhancing grid performance is Demand Response (DR). This customer-centric strategy encourages customers to schedule their load based on electricity prices, during peak demand periods, or when a system risk is detected [6]. Demand response mechanisms generally fall into two categories: price-based and incentive-based. Within the price-based category, the most common tariff structures include time-of-use (ToU) [7], real-time pricing (RTP) [8], and critical peak pricing (CPP) [9].

1.2. Literature Review

The primary focus of this study is to efficiently install and schedule the BESS and study the benefits of implementing a demand response program in the distribution system. Installing BESS in a system is a complex task; studies focus exclusively on determining optimal capacity, optimal location, or both. By installing the BESS, system performance improves; for example, in [10], an optimal location is determined to shave peak loads and minimize energy loss using a real-coded genetic algorithm. In [11], a time-of-use tariff is used to determine the optimal BESS location and hourly operation using the teaching–learning-based optimization (TLBO) algorithm, thereby reducing purchased energy costs and losses and curtailing peak load over 24 h. The author of [12] finds the optimal location for an energy storage system to minimize the social hourly cost of a deregulated power system, accounting for wind power penetration.

In [13], the MINLP algorithm is used to reduce total annual operating cost by determining the optimal location of the energy storage system. The author of [14] allocates BESS in an unbalanced distribution network, considers a nonlinear load, and also accounts for the different features of BESS using a hybrid model of GA and a quasi-static time series technique. A master-slave methodology is used, in which Chu & Beasley’s parallel genetic algorithm is the master, and vertex search optimization is the slave, to find the optimal placement of BESS that reduces total energy loss and carbon emissions [15]. The study in [16] identifies the optimal BESS location to mitigate voltage rise in a PV-based distribution network using a binary firefly algorithm (BFA).

In [17], a suitable location for installing a BESS is identified to improve system reliability and voltage and frequency stability. The author in [18] selects the location of BESS using a genetic algorithm to achieve peak load shaving and peak loss reduction. A comparison of four algorithms, GA, PSO, direct search optimization, and GWO, is performed to find the optimal location [19] for the minimization of power loss and the improvement of voltage quality, and also considers different battery configurations. Another approach to integrating BESS into the system is to optimize sizing.

In [20], an optimal size of BESS is determined by using an Enhanced Opposition-Based Firefly Algorithm (EOFA) to minimize bus voltage deviation and BESS inactivity. In contrast, ref. [21] uses NSGA-II for size optimization to reduce power loss and minimize grid demand cost under a time-of-use tariff plan. In article [22], the author applies an artificial bee colony optimization algorithm to minimize the total operating cost of a microgrid under different regulatory frameworks and a dynamic feed-in tariff (FiT) by selecting the optimal BESS capacity, thereby improving the microgrid’s reliability. In [23], the capacity of the energy storage system is optimized to integrate with PV power generation and electric vehicles for residential use, accounting for electricity consumption costs. The Author of [24] finds the optimal size by applying an Improved PSO with a variable-interval reference modulation strategy to minimize output deviation, maximize annual costs, and provide voltage support. Another study presented in [25,26] provides voltage and frequency support by selecting the optimal size for a 33-bus system. To achieve peak load shaving and mitigate grid congestion, the optimal BESS size is determined using the interior-point algorithm [27]. The bat-algorithm-based sizing investigation in [28] is improved to minimize the total operational cost in microgrid energy management. In [29], a multi-period mixed-integer linear programming (MILP) model is developed to optimize the integration of BESS and DG in power distribution systems.

To address uneven load distribution and improve power quality, the optimal location and placement of BESS are determined in [30]. Similarly, in [31], a Multi-Objective Particle Swarm Optimization (MOPSO) method is applied to minimize costs and reduce power losses, highlighting applications in power quality improvement and investment deferral for network reinforcement.

A trade-off risk approach is used in [32] to determine the optimal size and location of BESS under uncertainty in the input data in an unbalanced active distribution system. From an economic perspective, the author of [33] developed a modified PSO strategy to maximize the distribution company (DISCO) revenue, with a focus on network support. The author of ref. [34] uses a bi-level approach to reduce costs at the upper level and network loss at the lower level in the P-2-P energy distribution network. The author in [35] minimizes the costs of the energy storage system and the overall system, as well as transmission line losses, by selecting the optimal location and size of the BESS. Another study in [36] considers the residual value and minimizes investment cost by selecting the appropriate size and location of the BESS. In [37], a Hybrid Multi-Objective PSO (HMOPSO) is used to reduce the total expected operation cost of the system. The author of [38] uses an adaptive, particularly tunable, fuzzy particle swarm optimization to minimize frequency fluctuations and improve static and dynamic stability. Several optimization algorithms are used to determine BESS size and placement; for example, the author of [39] utilized a Modified Non-Sorting Dominated Genetic Algorithm (NSGA) to determine the optimal size and location of BESSs with dual objectives: minimizing power losses and improving the voltage profile, thereby enhancing voltage stability. The Whale Optimization Algorithm (WOA) was employed in [40] to minimize total system losses through optimal BESS siting and sizing. Further work on voltage stability was conducted using NSGA-II.

The author of [41] uses a genetic algorithm to support the distribution network by minimizing voltage deviations and power losses. Another application of GA is presented in [42], where a bi-level optimization model is used to concurrently reduce voltage deviation. In [43], a Sequential Quadratic Programming (SQP)-based inner algorithm is employed to locate BESSs to minimize cost and power loss. Integrated fuzzy logic with PSO is used to minimize total cost and power losses, focusing on energy arbitrage to reduce risk in the distribution system, as presented in [44]. A significant research effort has been devoted to the scheduling of BESS in distribution networks [45]. In [46], a battery energy storage system is installed to reduce energy not supplied and power losses by applying a decomposition-based evolutionary algorithm. In [47], the author schedules the BESS to reduce operating costs, transportation-related energy losses, and CO₂ emissions. The impact of BESS integration on distribution system reconfiguration is studied in [48]. In [49], charging and discharging of BESS for sessional load are studied to reduce voltage deviation and real power loss by using a max-min fuzzy logic system. The reduction in carbon trading revenue and the minimization of overall costs, with power loss reduction through BESS scheduling, are studied in [50]. Another study [51] examined the efficient management of BESS to reduce energy losses, operating costs, and carbon emissions. To improve system reliability and power flow, an efficient scheduling method for BESS is presented in [52].

Several energy management techniques are studied to optimize the operation of a grid-connected microgrid, taking into account the demand response program. Energy management optimization that considers demand-side response and the carbon trading market under uncertainty is discussed in [53]. The techno-economic impact of a grid participant, with the main aim of designing a demand response model that maximizes the microgrid customer’s benefit, is presented in [54]. In [55], a stochastic energy management system for a microgrid to reduce costs and emissions accounts for uncertainty through Monte Carlo simulation. The method for assessing the financial and environmental advantages of energy system planning was presented in [56], enabling decision-makers to make well-informed choices through demand response. An incentive-based demand response program that accounts for uncertainty in DG, using Hong (m + 1), is presented in [57].

More recent studies have moved from planning a single asset to the coordinated planning and use of several flexible resources. In [58], transportable and stationary flexible resources are simultaneously planned for distribution systems, thereby contributing to better renewable energy integration and improved resilience to extreme conditions. In [59], flexibility-based planning of a distribution system penetrated by photovoltaic generators is performed to enable absorption of generation variations with respect to operational constraints without exceeding network capacity. These studies show that the importance of using storage and demand-side resources lies in the need to plan them together with renewable DG. Nevertheless, they deal only with the planning aspect, modeling the operation of flexible resources within the optimization framework. The current study considers the flexibility-based perspective for planning DG and BESS together. It extends that view by adding a learning-based operational stage in which customers and BESS agents adapt their schedules to price and incentive signals.

A number of the studies listed above, despite being methodologically sound, suffer from shortcomings that inspire the current effort. For instance, sizing-only studies such as [20,21,22,23,24,25,26,27,28,29] do not consider siting co-optimization; siting-only studies such as [10,11,12,13,14,15,16,17,18,19] do not consider capacity co-optimization; and, for the most part, scheduling studies such as [45,46,47,48,49,50,51,52] treat BESS deployment as a parameter rather than a co-decision variable alongside DG. In addition, demand response studies such as [53,54,55,56,57,60,61,62,63] use deterministic optimization, which is solved repeatedly at decision-making time points when prices change very quickly, and typically do not pose the scheduling problem as a multi-agent Markov Decision Process with individual state, action, and reward definitions for agents. Related demand-side management studies, such as [64,65], solve microgrid scheduling as a single-stage metaheuristic optimization (Black Widow Optimization and quantum-behaved particle swarm optimization, respectively), and reinforcement-learning studies generally address pricing or scheduling alone. Here, we combine an MDP formulation and multi-agent Q-learning with the planning stage and, most importantly, compare two incentive regimes (price-only and grid-operator incentive) within a single decentralized learning framework.

In [60], methods for the optimal location of EV charging stations and their impact on the distribution network are reviewed. Also, the author of [61] implemented a time-of-use-based demand response program to shift load from peak to off-peak periods. In [62], a demand response program is implemented to improve power quality and mitigate harmonics in the system voltage. The author in [63] solves a multi-objective problem that combines cost and environmental factors and accounts for uncertainty in electricity prices, wind energy, and load demand. In [66], the integration of price-based demand response into a microgrid is carried out in two phases: first, estimate DG capacity, and second, apply demand response in the microgrid with EV loads. In [67], the impact of BESS and demand response programs on a hybrid energy resource-based microgrid is examined to reduce annual grid energy purchases. However, optimization-based methods are effective for deterministic formulations. They may require frequent model recalculations when electricity prices, load demand, and renewable energy generation change rapidly. Reinforcement learning provides an alternative data-driven framework in which agents learn scheduling policies through interaction with the environment. This motivates the use of a multi-agent Q-learning model for real-time customer load and BESS scheduling underprice-based demand response.

Energy management reinforcement learning algorithms have also been rapidly evolving from tabular Q-learning methods. The use of a deep Q-network (DQN) replaces the Q-table with a neural network to handle large or continuous state spaces [68]. Another development is actor–critic methods, in which continuous action spaces are considered: a deterministic policy is learned using DDPG [69]; the TD3 approach mitigates the overestimation bias of DDPG using twin critics and delayed policy updates [70]; and entropy regularization is introduced in SAC [71]. These methods have been applied to many different energy systems, including a TD3-based approach for simultaneously managing fuel-cell and battery conditions, as well as the air-conditioning load, in a hybrid bus [72].

In demand response, reinforcement learning has found application in the form of dynamic pricing between service provider and consumer sides [73], and the algorithms and modeling issues have been surveyed in [74]. Deep learning techniques, on the other hand, involve designing neural networks, a much larger number of hyperparameters, and higher training costs, with no guarantee of convergence. The current setting involves small, discrete state spaces (24 states per customer agent and 240 states per BESS agent), and the problem is deterministic, so Q-learning can converge to a consistent solution [75] with a readily interpretable schedule. Extending the framework to DRL methods such as TD3 or SAC is the natural next step once forecast uncertainty and richer state information are introduced.

1.3. Research Objective

This paper proposes a multi-objective framework for the stable and economical operation of a distributed microgrid through the coordinated installation of renewable DG units, BESS, and demand response. The method is divided into two stages. During installation, the optimal locations and sizes of DGs and BESS are determined by minimizing active and reactive power losses and voltage deviations while satisfying bus voltage limits. In the scheduling stage, customer operators and BESS operators are modeled as learning agents that select hourly actions under real-time pricing. The scheduling objective is to reduce grid procurement costs and power losses while maintaining customer comfort and BESS state-of-charge constraints. The effectiveness of the proposed framework is evaluated under two incentive regimes within the same decentralized learning framework: a price-only (customer-driven) regime and an incentive-coordinated (grid-operator-driven) regime. Table 1 provides a comparative assessment of the current work against recent studies.

Table 1. Comparative assessment of current work against recent studies.

Ref.	DG	BESS	Objective	Algorithm	Excludes	Input Data	Decision Variables
[67]	Y	Y	Minimize the net cost of energy	JAYA	No reactive power; no explicit scheduling agent model	Load, price series	DG/BESS size, cost
[64]	Y	Y	Overall system cost	Black Widow Optimizer	No demand-response scheduling stage	Load, cost data	DG/BESS size
[65]	Y	Y	Load shaping	QPSO	No BESS siting (capacity only)	Load profile	BESS capacity, schedule
[76]	Y	N.A.	Minimize PAR and the electricity bill	CMEWDO	No BESS; no network loss/voltage objective	Load, price	Load schedule
[77]	Y	N.A.	Maximize incentive	Elephant Hunt Optimization	No BESS; single-agent formulation	Load, incentive rates	Load schedule
[78]	Y	Y	Minimize energy cost	MINLP	No explicit multi-agent scheduling	Network + load data	DG/BESS siting
[79]	Y	Y	Minimize operational cost	MINLP	No RL-based scheduling	PV uncertainty, load	DG/BESS siting
[80]	Y	Y	Minimize buying cost, VD, loss	SOCP	No demand-response agent behavior	Network, price	DG/BESS siting
Proposed Study	Y	Y	Minimize grid procurement cost, system power loss, voltage deviation, and grid power demand	Q-learning (scheduling) + RPO (planning)	Forecast/renewable uncertainty not modeled (flagged as future work)	Network topology, 24 h load/price/irradiance/wind series	Joint DG + BESS siting/sizing (RPO) and hourly customer/BESS schedules (Q-learning)

N.A.: Not Applicable.

1.4. Major Contributions

In the literature, the planning and scheduling problems are mostly solved with optimization techniques [64,65,76,77,78,79,80]. The closest studies differ from the present work in three respects. In [64,65], microgrid energy management with demand-side management is solved as a single-stage metaheuristic optimization (Black Widow Optimization and quantum-behaved particle swarm optimization, respectively); DG and BESS are not sited and sized in the same procedure, and customer behavior is not learned. In [80], BESS is allocated considering a demand response program, but customer response is represented inside a centralized optimization model rather than by autonomous agents with individual preferences. The present work differs in that (i) the joint DG–BESS siting and sizing result directly sets the action bounds and SOC limits of the learning agents, so the two stages are coupled through one decision vector; (ii) each customer is an autonomous learning agent with its own cost–comfort trade-off; (iii) the effect of a grid-operator incentive on decentralized learning is quantified; and (iv) the benefit is decomposed into DR and BESS contributions and tested for economic viability and robustness. This paper does not propose a new optimization or learning algorithm, and no claim is made that RPO outperforms other metaheuristics: the planning stage is solver-agnostic, and PSO reaches a comparable design (Section 5.4). The main contributions are as follows:

• A two-stage DG–BESS–DR framework is developed in which the joint siting and sizing of DGs and BESS (solved with the parameter-free RPO algorithm) sets the operating envelopes of the scheduling agents.
• A demand response framework driven by dynamic pricing is developed in which each customer operator has a distinct cost-comfort trade-off represented through a discomfort coefficient and a weighting factor.
• The impact of grid-operator incentive signals on decentralized learning is quantified by comparing two regimes within the same multi-agent framework: a price-only (customer-driven) regime with individual comfort-based curtailment limits, and an incentive-coordinated (grid-operator-driven) regime in which a shadow-price signal and a uniform curtailment limit are broadcast to all agents.
• A multi-agent Q-learning framework is used to schedule customer load and BESS operation, where each agent learns an hourly action policy from the hourly price (hour index) and, for BESS agents, the state of charge.
• A techno-economic assessment of the BESS and a robustness check under random renewable, load, and price deviations are provided to test the practical value of the results. At the same time, the training/convergence data (total daily reward per training episode) have been explicitly provided for the Q-learning-based schedulers to demonstrate convergence and robustness across both phases of the framework, rather than relying on a single example.

The remainder of this paper is organized as follows. Section 2 presents the network, DG, BESS, and customer models, as well as the input data. Section 3 formulates the planning and scheduling objectives and constraints. Section 4 describes the two-stage methodology: RPO-based planning and multi-agent Q-learning scheduling. Section 5 reports and discusses the simulation results, and Section 6 concludes the paper.

2. Mathematical Modeling

The study is conducted on the IEEE 33-bus radial Distribution Network to assess the efficacy of the proposed technique. For mathematical calculations, the system is converted to per-unit quantities with a base voltage of 12.66 kV and a base power of 100 MVA. The total active power is 3.715 MW, and the total reactive power is 2.3 MVAr during peak load conditions. The system is modified by installing two renewable-based DGs (wind and solar) and a set of three energy storage devices. For the study of the demand response program, the model is further modified to include five customers assigned to buses 9, 14, 22, 25, and 30, each with different desires and characteristics. The RPO algorithm determines the optimal location and capacity of DGs and BESS. Further, the hourly scheduling of customer load and BESS power is performed using the proposed multi-agent Q-learning method.

2.1. Distributed Generator Modeling

To determine the power extracted by solar- and wind-based DG, the primary inputs are solar irradiation and wind speed, which depend mainly on weather conditions that vary hourly. Therefore, for every 24 h, solar irradiation and wind speed are different, as shown in Figure 1.

Figure 1. Hourly solar irradiance and wind speed [79].

The power extracted from a PV module is calculated by Equation (1) [81].

$$P_{\mathit{PV}}^h=P_{\mathit{PV}}^r{\displaystyle \frac{{G_h}^2}{G_{\mathit{std}}{G}_c}},\hspace{1em}0\le G_h<G_c;\hspace{1em}{P}_{\mathit{PV}}^r{\displaystyle \frac{G_h}{G_{\mathit{std}}}},\hspace{1em}{G}_c\le G_h<G_{\mathit{std}};\hspace{1em}{P}_{\mathit{PV}}^r,\hspace{1em}{G}_h\ge G_{\mathit{std}}$$

(1)

where $P_{\mathit{PV}}^h$ is the PV output in hour h (kW), $P_{\mathit{PV}}^r$ the rated PV power (kW), G_h the solar irradiance in hour h (kW/m²), G_std = 1 kW/m² the irradiance at standard test conditions and G_c = 0.15 kW/m² the certain (threshold) irradiance. The wind power extracted from a wind turbine is calculated by Equation (2) [82].

$$P_W^h=P_W^r{\displaystyle \frac{{v_h}^3-{v_{\mathit{ci}}}^3}{{v_r}^3-{v_{\mathit{ci}}}^3}},\hspace{1em}{v}_{\mathit{ci}}\le v_h<v_r; P_W^r,\hspace{1em}{v}_r\le v_h\le v_{\mathit{co}}; 0,\hspace{1em}\mathit{otherwise}$$

(2)

where $P_W^h$ is the wind turbine output in hour h (kW), $P_W^r$ the rated wind power (kW), v_h the wind speed in hours h, and v_ci = 3, v_r = 12 and v_co = 25 m/s the cut-in, rated, and cut-out speeds. The rated PV (bus 14) and wind (bus 31) powers are determined by RPO (Section 5.1).

2.2. Modeling Assumptions

The network is a balanced three-phase radial system, modeled by its single-phase equivalent and solved with the backward/forward sweep load flow [83] (base 12.66 kV, 100 MVA). The study covers one day at an hourly resolution (Δt = 1 h), and the day-ahead load, price, irradiance, and wind-speed data are taken as known. Loads are constant-power and follow the same normalized shape (Table A3), and the curtailed load is not recovered later. The PV, wind, and BESS units exchange only active power, and power is imported from the upstream grid at an exogenous real-time price. Each BESS has η_chg = η_dchg = 0.9, with no self-discharge or simultaneous charging and discharging. Siting and sizing are performed at peak load, and the resulting design is evaluated over the full 24 h.

2.3. Modeling of BESS

A Battery Energy Storage System (BESS) can serve either as a load or as a distributed generator (DG), with its operating mode dictated by the prevailing load demand and the generation from other DG units. In charging mode, the BESS consumes power and acts as a load; in discharging mode, it supplies power and performs as a DG. The system pursues three primary objectives: minimizing total charging cost, maximizing discharge revenue, and minimizing overall network power losses. The scheduling of charging and discharging intervals follows the analytical procedure proposed in [49], which utilizes the 24 h system load profile. More specifically, the BESS power capacity is determined by the average load over the entire study period, and the intervals are defined by comparing each hourly load to this average. A surplus indicates discharging, and a deficit signals charging. For load flow calculations, the BESS is represented as a positive power injection during charging and a negative injection during discharging (i.e., as a generator), with the net power exchange obtained from Equation (3). Typical seasonal operating patterns for summer are depicted in Figure 2. At the same time, Table 2 summarizes the state-of-charge (SOC) ranges for each battery under test, noting that the scheduling algorithm applies distinct SOC targets for individual BESS units.

$$P_{\mathit{BESS},\mathit{ref}}^h=\left({\lambda }_h-\overline{\lambda }\right){\displaystyle \sum _{k=1}^{N_{\mathit{bus}}}}{P}_k^{\mathit{load}}$$

(3)

where λ_h is the normalized load shape factor in hour h (p.u., Table A3), $\overline{\lambda }={\displaystyle \frac{1}{24}}{\displaystyle {\sum }_{h=1}^{24}}{\lambda }_h$ its daily mean, and $P_k^{\mathit{load}}$ the peak active load at bus k (kW). A positive value indicates a discharging hour and a negative value for a charging hour.

Figure 2. Battery charging and discharging slot estimation.

Table 2. Minimum and maximum state-of-charge limits adopted for each BESS unit.

	Min SOC	Max SOC
BESS 1	0.3	0.9
BESS 2	0.2	0.9
BESS 3	0.3	0.8

Equation (3) and Figure 2 describe a rule-based reference pattern used only in the planning stage to illustrate the hours in which charging and discharging are expected to be beneficial. This rule is not used during scheduling. In the operational stage, each BESS is controlled by its own Q-learning agent, whose Q-table is initialized to zero and whose hourly decisions are bounded only by Equations (10)–(17).

2.4. Modeling of Customer Operator

In a price-based demand response program, customers are entitled to adjust their electricity consumption patterns in response to real-time or time-varying market prices. In this framework, customer operators act as autonomous agents that make curtailment decisions based on both price signals and individual customer preferences. The optimization problem governing the customer operator’s decision-making process is formulated in Equation (4):

$${\displaystyle \sum _{h=1}^{24}}{\displaystyle \sum _{i=1}^{N_c}}\left[{\alpha }_{1,i}{E}_p^h{L}_{d,i}^h\Delta t+{\alpha }_{2,i}{\beta }_i{\left(L_{a,i}^h-L_{d,i}^h\right)}^2\right]$$

(4)

where $E_p^h$ is the real-time price in hour h ($/MWh), $L_{d,i}^h$ the demand of customer i after curtailment (MW), $L_{a,i}^h$ its reference demand without curtailment (MW), β_i the dissatisfaction coefficient of customer i ($/MW²), α_1,i and α_2,i the cost and comfort weights (Table 3), N_c = 5 the number of customers and Δt = 1 h.

Table 3. Customer characteristics.

Customer	Bus	Discomfort Coefficient β	Weights (α1, α2)	Comfort Index CI = α2β/α1	Case 1 Ceiling cimax (%)	Remark
C1	9	0.24	(0.5, 0.5)	0.240	40.0	Balanced cost and comfort
C2	14	0.18	(0.4, 0.6)	0.270	19.7	Comfort oriented
C3	22	0.12	(0.3, 0.7)	0.280	15.9	High comfort
C4	25	0.09	(0.2, 0.8)	0.360	3.5	Very high comfort
C5	30	0.045	(0.1, 0.9)	0.405	1.7	Extreme comfort

The first term in Equation (4) represents the total expenditure on electricity. In contrast, the second term captures the disutility or discomfort cost arising from load curtailment, and alpha is introduced to strike an appropriate balance between economic cost and user discomfort; its value depends on customer-specific characteristics and preferences, as summarized in Table 3. Equation (4) gives the general quadratic form of the dissatisfaction cost.

The dissatisfaction coefficients β_i in Table 3 are scenario parameters rather than values calibrated from field data. Following the dissatisfaction-cost models commonly used in price-based demand response [73,84], the comfort index ${\mathit{CI}}_i={\displaystyle \frac{{\alpha }_{2,i}{\beta }_i}{{\alpha }_{1,i}}}$ has been chosen to increase monotonically from C1 to C5, representing customers that range from cost-oriented to strongly comfort-oriented. The curtailment ceiling in Case 1, $c_i^{\mathit{max}}=0.40{\left({\displaystyle \frac{{\mathit{CI}}_{\mathit{min}}}{{\mathit{CI}}_i}}\right)}^6$ depends only on the ratio of comfort indices, where CI_min is the smallest comfort index among the customers. Scaling all β values by a common factor, therefore leaves the ceilings unchanged, and the uniform 40% ceiling of Case 2 is independent of β. The results are thus governed by the relative ordering of the customers rather than by the absolute scale of β.

In the scheduling stage, the discomfort term of Equation (7) is implemented in normalized form as α₂βK_D x⁶, where x is the curtailed fraction of the hourly reference demand and K_D = $1000 is a scaling constant (Appendix A, Table A1). The higher exponent keeps shallow curtailment almost free of discomfort while steeply penalizing curtailment that approaches the ceiling. A sensitivity analysis (Appendix A, Table A6) shows that scaling all β values by a common factor between 0.5 and 1.5 changes the system cost reduction in Case 1 only from 2.72% to 2.56%. However, changing the β of the least comfort-oriented customers (C1) by ±20% shifts all comfort ceilings and changes the cost reduction from 1.22% to 4.38%. The relative calibration of β, therefore, matters more than its absolute scale.

2.5. System Configuration and Load Data

The case study network used in this paper is the IEEE 33-bus radial distribution system with a 12.66 kV and 100 MVA base, having a total peak load of 3.715 MW/2.3 MVAr. The objectives of planning and scheduling considered in this research are economic and technical (cost of procurement, power losses, and voltage deviation), with reliability extension as a consideration. Load profiles for every hour within the 24 h case study are generated by multiplying the normalized daily load shape factor (with a midday and an evening plateau; Table A3) by each bus’ peak load.

The normalized daily load shape factor and hourly real-time price are presented in Table A3 (Appendix A). The load shape factor is associated with the 24 h system load curve employed in BESS slot determination (see Figure 2); the nighttime minimum is 0.54 p.u., while two plateaus are observed at midday (0.98–1.00 p.u.) and in the evening (0.87–0.94 p.u.). The hourly real-time price ranges from $75.9/MWh to $163.4/MWh; its maximum occurs at 20:00. The hourly solar irradiance and wind speed data in Figure 1 are also given in Table A3. The five demand response participants, C1–C5, are located at nodes 9, 14, 22, 25, and 30, respectively. The solar irradiance and wind speed data (Figure 1, Table A3) represent a single typical summer day, with hourly average values taken from [79], and are used over the same 24 h horizon as the load and the real-time price. A typical day, rather than a full year or a worst-case year, is used because the study is a day-ahead scheduling problem.

3. Problem Formulation

This study addresses demand-side energy management through a coordinated framework in which the grid operator and the customer operator exchange information to jointly optimize their respective objectives, as formalized in Equation (8). The overall problem is structured as a multi-objective optimization scheme, wherein the grid operator, the customer operator, and the BESS operator each seek to minimize their individual costs while operating within the shared system constraints.

$$O{F}_1={\displaystyle \sum _{h=1}^{24}}{E}_p^h{P}_{\mathit{Grid}}^h\Delta t$$

(5)

$$O{F}_2={\displaystyle \sum _{h=1}^{24}}{\displaystyle \sum _{n=1}^{N_B}}{E}_p^h{P}_{\mathit{BESS},n}^h\Delta t$$

(6)

$$O{F}_3={\displaystyle \sum _{h=1}^{24}}{\displaystyle \sum _{i=1}^{N_c}}\left[{\alpha }_{1,i}{E}_p^h{L}_{d,i}^h\Delta t+{\alpha }_{2,i}{\beta }_i{\left(L_{a,i}^h-L_{d,i}^h\right)}^2\right]$$

(7)

OF = OF1 + OF2 + OF3

(8)

where $P_{\mathit{Grid}}^h$ is the power imported from the upstream grid in hour h (MW, Equation (9)), $P_{\mathit{BESS},n}^h$ is the power of BESS n in hour h (MW; positive when charging, negative when discharging) and N_B = 3.

The primary goal, as expressed in Equation (5), is to reduce the overall system expenditure. The secondary aim, outlined in Equation (6), focuses on lowering battery-related expenses. The final objective, detailed in Equation (7), aims to minimize the total per-customer cost while accounting for user comfort levels (in the scheduling stage, the discomfort term is implemented in the normalized form). These objectives are subject to the operational limitations of the system, which are listed below:

Equation (9) describes the power equilibrium condition, in which the electricity drawn from the grid must match the total load demand reduced by the combined output of all distributed generators. The charging and discharging rates of the BESS are determined by minimizing the overall costs incurred during these processes, subject to the following constraints. Equation (10) represents the maximum and minimum SOC ranges for each battery. Equations (11) and (12) give the intertemporal SOC update for charging and discharging, and Equation (13) specifies the terminal state-of-charge requirement enforced during training, ensuring that the reported BESS profit reflects genuine daily price arbitrage rather than one-directional depletion of stored energy.

$$P_{\mathit{Grid}}^h=P_d^h+P_{\mathit{loss}}^h-P_{\mathit{PV}}^h-P_W^h+{\displaystyle \sum _{n=1}^{N_B}}{P}_{\mathit{BESS},n}^h,\hspace{1em}h=1,\dots ,24$$

(9)

$${\mathit{SOC}}_{\mathit{min}}^n\le {\mathit{SOC}}^h\le {\mathit{SOC}}_{\mathit{max}}^n$$

(10)

$${\mathit{SOC}}_n^h={\mathit{SOC}}_n^{h-1}+{\displaystyle \frac{{\eta }_{\mathit{chg}}{P}_{\mathit{BESS},n}^h\Delta t}{B_n}},\hspace{1em}{P}_{\mathit{BESS},n}^h\ge 0$$

(11)

$${\mathit{SOC}}_n^h={\mathit{SOC}}_n^{h-1}+{\displaystyle \frac{P_{\mathit{BESS},n}^h\Delta t}{{\eta }_{\mathit{dchg}}{B}_n}},\hspace{1em}{P}_{\mathit{BESS},n}^h<0$$

(12)

$${\mathit{SOC}}_{24}^H\approx {\mathit{SOC}}_0^H$$

(13)

The range of charging power is expressed as Equation (14).

$$0\le {P_n}_{\mathit{BESS}/\mathit{chg}}^h\le {P_n}_{\mathit{chg}}^{\mathit{rated}}$$

(14)

where $P_d^h$ is the total demand after curtailment and $P_{\mathit{loss}}^h$ the network active power loss in hour h (kW), B_n is the energy capacity of BESS n (kWh, determined by RPO in Section 5.1) and η_chg = 0.9, and η_dchg = 0.9; because $P_{\mathit{BESS},n}^h$ is negative when discharging, the SOC decreases.

The range of discharging power is expressed as Equation (15).

$${{-P}_n}_{\mathit{dchg}}^{\mathit{rated}}\le {P_n}_{\mathit{BESS}/\mathit{dchg}}^h\le 0$$

(15)

The rated power of each battery for charging is obtained by Equation (16).

$$P_{n,\mathit{chg}}^{\mathit{rated}}={\displaystyle \frac{\left({\mathit{SOC}}_{\mathit{max}}^n-{\mathit{SOC}}_{\mathit{min}}^n\right)B_n}{{\eta }_{\mathit{chg}}\Delta t}}$$

(16)

The rated power of each battery for discharging is obtained by Equation (17).

$$P_{n,\mathit{dchg}}^{\mathit{rated}}={\displaystyle \frac{{\eta }_{\mathit{dchg}}\left({\mathit{SOC}}_{\mathit{max}}^n-{\mathit{SOC}}_{\mathit{min}}^n\right)B_n}{\Delta t}}$$

(17)

In this context, the symbols ${P_n}_{\mathit{chg}}^{\mathit{rated}},$ ${P_n}_{\mathit{dhg}}^{\mathit{rated}}$ denote the nominal charge and discharge power ratings, respectively. The quantities ${P_n}_{\mathit{BESS}/\mathit{chg}}^h$ and ${P_n}_{\mathit{BESS}/\mathit{dchg}}^h$ represent the charge and discharge power levels of each battery at hour h, while η_chg and η_dchg indicate the respective efficiency factors during the charge and discharge phases. The term ${B_n}_{\mathit{cap}}$ refers to the storage capacity of each battery unit.

An additional explicit constraint (Equation (18)) ensures that the curtailed power for each customer operator does not exceed the predetermined allowed value. In particular, the curtailed power for any individual customer should not exceed 40% of its load requirement:

$$0.60P_{\mathit{demand},i}^H\le P_i^H\le P_{\mathit{demand},i}^H$$

(18)

This 40% cap applies uniformly to a grid-operator-led initiative. In the customer-led case, an individual comfort-consistent cap c_i^max ≤ 0.40 (Table 3) is applied.

4. Proposed Methodology

The developed methodology consists of two hierarchical levels of planning/operation, as shown in Figure 3. The first level, dealing with the planning stage, considers the optimal sizing and positioning of the DG and BESS units at peak load. The second level, corresponding to the operational stage, employs the fixed infrastructure to perform hourly dispatch optimization, accounting for dynamic pricing, customer load shapes, and the charge/discharge cycles of the BESS units. The stages of the methodology are connected in the following way: DG buses and their sizes determined at the planning stage set the hourly PV/wind generation availability envelopes for the scheduling stage, whereas BESS buses, energy capacities, and power charge/discharge ratings determined at the planning stage through RPO set the action boundaries and SOC constraints for BESS scheduling agents. A single peak-load snapshot for site selection is a design simplification, often found in the two-stage planning/operation literature.

Figure 3. Two-Stage Optimization Framework for BESS Placement and Energy Management.

4.1. Optimal Placement and Sizing of DGs and BESS

Before implementing demand response, the baseline IEEE 33-bus test system is augmented with distributed generation (DG) and battery energy storage systems (BESS). The optimal sizing and placement of these units are determined using the Red Panda Optimization (RPO) algorithm. The optimization framework is formulated to concurrently minimize active/reactive power losses and nodal voltage deviations, as quantified by Equations (19), (20) and (21), respectively, through the strategic allocation and rating of DG units. Power flow at each candidate solution is evaluated using the backward/forward sweep method [83], applied to the IEEE 33-bus radial test system [85]. RPO decision vector co-decides the placement and sizing of the DG. All three BESS units simultaneously use one optimization problem in 10 dimensions: x = [bus_DG1,size_DG1,bus_DG2,size_DG2,bus_BESS1,cap_BESS1,bus_BESS2,cap_BESS2,bus_BESS3,cap_BESS3], where bus ∈ {2,…,33}, DG size ∈ [100, 1200] kW, and BESS energy capacity ∈ [100, 500] kWh. The constraint on the renewable-DG penetration rate (total DG capacity < 45% of the maximum system demand) is explicitly incorporated into the objective function via a bounded penalty term to ensure the resulting design is consistent with this constraint.

$$P_{\mathit{Loss}}={\displaystyle \sum _{j=1}^{N_{\mathit{br}}}}{\left(\left|I_j\right|\right)}^2{R}_j$$

(19)

$$Q_{\mathit{Loss}}={\displaystyle \sum _{j=1}^{N_{\mathit{br}}}}{\left(\left|I_j\right|\right)}^2{X}_j$$

(20)

$$\mathit{VD}={\displaystyle \sum _{k=1}^{N_{\mathit{bus}}}}{\left(V_k-V_{\mathit{ref}}\right)}^2$$

(21)

where I_j is the current of branch j (A), R_j and X_j its resistance and reactance (Ω), and N_br = 32 the number of branches; P_Loss is in kW and Q_Loss in kVAr, V_k is the voltage magnitude at bus k (p.u.), V_ref = 1.0 p.u. and N_bus = 33.

Red Panda Optimization (RPO) [86] is a metaheuristic that exploits the foraging (exploration) and tree-climbing (exploitation) behaviors of red pandas. The use of RPO in this paper is motivated by two considerations. First, since RPO has no algorithm-specific control parameters, its only user-defined inputs are the number of individuals in the population and the number of iterations [86]; in the case of PSO, the inertia weight and two acceleration coefficients must be determined. Second, in our experiments, RPO exhibited less run-to-run variation than PSO, which is an important property when determining the optimal planning design for the operational process. The framework itself does not depend on RPO; any metaheuristic capable of handling the mixed discrete–continuous decision vector can be substituted. The complete planning procedure is summarized in Algorithm 1, and the resulting design is passed to the scheduling stage described in Algorithm 2 and Section 4.2.

Algorithm 1. RPO-based joint siting and sizing of DG units and BESS (planning stage).

Input:       IEEE 33-bus network data [85] at peak load; bounds: bus bk ∈ {2, …, 33}, DG size PDG ∈ [100, 1200] kW, BESS capacity Bcap ∈ [100, 500] kWh; population N = 25; iterations T = 80. Output:    Best design Xbest = [busDG1, sizeDG1, busDG2, sizeDG2, busB1, capB1, …, busB3, capB3] (dimension m = 10) and BESS power ratings.

1:        Define the fitness $F\left(X\right)=w_1{P}_{\mathit{Loss}}+w_2{Q}_{\mathit{Loss}}+w_3\mathit{VD}+\rho \mathit{max}\left(0, \sum P_{\mathit{DG}}-0.45P_D^{\mathit{peak}}\right)$ using Equations (19)–(21); losses and voltages are obtained from the backward/forward sweep load flow [83]. 2:        for i = 1 to N do ▹ initialization 3:              xi,j = lbj + ri,j(ubj − lbj), j = 1, …, m, with ri,j ∼ U(0,1) 4:              Round bus entries to integers; run load flow and evaluate Fi = F(Xi) 5:        end for; set Xbest to the member with the lowest Fi 6:        for t = 1 to T do 7:              for i = 1 to N do 8:                      Phase 1—foraging (exploration) 9:                      PFSi = {Xk | Fk < Fi} ∪ {Xbest} 10:                    Select SFSi at random from PFSi; draw I ∈ {1,2} 11:                    $x_{i,j}^{P1}=x_{i,j}+r\left({\mathit{SFS}}_{i,j}-I·x_{i,j}\right)$ 12:                    Clip to [lbj, ubj], round bus entries, run load flow, and evaluate $F_i^{P1}$ 13:                    if $F_i^{P1}<F_i$ then $X_i\leftarrow X_i^{P1}$, $F_i\leftarrow F_i^{P1}$ end if 14:                    Phase 2—climbing trees (exploitation) 15:                    $x_{i,j}^{P2}=x_{i,j}+{\displaystyle \frac{{\mathit{lb}}_j+r\left({\mathit{ub}}_j-{\mathit{lb}}_j\right)}{t}}$ 16:                    Clip to [lbj, ubj], round bus entries, run load flow, and evaluate $F_i^{P2}$ 17:                    if $F_i^{P2}<F_i$ then $X_i\leftarrow X_i^{P2}$, $F_i\leftarrow F_i^{P2}$ end if 18:              end for 19:              Update Xbest 20:        end for 21:        Compute the BESS charge/discharge ratings ${P_n}_{\mathit{chg}}^{\mathit{rated}}$ and ${P_n}_{\mathit{dchg}}^{\mathit{rated}}$ from Equations (16) and (17) 22:        return Xbest and the BESS ratings (passed to Algorithm 2)

Algorithm 2. Multi-agent Q-learning scheduling of customer load and BESS (operational stage).

Input:       Hourly price $E_p^h$, reference demand $L_{\mathit{an},i}^h$, PV/wind output (Equations (1) and (2)); BESS buses, Bn,cap, ratings and SOC limits from Algorithm 1 and Table 2; α1,i, α2,i, βi (Table 3); θ, γ, ε0, E (Table A1); regime ∈ {Case 1, Case 2}. Output:       Greedy hourly curtailment of each customer, charge/discharge schedule of each BESS, and daily grid demand, loss and cost.

1:         if Case 1 then $c_i^{\mathit{max}}=0.40{\left({\displaystyle \frac{{\mathit{CI}}_{\mathit{min}}}{{\mathit{CI}}_i}}\right)}^6$ with ${\mathit{CI}}_i={\displaystyle \frac{{\alpha }_{2,i}{\beta }_i}{{\alpha }_{1,i}}}$ else $c_i^{\mathit{max}}=0.40$ end if 2:         Initialize Qi(s, a) ← 0 for every agent, state and action 3:         for episode e = 1 to E do ▹ customers: E = 500; BESS: E = 3000 4:               ${\epsilon }_e={\epsilon }_0\left(1-{\displaystyle \frac{e-1}{E}}\right)$; reset ${\mathit{SOC}}_n^0$ to the initial values (Table A1) 5:               for h = 1 to 24 do 6:                      Customer agents i = 1, …, 5 (state s = h) 7:                            Choose a ∈ {0, 1, …, 10} at random with probability εe, otherwise a = arg max Qi(h, a) 8:                            $x_i^h={\displaystyle \frac{a}{10}}{c}_i^{\mathit{max}}$; $\Delta L_i^h=x_i^h{L}_{\mathit{an},i}^h$ 9:                            $r_i^h={\alpha }_{1,i}{E}_p^h\Delta L_i^h-{\alpha }_{2,i}{\beta }_i{K}_D{\left(x_i^h\right)}^6$ 10:                            if Case 2 then $r_i^h\leftarrow r_i^h+0.9E_p^h\Delta L_i^h$ end if 11:                     BESS agents n = 1,2,3 (state $s=\left(h, b({\mathit{SOC}}_n^{h-1})\right)$, 10 SOC bins) 12:                            Choose k ∈ {−4, …, 4} ε-greedily; $P_n^h={\displaystyle \frac{k}{4}}{P_n}_{\mathit{chg}}^{\mathit{rated}}$ if k ≥ 0, else $P_n^h={\displaystyle \frac{k}{4}}{P_n}_{\mathit{dchg}}^{\mathit{rated}}$ 13:                            Limit $P_n^h$ so that ${\mathit{SOC}}_{\mathit{min}}^n\le {\mathit{SOC}}_n^h\le {\mathit{SOC}}_{\mathit{max}}^n$; update ${\mathit{SOC}}_n^h$ with Equations (11) and (12) 14:                            $r_n^h=-E_p^h{P}_n^h$ 15:                            if Case 2 and h ∈ {18, …, 21} then $r_n^h\leftarrow r_n^h+0.5E_p^h\left(P_{n,\mathit{dchg}}^h-P_{n,\mathit{chg}}^h\right)$ end if 16:                            if h = 24 then $r_n^h\leftarrow r_n^h-\kappa |{\mathit{SOC}}_n^{24}-{\mathit{SOC}}_n^0|B_{n,\mathit{cap}}$, κ = $ 300/MWh ▹ Equation (13) end if 17:                     Learning update for every agent (Equation (22)), with max Q(s′, a′) = 0 when h = 24: 18:                            Q(s, a) ← Q(s, a) + θ[rh + γ max Q(s′, a′) − Q(s, a)] 19:               end for 20:         end for 21:         Extract greedy policies a∗(s) = arg max Q(s, a) and the resulting 24 h schedules 22:         Run the load flow for each hour; compute $P_{\mathit{Grid}}^h$ (Equation (9)), losses and procurement cost (Equation (5)); repeat for 5 random seeds and report the mean 23:         return schedules, $\sum P_{\mathit{Grid}}^h$, daily energy loss and cost

4.2. Scheduling of Load Using the Q-Learning Algorithm

The demand response problem is first formulated as a discrete finite-horizon Markov Decision Process (MDP). Then, the Q-learning algorithm is used to develop an efficient demand-response scheduling scheme under real-time pricing. In this study, the demand response framework is realized through two categories of autonomous agents: customer operators and BESS agents, as shown in Figure 4.

Figure 4. Multi-agent Q-learning architecture for the scheduling of customer load and BESS. The orange dashed arrows denote the incentive (shadow-price) signal broadcast by the grid operator, which applies only in Case 2.

Specifically, five customer agents are deployed at buses 9, 14, 22, 25, and 30, while three BESS agents are positioned at buses 17, 10, and 30. The underlying demand response problem is formulated as a Markov Decision Process (MDP), characterized by the canonical quadruple: state space, action space, reward function, and transition probability function.

For customer agents, the state is the hour index h ∈ {1, …, 24}, corresponding to the hourly price E_p^h (24 states). For BESS agents, the state is the pair (h, SOC bin), with the SOC range of each battery divided into 10 equal bins (240 states). Because the study is a deterministic day-ahead problem, the load level and the PV and wind outputs at each hour are fixed functions of h, and the current SOC summarizes the full charge/discharge history, since the SOC update of Equations (11) and (12) depends only on the present SOC and action.

Appendix A (Table A4) confirms that adding load-level and price-deviation variables to the state leaves the learned policies unchanged, whereas a state built from three price intervals alone is insufficient for the BESS agents. Under random price and load deviations, adding price-deviation information reduces the BESS optimality gap from 38.8% to 22.8%, motivating deep reinforcement learning.

The action defines the agent’s decision in a given state to maximize cumulative reward; here, the action corresponds to the dispatched power demand. Since the Q-learning algorithm operates on discrete action spaces, the continuous power range is quantized into discrete levels indexed from 1 to N. Finally, the reward signal incentivizes cost-minimizing behavior to minimize total operational cost; the state, action, and reward sets are summarized in Table 4 and Table 5, and the case-specific definitions are given in Table 6.

Table 4. Set of states, actions, and rewards for each customer agent.

Agent	Bus	State Set	Action Set (Case 2, Uniform)	Reward
C1	9	h ∈ {1,…, 24} (hourly price $E_p^h$)	0.60 Pdem at bus 9 to Pdem at bus 9	Equation (7)
C2	14		0.60 Pdem at bus 14 to Pdem at bus 14	Equation (7)
C3	22		0.60 Pdem at bus 22 to Pdem at bus 22	Equation (7)
C4	25		0.60 Pdem at bus 25 to Pdem at bus 25	Equation (7)
C5	30		0.60 Pdem at bus 30 to Pdem at bus 30	Equation (7)

Table 5. Set of states, actions, and rewards for each BESS agent.

Agent	State Set	Action Set	Reward
BESS 1	(h, SOC bin), h ∈ {1,…, 24}	${{-P}_n}_{\mathit{dhg}}^{\mathit{rated}}$ to ${P_n}_{\mathit{chg}}^{\mathit{rated}}$	-Equation (6)
BESS 2		${{-P}_n}_{\mathit{dhg}}^{\mathit{rated}}$ to ${P_n}_{\mathit{chg}}^{\mathit{rated}}$	-Equation (6)
BESS 3		${{-P}_n}_{\mathit{dhg}}^{\mathit{rated}}$ to ${P_n}_{\mathit{chg}}^{\mathit{rated}}$	-Equation (6)

Table 6. Explicit state/action/reward/constraints by case.

	Case 1: Customer-Driven	Case 2: Grid-Operator-Driven
Customer state	Hour h (1–24)	Hour h (1–24)
Customer action bound	$c_i^{\mathit{max}}=0.40{\left({\displaystyle \frac{{\mathit{CI}}_{\mathit{min}}}{{\mathit{CI}}_i}}\right)}^6$, where ${\mathit{CI}}_i={\displaystyle \frac{{\alpha }_{2,i}{\beta }_i}{{\alpha }_{1,i}}}$ individual, comfort-consistent ceiling ≤ 40%	Uniform 40% ceiling for every customer, independent of individual comfort preference
Customer reward	Ui = α1Price × Saving − α2βKD(curtailed fraction)6	As Case 1, plus an explicit grid-operator shadow-price incentive term 0.9·price· (curtailed energy), reflecting a centrally broadcast system signal
BESS state	(h, SOC bin)	(h, SOC bin)
BESS action	$P_i^{\mathit{BESS}}\in \left[-P_{\mathit{dchg}}^{\mathit{rated}},P_{\mathit{chg}}^{\mathit{rated}}\right]$	As Case 1
BESS reward	min − Price.P (profit-seeking arbitrage), with terminal constraint SOC (24) ≈ SOC (0)	As Case 1, plus a system-support term 0.5 × price × (discharged − charged energy) in the four highest-price hours (18:00–21:00)
Information exchange	Each agent observes only price (and, for BESS, its own SOC); no inter-agent communication.	Each agent additionally receives the grid operator’s shadow-price/incentive signal; learning remains decentralized (each agent still runs its own Q-learning update), and the operator does not directly compute or dictate actions.
Training	Tabular Q-learning; Q-tables initialized to zero; customer agents: θ = 0.3, γ = 0, ε0 = 0.4, 500 episodes; BESS agents: θ = 0.3, γ = 1.0, ε0 = 1.0, 3000 episodes, 10 SOC bins; ε decays linearly to 0; 1 simulated day per episode (Appendix A, Table A1)	Same as Case 1

In Q-learning [87], the action value function is updated at each iteration [75], expressed as:

Q(s_h, a_h) ← Q(s_h, a_h) + θ[r_h + γ max Q(s_h+1, a) − Q(s_h, a_h)]

(22)

where θ refers to the learning rate, while γ is the discount factor.

In terms of action selection, an ε-greedy policy is employed with a linear decay of ε₀ to zero, and all Q-tables start with zero. The choice of parameter values (see Appendix A, Table A1) resulted from a sensitivity analysis using a one-at-a-time approach, in which each learned policy is compared with the exact optimum of the same discrete problem (Table A2). Customer agents have γ = 0 because the curtailment decision in hour h does not influence the next state in hour h + 1; however, increasing γ will not change the optimal policy, only slow learning. For BESS agents, γ = 1.0 is chosen because the profit from nighttime charging is realized only at the evening price peak; using γ ≤ 0.95 prevents BESS agents from learning the arbitrage scheme. Fully exploring actions is required at the beginning (ε₀ = 1.0) for BESS agents to find out feasible charge–discharge cycles. The training budgets of 500 and 3000 episodes exceed the episodes at which the greedy policies stabilize (45–455 and 1950–2610; see Section 5.4), and the final gaps to the optimum are 0.45% and 0.11%, respectively.

5. Discussion

To assess the multifaceted benefits of distributed resources, we use a 33-bus test distribution system with five responsive loads at buses 9, 14, 22, 25, and 30, and the hourly real-time price series listed in Appendix A, Table A3. The network also includes two renewable-based DG units and three BESS, each characterized by a unique initial state of charge. The simulation is structured as a two-tiered process: the first stage optimizes the siting and rating of the DG and BESS. In contrast, the second stage executes a coordinated load and storage management strategy based on volatile real-time prices. The overarching goal is to systematically examine how the integration of DG, BESS, and price-based demand response collectively influences network power losses, bus voltage deviations, and the system’s total cost.

5.1. Installation Results

The optimal placement and sizing of the distributed generators are determined using the RPO algorithm under peak-load conditions. To benchmark the efficacy of this approach, three key performance indicators are evaluated: real power loss, minimum bus voltage, and DG penetration level. The results, summarized in Table 7, are obtained from the best of 20 independent RPO runs (population 25, 80 iterations) and are compared against the optimized configuration reported in [66].

Table 7. Performance comparison and optimal capacity and location of DG.

Parameters	Base Case	[66]	RPO (This Work)
Bus location	–	5, 14	14, 31
DG size	–	1624.85 kW, 442.57 kW	638.3 kW, 1027.6 kW
Real power loss	0.2027 MW	0.1322 MW	0.0715 MW
Reactive power loss	0.1351 MVAr	not reported	0.0496 MVAr
Minimum bus voltage	0.9131 at bus 18	0.9443 at bus 18	0.9790 at bus 25
% DG penetration	–	52%	44.8%

The optimal location and capacity of BESS are expressed in Table 8.

Table 8. Optimal location and capacity of BESS.

	BESS 1	BESS 2	BESS 3
Bus number	17	10	30
Energy capacity	0.500 MWh	0.362 MWh	0.443 MWh
SOC range	0.30–0.90	0.20–0.90	0.30–0.80
Charging power rating (Equation (16))	333.3 kW	281.8 kW	246.2 kW
Discharging power rating (Equation (17))	270.0 kW	228.3 kW	199.4 kW

The three BESS units generate very similar arbitrage revenues in both cases: $15.91, $13.44, and $11.75 per day in Case 1, and $15.91, $13.36, and $11.75 per day in Case 2. Each BESS unit charges during low-price hours (approximately 04:00 and 24:00) and discharges at its full rated power (270.0, 228.3, and 199.4 kW, respectively; see Table 8) at 20:00, when the evening price peak occurs, returning to its initial state of charge by the end of the day. In Case 2, the system-support incentive provided by the grid operator (see Table 6) adds $22.06, $18.63, and $16.29 per day, raising the total BESS income to $37.97, $31.99, and $28.03 per day. Because the evening price peak already lies inside the system peak window, the incentive increases BESS income without materially changing the schedule.

5.2. Scheduling Results

To validate the efficacy of the proposed Q-learning-based load scheduling framework, which operates on real-time pricing signals, we present two case studies. Both cases use the same decentralized multi-agent Q-learning framework: each customer and BESS agent maintains its own Q-table and selects its hourly action. The two cases differ only in what each agent is told and what it is rewarded for (Table 6):

• Case 1-Customer-driven (no coordination signal):

Information signal: The hourly real-time price only (plus own SOC for BESS agents). Reward: customer agents minimize their own electricity cost plus discomfort cost (Equation (7)); BESS agents maximize their own arbitrage profit (Equation (6)).

Decision logic: each customer may curtail up to an individual comfort-consistent ceiling c_i^max (40.0%, 19.7%, 15.9%, 3.5%, and 1.7% for C1–C5), so curtailment follows customer preference.

• Case 2-Grid-operator incentive (coordination signal):

Information signal: the hourly price plus a shadow-price incentive broadcast by the grid operator.

Reward: the Case 1 reward plus an incentive of 0.9 × price × curtailed energy for customers, and, for BESS agents, a system-support term that rewards discharge (and penalizes charging) in the system peak-price hours.

Decision logic: all customers share a uniform 40% curtailment ceiling, so curtailment follows the system’s cost-and-loss objective rather than individual comfort. The operator never computes or dictates actions; it only shapes rewards and action bounds.

The customers’ load curtailment is shown in Figure 5. In Case 1, the load curtailment is highly uneven and is based on each customer’s comfort-consistent limit (Table 6). The largest load curtailment occurs for C1, which equals 0.387 MWh/day (33.7% of the daily reference demand at bus 9); the smallest load curtailment happens for C5 with 0.066 MWh/day (1.7% at bus 30). In Case 2, the load curtailment is approximately uniform, at 39.0–40.0% of the daily reference demand for all customers.

Figure 5. Comparison of power curtailment and percentage reduction in load demand.

Figure 6a clearly indicates that each BESS agent has learned one arbitrage cycle per day, where the battery charges during the time intervals when prices are lowest, discharges at full-rated power during the 20:00 peak and fully recharges until reaching the same initial state of charge (SOC), according to Equation (13). Figure 6b shows that the arbitrage profit is almost identical in the two cases, whereas in Case 2 the system-support incentive more than doubles the total income of every unit.

Figure 6. (a) Learned hourly schedule of the three BESS units in Case 1 (the Case 2 schedule differs only in a split of the night-time charging of BESS 1 and 2); (b) daily profit of each BESS agent in both cases.

5.3. Overall Performance of the System

The efficacy of the proposed approach is assessed via four key indices: system power loss, aggregate grid power demand, per-bus voltage deviation, and total electricity procurement cost. The first three metrics characterize technical performance, while the latter reflects economic viability.

All percentages are relative to the DG-only case. Reduction from DR = (DG only – DG + DR)/DG only; change from BESS (pp, percentage points) = (DG + DR – DG + DR + BESS)/DG only, where a negative value is an increase; net reduction = the sum of the two. Case 2 values are means of five training seeds (standard deviation ≤ 0.02 MWh/day for demand and ≤$2.7/day for cost); Case 1 results are identical for all seeds.

Table 9 demonstrates the contribution of customer load curtailment (DG + DR vs. DG) and BESS schedule (DG + DR + BESS vs. DG + DR). For both cases, curtailment contributes to the entire reduction in demand, loss, and most of the cost reduction: 2.65% of demand, 4.90% of loss, and 2.63% of cost in Case 1, and 12.50% of demand, 28.15% of loss, and 12.37% of cost in Case 2. The BESS scheduling then reduces the procurement cost by an additional 0.62% points ($44.52/day) in Case 1 and 0.61% points ($43.99/day) in Case 2. This is due to charging during low-priced hours and discharging during the evening peak price, which also moves part of the network losses from expensive peak hours to cheaper night hours. Meanwhile, it increases the daily demand by 0.29% points and the loss by 0.37 and 0.36% points in each case. The net reductions are 2.36%, 4.53%, and 3.25% in Case 1 and 12.21%, 27.79%, and 12.99% in Case 2.

Table 9. Daily grid demand, power loss, and procurement cost, and the share of each reduction due to DR and BESS (Case 1: customer-driven; Case 2: grid-operator incentive).

Case	Quantity	DG Only	DG + DR	DG + DR + BESS	Reduction from DR (%)	Change from BESS (pp)	Net Reduction (%)
Case 1	Grid demand (MWh/day)	59.2644	57.6927	57.8648	2.65	−0.29	2.36
Case 1	Power loss (MWh/day)	2.2782	2.1665	2.1750	4.90	−0.37	4.53
Case 1	Procurement cost ($/day)	7176.33	6987.36	6942.84	2.63	+0.62	3.25
Case 2	Grid demand (MWh/day)	59.2644	51.8541	52.0261	12.50	−0.29	12.21
Case 2	Power loss (MWh/day)	2.2782	1.6369	1.6452	28.15	−0.36	27.79
Case 2	Procurement cost ($/day)	7176.33	6288.40	6244.42	12.37	+0.61	12.99

For comparison, applying the rule-based pattern of Equation (3) directly as the operating policy yields only $2.45 per day of total arbitrage profit, leaves the final SOC away from its initial value, and slightly increases the procurement cost (+$3.21 per day relative to DG + DR), whereas the Q-learning agents earn $41.10 per day and reduce the cost by $44.52 per day (Appendix A, Table A5).

5.4. Effectiveness of the RPO for Installing a Q Algorithm or Scheduling Purposes

To perform analysis on the performance of the RPO, we conduct a comparison with PSO, where we consider the same population size (25) and number of iterations (80) for both algorithms, parameters w = 0.7, c₁ = c₂ = 1.5 for PSO, and 20 runs of both algorithms with different random seeds. The results obtained by RPO give the best objective function value of 0.1205 (mean 0.1296, standard deviation 0.0053 for the 20 runs), while those obtained by PSO give a best objective function value of 0.1133 (mean 0.1197, standard deviation 0.0066). In general, RPO enters the 1% of the final value band at an average of 60.0 out of 80 iterations, while PSO does so after 40.9 out of 80 iterations. In this fair comparison, PSO achieves slightly better best and mean objective values and converges faster, whereas RPO shows about 20% lower run-to-run standard deviation and requires no parameter tuning. The two algorithms, therefore, yield comparable planning designs, indicating that RPO is retained for its practical properties (no algorithm-specific control parameters and lower run-to-run spread), not for superior accuracy; any metaheuristic that handles the mixed discrete–continuous decision vector can be used in the planning stage. Table 10 presents RPO and PSO planning-stage statistics, and Figure 7 shows the performance of RPO and PSO algorithms.

Table 10. RPO vs. PSO planning-stage statistics over 20 independent runs (population 25, 80 iterations each).

Algorithm	Best F	Mean F	Std. Dev. F	Avg. Iteration to 1% Band (of 80)
RPO	0.1205	0.1296	0.0053	60.0
PSO	0.1133	0.1197	0.0066	40.9

Figure 7. Convergence plot of the RPO and PSO algorithms.

The cost of discomfort for each consumer is illustrated in Figure 8 and Figure 9. In Case 1 (Figure 8), the discomfort is experienced by C1 ($4.32/day), the least comfort-oriented customer; C2 and C3 have a much smaller daily cost of $0.15 and $0.03, respectively, whereas the cost of discomfort for C4 and C5 is virtually nil. In Case 2 (Figure 9), the uniform cap of 40% increases the cost of discomfort for all consumers to $3.67–$10.41 per day.

Figure 8. Penalty due to discomfort in case 1.

Figure 9. Penalty due to discomfort in case 2.

Figure 10 clearly indicates that the reported policies have been achieved through converged Q-Learning. The daily reward of the greedy policy for each agent, calculated without exploration after every 5 (customers) and 10 (BESS) training episodes, reaches a stable level from episodes 45 to 455 out of 500 for the customer agents and from episodes 1950 to 2610 out of 3000 for the BESS agents, over five random seeds. At convergence, the converged policies remain only 0.45% (customers) and 0.11% (BESS) off the optimal solution of the discrete problem.

Figure 10. Q-learning training convergence: daily reward of the greedy policy versus training episodes for representative customer agents (a) and all three BESS agents (b), in both cases.

5.5. Techno-Economic Assessment of the BESS

Table 9 shows that BESS scheduling lowers the daily procurement cost by a further 0.62 percentage points ($44.52 per day) in Case 1, while slightly increasing demand and losses because of round-trip losses. To test whether this benefit justifies the investment, a life-cycle assessment was conducted for the three BESS units in Table 8 (1305.4 kWh and 697.7 kW in total). The capital cost of utility-scale lithium-ion storage was taken as $334/kWh (2024 value) and, as an optimistic case, $147/kWh (2035 low projection), with fixed O&M equal to 4% of the $/kW cost of a 4 h system per year and a 15-year lifetime at one full cycle per day, all following [88]; a discount rate of 8% was assumed. The learned schedules give one equivalent full cycle per day (254.7 MWh discharged per year), so the cycle-related degradation is covered by the 15-year lifetime.

The results are given in Table 11. From the system perspective (Case 1, $16,250 per year), the benefit–cost ratio is 0.18 at the 2024 capital cost and 0.42 at the 2035 low projection, and the net present value is negative in both cases. Even for the BESS owner in Case 2, whose income includes the grid operator’s incentive ($35,766 per year), the ratio stays below one (0.41 and 0.92). The break-even capital cost is $61.5/kWh for the system benefit and $135.4/kWh for the owner in Case 2. Equivalently, spreading the 2024 capital cost over 5475 cycles gives about $80 per day, which exceeds the daily cost reduction of $44.52. Price arbitrage alone, therefore, does not recoup the BESS investment under the price spread used in this study ($75.9–163.4 /MWh). The BESS sizes in this work were selected for technical objectives (loss and voltage), and their economic case depends on additional services that are not valued here, such as peak-capacity deferral, voltage support, reserve provision, and resilience, or on a larger price spread.

Table 11. Techno-economic assessment of the three BESS units (15-year life, 8% discount rate, cost data from [88]).

Item	Capital Cost $334/kWh (2024)	Capital Cost $147/kWh (2035, Low)
Installed energy/discharge power	1305.4 kWh/697.7 kW	1305.4 kWh/697.7 kW
Capital cost ($)	436,004	191,894
Fixed O&M ($/year)	37,285	16,410
Annualized cost, capital + O&M ($/year)	88,223	38,829
System benefit, Case 1 ($/year)	16,250	16,250
Benefit–cost ratio/NPV ($), Case 1	0.18/−616,055	0.42/−193,264
Owner income incl. incentive, Case 2 ($/year)	35,766	35,766
Benefit–cost ratio/NPV ($), Case 2 owner	0.41/−449,003	0.92/−26,213
Break-even capital cost ($/kWh)	61.5 (Case 1); 135.4 (Case 2 owner)	—

—: not repeated; the break-even capital cost does not depend on the assumed capital cost.

5.6. Robustness to Forecast Uncertainty

Two checks were conducted to validate the deterministic day-ahead results. First, the peak-load design of Table 7 and Table 8 was evaluated over all 24 h: with the DG units alone, the network loss is reduced in every hour (by 0 to 54.7%), and the daily loss falls from 3.167 to 2.278 MWh (28.1%), so the design obtained from the peak snapshot does not cause higher losses at off-peak hours.

Second, the day-ahead schedules of both regimes were kept fixed and evaluated under input deviations. In the sensitivity cases, the PV and wind outputs were changed by ±20%, the prices by ±20%, and the loads by ±10% for the whole day (for price and load changes, the agents were retrained on the changed data). In the Monte Carlo test, 300 random days were generated in which the hourly PV and wind outputs (±20%), loads (±10%), and prices (±20%) deviated independently and uniformly from the forecast, while the schedules learned from the forecast were unchanged. For this test, the agent policies were represented by the exact optimum of the same discrete problems, which the learned policies match within 0.5% (Table A2); the nominal values, therefore, differ from those in Table 9 by at most 0.08 percentage points.

Table 12 shows that the reductions are stable. Across the 300 random days, the procurement-cost reduction ranges from 2.91% to 3.61% in Case 1 and from 12.65% to 13.41% in Case 2 (5th–95th percentiles), and all reductions remain within ±0.5 percentage points of their nominal values. The additional cost reduction from BESS scheduling was positive on all 300 days (mean $44.7 per day, 5th–95th percentiles $19.5–$70.0 per day). The relative advantage of the incentive-coordinated regime is therefore not an artifact of the deterministic data.

Table 12. Reduction relative to the DG-only case (%) under input deviations; schedules learned on the day-ahead data.

Scenario	Case 1 Demand	Case 1 Loss	Case 1 Cost	Case 2 Demand	Case 2 Loss	Case 2 Cost
Nominal (day-ahead data)	2.36	4.53	3.25	12.25	27.87	13.03
PV and wind − 20%	2.26	4.55	3.12	11.72	27.78	12.50
PV and wind + 20%	2.47	4.43	3.40	12.83	27.54	13.60
Price − 20%	2.31	4.43	3.20	12.22	27.84	13.00
Price +20%	2.37	4.53	3.26	12.26	27.88	13.03
Load − 10%	2.34	4.17	3.35	12.45	27.39	13.32
Load + 10%	2.36	4.74	3.16	12.11	28.17	12.81
300 random days: mean [5th–95th pct.]	2.36 [2.34–2.39]	4.51 [4.09–4.94]	3.26 [2.91–3.61]	12.26 [12.14–12.37]	27.81 [27.40–28.21]	13.03 [12.65–13.41]

6. Conclusions

This study presents a comprehensive framework to improve the operational efficiency of a grid-connected microgrid by coordinating the integration of distributed generation (DG), battery energy storage systems (BESS), and a price-based demand response (DR) program. A two-stage methodology is proposed: in the first stage, the optimal sizing and placement of DG units and BESS are determined using a multi-objective Red Panda Optimization (RPO) algorithm. In contrast, the second stage employs real-time energy management via a multi-agent Q-learning-based reinforcement learning approach. The framework is validated on the IEEE 33-bus test system, with results demonstrating notable performance enhancements, including reduced power losses, improved voltage profiles, and lower operational costs. The optimal allocation of DG and BESS yields a substantial decrease in real power losses and an elevation in the minimum bus voltage, confirming the efficacy of the RPO-based planning phase. Additionally, the integration of BESS enhances system flexibility through effective energy storage and peak-load management.

At the operational stage, the multi-agent reinforcement learning framework facilitates coordinated decision-making among customer operators and BESS agents under time-varying pricing conditions. The proposed DR strategy effectively reconciles the competing objectives of minimizing electricity costs and preserving user comfort. The comparison of the two incentive regimes shows that the same decentralized agents behave very differently once a grid operator’s incentive is broadcast: the incentive-coordinated regime (Case 2) yields greater reductions in cost, demand, and loss, whereas the price-only regime (Case 1) better respects individual customers’ comfort. The incentive design, rather than a change in control architecture, is therefore the main lever available to the operator. Overall, the system attains significant improvements in power loss reduction, peak demand shaving, and grid energy procurement cost. Of these improvements, customer load curtailment provides all of the demand and loss reduction and most of the cost reduction (2.63% of cost in Case 1 and 12.37% in Case 2), whereas BESS arbitrage contributes a further 0.62 and 0.61 percentage points of cost reduction at the expense of slightly higher demand and losses due to round-trip efficiency.

The findings verify the capacity of learning-based scheduling to harness the advantages of both demand response and storage in a converged, decentralized way, with the agents, on average, within 0.5% of the optimum for their discrete problems. However, there are four limitations. Firstly, the scheduling is deterministic day-ahead: the reductions remain stable when renewable outputs, loads, and prices deviate randomly from the forecast (Table 12), but tabular agents cannot exploit uncertainty-aware state information (Table A4). Secondly, the scalability of tabular Q-learning decreases with increasing numbers of states and agents. Thirdly, the dissatisfaction coefficients β are scenario parameters, and the findings depend on their relative values (Table A6). Fourthly, the BESS is justified here by technical benefits: at current costs, price arbitrage alone recovers only 18–42% of its life-cycle cost (Table 11), so additional value streams must be included for an investment decision. Future work will extend the framework to deep reinforcement learning methods (e.g., TD3 and SAC), stochastic multi-period planning with seasonal data, estimation of β from measured customer behavior, and valuation of BESS grid services.