A Critical Review of Reinforcement Learning for Optimal Coordination and Control of Modern Power Systems Under Uncertainties

Abstract

The increasing penetration of distributed energy resources (DERs), electric vehicles (EVs), dynamic line ratings (DLRs), and flexible loads is reshaping modern power systems while introducing significant operational uncertainties. Reinforcement learning (RL) has gained attention as a data-driven solution for optimal coordination and control under uncertainty. However, existing studies that used RL for optimal coordination reviewed in this research primarily address uncertainties from DERs and load variability, largely neglecting DLRs and EVs as a time-varying network constraint. Moreover, long training times and limited interpretability hinder the practical deployment of RL-based controllers. This paper presents a comprehensive review of RL applications in power system operational control, categorizing approaches based on uncertainty sources, control objectives, and learning architectures. The review highlights the operational advantages of incorporating DLR uncertainty, including improved line utilization, congestion mitigation, enhanced renewable hosting capacity, and increased system flexibility. A critical research gap is identified in the absence of integrated RL frameworks that jointly consider DLRs and learning efficiency. To address this gap, a future research direction integrating a Belief–Desire–Intention (BDI) framework within RL is proposed, enabling faster convergence, constraint-aware decision-making, improved transparency, and enhanced resilience in modern power system coordination and control.

1. Introduction

The adoption of the Paris Agreement has obligated member states of the United Nations to reduce reliance on fossil fuel-based power generation in order to mitigate carbon emissions that pose significant environmental and health risks [1,2]. This global commitment has accelerated the transition toward low-carbon power systems, driving widespread integration of renewable energy-based distributed energy resources (DERs), electric vehicles (EVs), power–electronic interfaced devices, flexible loads, and dynamic line rating (DLR) technologies [3,4,5,6,7,8,9] into power system networks as described in Figure 1.

Figure 1. Overview of the uncertainties considered in power systems.

While these developments improve sustainability and operational flexibility, they introduce substantial uncertainties and nonlinear dynamics into power system operation, thereby increasing system complexity and posing new challenges for planning and control [6,10]. Traditional physics-based models face limitations in accurately capturing the stochastic and time-varying behavior associated with renewable generation, EV charging/discharging, dynamic loading, and DLRs [7,8]. Meanwhile, the increasing deployment of sensors, smart meters, and advanced monitoring infrastructure has enabled the collection of large volumes of high-resolution data, creating opportunities to extract actionable intelligence using data-driven methods. Machine learning (ML) techniques offer a complementary approach by learning system behavior directly from data and enhancing operational decision making under uncertainty [11,12]. Among ML approaches, reinforcement learning (RL) has gained particular attention in operational control due to its ability to learn optimal control policies through interaction with the environment and reward-based feedback [13,14]. However, the application of RL for coordinated control of power system devices under uncertainty remains an emerging research area, with challenges related to safety, interpretability, scalability, and constraint handling. There is a need to analyze the existing application of RL in optimal coordination of modern power systems that integrates several uncertainties and addresses the aforementioned problems facing the effective application in modern power systems. More so, the review considered the uncertainties in DERs, loading, and EVs without considering DLRs, which tends to increase the hosting transmission capacity of the wind and PV energy. This review critically discusses the influence of uncertainties arising from DERs, EVs, DLRs, and dynamic load behavior on power system operation, surveys RL-based control strategies, identifies their limitations, and outlines future research directions for safe and adaptive reinforcement of learning-based power system control under uncertainties of DERs, EVs, and DLRs. The databases considered for this study include Scopus, Google Scholar, and IEEEXplore. The search was conducted using keywords and phrases such as ‘reinforcement learning’, ‘optimal coordination and control’, ‘electric vehicles’, ‘distributed energy resources’, ‘dynamic line rating’, and ‘intelligent control in power systems’.

The main contributions of the research are as follows:

• We discussed uncertainties caused by DERs based on wind and PV energy, dynamic line rating, EVs’ charging/discharging behavior, and dynamic load profiles, and how they influence power flow and voltage regulation.
• We also explore various RL algorithms and their applications in operational control, demonstrating their suitability for adaptive decision making in power systems. We also analyzed different RL-based methods used in the literature for optimizing operational control in power systems.
• The study provides the limitations of the classical RL framework in power system operational control and suggests a possible need for a constraint-based reward function within the RL framework to ensure safe operational control.
• A future research framework is presented, stating how BDI-RL can be effectively adopted for coordinating and controlling voltage regulating devices under uncertainties to enhance grid resilience, operational efficiency, and safety.

The remaining section of the research is discussed under the following subheading: Section 2, modeling of uncertainty in modern power systems; Section 3, reinforcement learning architecture and agents; in Section 4, a review of reinforcement learning-based control strategies for power systems under uncertainty is discussed; Section 5, existing challenges of reinforcement learning agents and the Belief–Desire–Intention (BDI-RL) framework as a future research direction; Section 6, practical feasibility of BDI-RL; and Section 7, summarizes the findings of the review in the conclusion.

2. Modeling of Uncertainty in Modern Power Systems

The architecture of reinforcement learning consists of three major things: the environments, the actions, and the agents. One of the agents’ functions is to understand the environment’s operation. Since the power system is subject to various uncertainties, it is best to discuss the uncertainties in the power system and their effects on the dynamics of environmental parameters, such as voltage and losses. This will allow the agents to learn the uncertainties and initiate the appropriate actions that will be safe for the environment. Modeling uncertainties in power system networks is essential for ensuring reliable and efficient operation in modern grids [15,16,17]. Accurate uncertainty modeling enables control agents to better understand system dynamics and make informed decisions that enhance grid resilience. This section presents modeling approaches for wind and PV-based DERs and dynamic line rating (DLR) using physical models, while EV charging behavior and load dynamics are represented using behavioral models [18,19,20,21], as illustrated in Figure 2.

Figure 2. Uncertainties classification.

2.1. Wind Energy Generator

The turbine output is modeled as a function of wind speed, defined by cut-in, rated, and cut-out thresholds, capturing zero, increasing, and rated power operating regions, as expressed in (1) [22,23].

$$P_w=\left\{\begin{array}{c}0 0\le W_{sm}\le W_{si}\\ X+Y{W}_{sm}+Z{W_{sm}}^2 W_{si}\le W_{sm}\le W_{sr} \\ P_r W_{sr}\le W_{sm}\le W_{so} \\ 0 W_{sm}\ge W_{s0} \end{array} \right.$$

(1)

where $W_{sm}$ is the original wind speed, $W_{si}$ is the cut in wind speed of the turbine, $W_{sr}$ is the rated wind speed, $W_{so}$ is the cutout wind speed, $P_r$ is the rated power of the turbine.

The constants X, Y, and Z are functions of the wind speed as presented in (2) to (4) [23].

$$X={\displaystyle \frac{1}{{(W_{si}-W_{sr})}^2}}\left[{W_{si}(W_{si}+W}_{sr})-4{(W}_{si}{W}_{sr}){\left({\displaystyle \frac{W_{si}+W_{sr}}{2W_{sr}}}\right)}^3\right]$$

(2)

$$Y={\displaystyle \frac{1}{{(W_{si}-W_{sr})}^2}}\left[4{(W}_{si}{W}_{sr}){\left({\displaystyle \frac{W_{si}+W_{sr}}{2W_{sr}}}\right)}^3-{(3W_{si}+W}_{sr})\right]$$

(3)

$$Z={\displaystyle \frac{1}{{(W_{si}-W_{sr})}^2}}\left[2-4{\left({\displaystyle \frac{W_{si}+W_{sr}}{2W_{sr}}}\right)}^3\right]$$

(4)

The graphical representation of the wind turbine that describes the uncertainties in the output power based on the wind speed at any given condition is depicted in Figure 3. Since the wind speed is not constant over time, this implies that the power output varies with time. The curves show that the power generated varies with the wind speed of the location change; this affects the output power of the wind turbines, which can affect other power system parameters such as voltage, power flow dynamics, and frequency in the networks.

Figure 3. Dynamics of wind energy generator based on wind speed.

2.2. PV Energy Generator

The output power of the Photovoltaic (PV) energy depends on factors such as solar radiation and temperature. The output of the PV energy varies with time; this implies that the integration into the power system tends to affect other operating parameters, such as voltage and frequency. The output power model of the PV module is described in (5) [24,25].

$$P_{PV}\left(t\right)=P_{rated}\times \left({\displaystyle \frac{G}{G_{rated}}}\right)·\left[1+\propto \left(T-T_{ref}\right)\right]$$

(5)

where P_rated is the rated power at standard test conditions, G is the actual solar irradiance, $P_{PV}\left(t\right)$ is the output power, $G_{rated}$ is the rated irradiance at standard test conditions, T is the PV temperature, T_ref is the reference temperature, and $\propto$ is the temperature coefficient, which is usually between −0.004 and −0.006. Equation (5) shows that the PV module output is influenced by solar irradiance and temperature. The dynamics in the output power of PV-based energy generators tied to a grid network, as decentralized generators, can negatively affect voltage-regulating devices in power system networks under different loading conditions if not properly coordinated [23].

2.3. Dynamic Line Ratings

Dynamic line ratings (DLRs) are a critical advancement in power transmission, allowing for the adjustment of a transmission line’s current-carrying capacity based on real-time environmental conditions, as opposed to static line ratings (SLRs), which rely on worst-case scenarios [26,27,28]. DLRs leverage real-time meteorological conditions such as ambient temperature, wind velocity, and conductor characteristics to continuously update transmission line capacity while ensuring compliance with safe operational constraints [27,28]. By integrating real-time atmospheric data and continuous monitoring of line conditions, DLRs optimize the transmission capacity, enhancing both the efficiency and reliability of the power grid [29,30,31,32]. Compared to conventional fixed-rating methods, DLRs significantly improve network flexibility and operational efficiency, making them especially suitable for power systems with high renewable energy penetration and rapidly changing load patterns [33,34], improved utilization of existing transmission infrastructure, reducing the need for costly grid upgrades; enhanced grid stability and reduced congestion, leading to potential cost savings and improved reliability [35,36]. Despite the benefits of DLRs in modern power systems, an inappropriate approach to modeling the uncertainties of DLRs will lead to operational problems in power system networks. There are two major standards that govern the modeling of DLRs in power systems: the IEEE standard 738 and the “Conseil International des Grands Reseaux Electriques” (CIGRE) standards. However, these two standards have some similarities. In this review, we considered the factors used based on IEEE 738 standards to model the uncertainties caused by DLRs in power systems. In IEEE 738, the standard hourly current capacity of the line is computed based on (6) [20].

$$I=\sqrt{{\displaystyle \frac{Q_c+Q_r+Q_s}{R\left(T_{avg}\right)}}}$$

(6)

Qc, Qr, and Qs are the heat losses due to convection, radiation, and solar gain, respectively. Equation (7) is applied to determine heat loss due to forced convection when the wind velocity is nonzero. If no airflow exists, the heat loss mechanism transitions to natural convection, and the corresponding value is obtained from Equation (8) [20,23].

$$Q_c=max\left\{\begin{array}{c}{K}_{angle}[1.01+1.35{N_{Re}}^{0.52}{k}_f\left(T_s-T_a\right)]\\ K_{angle}{{ 0.754N}_{Re}}^{0.6}{k}_f(T_s-T_a)\end{array}\right.$$

(7)

$$Q_C=3.645{\rho }^{0.5}{d_o}^{0.5}{\left(T_s-T_a\right)}^{1.25}$$

(8)

In addition, the radiation heat transfer is determined based on Equation (9).

$$Q_r=17.86d_o\epsilon \left[{\left({\displaystyle \frac{T_s+273}{100}}\right)}^4-{\left({\displaystyle \frac{T_a+273}{100}}\right)}^4\right]$$

(9)

where T_s and T_a are the conductor’s surface temperature and the ambient temperature, both expressed in absolute (Kelvin) degrees, $\epsilon$ emissivity and d₀ outer diameter of the conductor [20,37].

Equation (10) is used to evaluate the sun’s heat on the conductor.

$$Q_s=\infty Q_{se} \sin \left(\theta \right) A$$

(10)

where A, $\infty$ is the solar absorptivity of the conductor surface, is the conductor area, θ is the sun incidence angle on the conductor, and Q_se is the total solar and sky radiated heat adjusted for elevation at a specific date, time, and location [20]. More details on modeling the uncertainties of transmission lines based on dynamic line rating can be found in reference [20]. The DLR is determined through parameters that vary in response to real-time meteorological conditions, which vary with time and affect the power flow dynamics. This means that the capacity of the line under the variation of the weather parameters changes with time, as described in Figure 4. It is, therefore, a recommended approach for system operators to model the uncertainties caused by the DLR based on the weather parameters for effective operational control in power system networks with integration of DERs.

Figure 4. Line current and weather conditions over a two-day period.

2.4. Electro Mobility (Electrical Vehicles)

There are different approaches that have been adopted for modeling the uncertainties of EVs. The use of a physical model, such as the state of charge and discharge in the battery in [38,39,40], and the use of probability approaches that estimate the operational behavior of the EV [41,42,43]. The EV operation is influenced by human behavior, which can be characterized by the departure time and arrival time from the charging station. The real-time approach is described in (11) and (12) in modeling the EV behaviors, which is the best approach; it will allow effective coordination of EVs with other power system devices under real-time conditions for better decision making.

$$P^{V2G}\left(t\right)=\sum _{i=1}^N{P_i}^{V2G}\left(t\right)$$

(11)

$$P^{G2V}\left(t\right)=\sum _{i=1}^N{P_i}^{G2V}(t)$$

(12)

where ${P_i}^{V2G}$ and ${P_i}^{G2V}$ are the discharging and charging power of the EV at i. These dynamics in the operation of EVs caused a lot of uncertainties in power system networks. A typical EV penetration profile in a grid network is presented in Figure 5. It shows how the penetration of EVs varies with time. The approach of managing the EV dynamics with other uncertainties that vary with time in power system networks requires an intelligent system for proper control in power system optimization and decision making.

Figure 5. EV penetration profile with time.

2.5. Dynamic Loading

The loading conditions, which depend on human behavior, can affect the voltage operating conditions in the power system [44,45]. Dynamic loading conditions in a power system in the presence of DERs can significantly impact voltage stability [45,46]. The impact can be analyzed using power flow equations. The dynamic load profile of any bus in the power system networks, which varies with time, can be presented using (13).

$$P\left(t\right)=\sum _i^N{P}_i\left(t\right)$$

(13)

where P(t) is the total load in the network at time t, Pi(t) is the load of any bus i in the network at time (t). The inability to handle this uncertainty in the power system affects the operating parameters in the networks, such as voltage and power flow dynamics [46,47].

The renewable generation variability, weather-dependent line capacity, and human-driven load behavior and EVs significantly influence power system parameters such as voltage profiles [48,49], power flow [50,51], and line loading. Furthermore, the integration of DERs also introduces the bidirectional flow into power system networks, which affects the operation of voltage regulating devices, such as OLTC, designed for unidirectional flow in the power system networks [52,53,54,55]. The ability of the DERs through grid-connected inverters to import and export reactive power at the point of connections also tends to affect the voltage of other buses in the networks if the inverters and EVs are not coordinated with other voltage-regulating devices in the presence of uncertainties [56,57]. Many researchers have introduced various techniques, such as optimization, probability, and fuzzy controllers [58,59,60,61,62,63,64], to address the uncertainties present in modern power systems. Optimization approaches are relatively easy to implement but face exponentially increasing computational demands as the number of control variables grows [65,66,67]. Probabilistic methods rely on the availability of well-defined probability distributions for uncertain inputs (e.g., load demand, wind speed, solar irradiance). In practice, these uncertainties are often non-Gaussian, multi-modal, and non-stationary, which makes accurate modeling difficult [68,69]. Fuzzy controllers rely on predefined rules and membership functions [68,69], limiting adaptability to highly dynamic and uncertain power system conditions [70,71,72,73,74,75,76]. More so, their performance degrades under large-scale uncertainties, high-dimensional state spaces, and changing operating scenarios, requiring extensive tuning and lacking optimality guarantees for complex real-time operational control [77,78]. Given the limitations of these traditional techniques, an intelligent system such as reinforcement learning, capable of learning from the dynamics and complexities of power systems to make real-time decisions and initiate necessary actions, would be highly beneficial for modern power system operations under uncertainties.

3. Reinforcement Learning Architecture and Agents

This section provides a short description of machine learning classification, the main types, and a detailed discussion of different reinforcement learning agents that can be used for operational control and optimization in power system networks. Machine learning methods generally fall into three fundamental categories: (a) supervised learning, (b) unsupervised learning, and (c) reinforcement learning [14]. These categories are based on the manner in which the algorithms interact with data and their environment. Supervised learning, often serving as a function estimator, strives to establish a linear mapping from training inputs to labeled outputs based on a predefined assessment metric [14]. Within supervised learning deep neural networks (DNNs) have garnered significant attention in recent years [79]. In contrast to supervised and unsupervised learning, RL is considered an active learning approach. The fundamental framework of RL is illustrated in Figure 6 [80,81].

Figure 6. Reinforcement learning architecture.

There are two key elements: the agent and the environment. The decision-making process in reinforcement learning (RL) can be outlined as follows. At each time step (t), the agent perceives the environment through an observation. Based on this perception, it selects an action according to the current policy. The chosen action influences the environment, causing it to transition to a new state. Simultaneously, the environment provides a reward value to the agent, serving as feedback on the action’s effectiveness. The agent’s objective is to maximize the cumulative reward from the environment by refining its control strategy through continuous interactions. RL develops optimal decision-making behavior by persistently interacting with the environment and leveraging the gradient computed from the reward feedback signal. This gives superiority over the supervised and unsupervised algorithm that learns from the historical data to form an estimator model. RL learns from the environment based on the reward after each action has been initiated. This makes it suitable for operational control in a power system network that is dynamic in nature in the presence of uncertainties.

Traditional RL uses a Q-table, which becomes infeasible for large-scale power systems. Deep RL overcomes this by using deep neural networks to approximate the Q-function [82]. Deep Reinforcement Learning (DRL) is an advanced RL approach that integrates deep neural networks with RL algorithms to handle complex, high-dimensional power system problems. It is useful in optimizing Distributed Energy Resources (DERs) and voltage regulating devices in power systems where traditional RL struggles with large state spaces. The integration of neural networks makes DRL different from Q-Learning; instead of Q-table, the DRL uses a neural network $Q_{Ø}\left(s,a\right)$ parameterized to approximate Q-values as described in (14) [80].

$$Q_{Ø}\left(s,a\right)=Q_{Ø}\left(s,a\right)+\alpha [R\left(s,a\right)+\gamma \underset{a^{\prime }}{\max }{Q}_{{\varnothing }^{\prime }}\left(s^{\prime },a^{\prime }\right)-Q_{Ø}\left(s,a\right)]$$

(14)

$Q_{Ø}\left(s,a\right)$ is the neural network estimating Q-values, $Q_{{\varnothing }^{\prime }}\left(s^{\prime },a^{\prime }\right)$ target the network used for stable learning, and $\alpha$ and $\gamma$ are the learning rate and discount factor, respectively. R represents rewards factor s, and a is the initial states and actions, respectively. s′ and a′ are the next state and next action taken by the agent. The Deep RL can be classified into a value-based algorithm and policy gradient algorithm [14,83].

3.1. Value-Based Algorithm

One of the key advancements in deep reinforcement learning (DRL) is the value-based Deep Q-Network (DQN) algorithm, which employs deep neural networks (DNNs) to approximate the action-value function. The structure of the DQN algorithm is depicted in Figure 7 [84]. The Deep Q-Network (DQN) algorithm utilizes a replay buffer to store a large set of transition experiences, denoted as (s, a, r, s′). The experience replay mechanism plays a crucial role in stabilizing the learning process by breaking the correlation among training samples. This is achieved by randomly sampling mini-batches of past experiences from memory during neural network updates, rather than relying on sequential data.

Figure 7. DQN framework.

Additionally, DQN incorporates a target Q-network, which helps mitigate the issue of non-stationary data distribution. By periodically updating the target network, DQN enhances training stability and prevents divergence. At each time step, the parameters of the action-value function are optimized by minimizing the following loss function in (23) [84]. Several enhanced variants of DQN, such as Double DQN and Dueling DQN, have been proposed to address the issue of overestimation [85,86]. The use of deep neural networks in DQN eliminates the need for discretizing the state space. However, because DQN selects actions by maximizing the action-value function, it still requires the discretization of the action space when dealing with continuous control problems such as those involving flexible alternating current transmission (FACT) devices. For example, the control of reactive power of Static synchronous compensator (STATCOM) and the Static var compensator (SVC) are continuous variables that are used in operation control for voltage regulation in the power system. The discretization of any of these continuous variables can result in the curse of dimensionality, where the total number of possible actions grows exponentially with the number of action variables.

$$L\left(\varnothing \right)=E_{\pi }\left(\right({Q\left(S_t, a_t|\varnothing \right)-r\left(S_t, a_t\right)-\gamma \begin{array}{c}max\\ a\end{array}Q\left( S_{t+1}, a_{t+1}|{\varnothing }^{\prime }\right))}^2)$$

(15)

where $\varnothing$ and ${\varnothing }^{\prime }$ are the parameters of the action-value function.

Additionally, breaking the action space, such as the voltage profile of nodes in power system networks, into discrete intervals may lead to information loss, potentially resulting in suboptimal performance. Due to this problem of discretization, the value-based algorithms are not the most suitable RL agents for operational control of continuous states such as STATCOM, and active and reactive power of generators. However, value-based algorithms are more suitable for discrete control variables such as an on-load tap changer, OLTC, and switch capacitor banks (SCBs). The value-based algorithms are more suitable in power system applications where the observed states are discrete or can be discretized, such as voltage profiles of buses in the networks, which can be discretized into low, normal, and high voltage. In addition, value-based algorithms can be used to observe the generator’s response state, which is suitable for demand response management. In general, in using a value-based algorithm, the best observation and action space should be discrete to ensure optimal performance.

3.2. Policy Gradient Algorithm

The policy gradient algorithm is well-suited for tasks involving continuous and high-dimensional action spaces. Unlike value-based methods that estimate the action-value function, this approach directly learns a policy mapping from observed states to control actions. The policy is typically represented by a function parameterized by a set of weights, denoted as $\varnothing$. The objective is to optimize $\varnothing$ in order to maximize the expected cumulative reward, expressed as E_(x~P_$\varnothing$ (t)(R│π)). The optimization is carried out by computing the gradient, as shown in (16) [87].

$$\nabla R_{\varnothing }=E_{p\left(\mathsf{\tau }|\varnothing \right)}(R\left(\mathsf{\tau }\right){\nabla }_{\varnothing }lg{P}_{\varnothing }\left(\mathsf{\tau }\right))$$

(16)

A trajectory, denoted as τ = {S₁, a₁, S₂, a₂, …, S_T, a_T}, represents a sequence of states and actions over an episode of length T, with R(τ) indicating the total reward accumulated along the trajectory. In policy gradient methods, the neural network parameters are updated in a direction that increases the likelihood of trajectories yielding higher rewards. However, these algorithms often suffer from high gradient variance. To mitigate this, a baseline is subtracted from R(τ), commonly replaced by the value function V(s) learned by a critic network. This approach forms the foundation of actor-critic methods, a class of policy gradient algorithms. The general framework of actor-critic algorithms is illustrated in Figure 8.

Figure 8. Structure of actor critic algorithms.

Deep Deterministic Policy Gradient (DDPG) is a type of actor-critic algorithm. It utilizes two separate functions: the actor μ(^.|θμ), which learns the policy for selecting actions, and the critic Q(^.|θQ), which evaluates the quality of the actor’s decisions. These two components are trained concurrently, the actor improves its policy based on feedback from the critic, while the critic refines its evaluations to better support the actor. To enhance training stability, DDPG incorporates techniques such as experience replay and target networks. The critic’s parameters, θ^Q, are updated by minimizing the loss function defined in (17) [88].

$$\left\{\begin{array}{c}L\left({\varnothing }^Q\right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}(Q{\left(S_t,a_t|{\varnothing }^Q\right)-y_t)}^2 \\ y_t=r\left(s_{t,} a_t\right)+\gamma Q^{\prime }(S_{t+1}{\mu }^{\prime }(S_{t+1}|{\varnothing }^{{\mu }^{\prime }})|{\varnothing }^{Q^{\prime }}) \end{array}\right.$$

(17)

N denotes the number of samples within a batch; Q′ (.) and μ′ (.) represent the target critic and target actor networks, respectively; and θ^Q′, θ^μ′ are their corresponding parameters. The actor network is optimized using the deterministic policy gradient, as defined in (18) [89].

$$\begin{gathered} {\nabla }_{{\varnothing }^{\mu }}\mu =E\left({\nabla }_{{\varnothing }^{\mu }}Q\left(s,a|{\varnothing }^{\mu }\right){|}_{s=s_t,u=u\left(s_t|{\varnothing }^u\right)}\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=E\left({\nabla }_{\varnothing }{\mu }_{\varnothing }\left(S|{\varnothing }^{\mu }\right){|}_{s=s_t}{\nabla }_{a}Q\left(s,a|{\varnothing }^Q\right){|}_{s=s_{t,}u=u\left(s_t|{\varnothing }^u\right)}\right) \end{gathered}$$

(18)

The target networks are updated using a soft update mechanism, which helps mitigate the effects of non-stationary training data distribution. This makes DDPG suitable for continuous variables observation and action states. In power system operational control, the DDPG agent is suitable for observation states that are continuous, such as reactive and active power of generators, and the reactive power of voltage compensating devices. Unlike the experience replay technique utilized in algorithms such as DQN and DDPG, the Asynchronous Advantage Actor-Critic (A3C) algorithm adopts multiple parallel worker agents to reduce data correlation and enhance training stability. Each local worker computes gradients independently, which are then aggregated and applied to update the global neural network. To encourage better exploration and facilitate convergence to an optimal policy, an entropy term is incorporated into the loss function. The policy network parameters π(^.|θμ) are optimized using the expression given in (19) [90].

$$\nabla R_{{\varnothing }^{\mu }}\approx {\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{t=0}^T(A(s_t, a_t){\nabla }_{{\varnothing }^{\mu }}lg{\pi }_{{\varnothing }^{\mu }}({a_t}^n\left|{S_t}^n\right)+\beta {\nabla }_{{\varnothing }^{\mu }}H(\pi \left(S_t|{\varnothing }^{\mu }\right))$$

(19)

In this context, A(s_t, a_t) represents the advantage function, H(^.) denotes the entropy function, β is the entropy coefficient, and T is the time horizon. The Soft Actor-Critic (SAC) algorithm is another DRL method that incorporates entropy regularization. It employs a value function along with two Q-functions and alternates between updating these networks using sampled batches from experience replay and gathering new experiences based on the current policy [91]. Among the classical policy gradient methods are Trust Region Policy Optimization (TRPO) [92] and Proximal Policy Optimization (PPO) [93], both introduced to address the low data efficiency typical of standard policy gradient techniques. PPO, in particular, prevents drastic policy updates by using a clipped surrogate objective function, leading to improved training stability and robustness. Compared to TRPO, PPO is simpler to implement while maintaining strong performance and reliability.

In the context of optimization in power systems, any of the reinforcement learning algorithms can be applied to enhance voltage regulation and minimize power losses in the presence of uncertainties. However, the application of any of the RL agents depends on the actions and observed state. The observed variables influence the policy learning. A well-structured observation space is critical for successful learning policy. Therefore, observation space must capture enough relevant system information to allow the agent to learn robust, optimal policies. However, this tends to increase the training time of the RL agents in operational control in the power system. The observed space depends on the control action, in which all variables the policy uses to decide actions are important. For example, in the coordination control of PV energy and voltage regulating devices under the uncertainty of loading and PV power generated, the observed variables should include the voltage of all buses in the network, the active and reactive power generation at the point of connection of PV, and the voltage regulating devices. In general, the objective of the reward function influences what should be in the observed space. RL algorithm can be used to coordinate the action of voltage regulating devices in the power system, and the reward of the actions will be used to optimize the operation of the next action until the optimal solution is achieved. In addition, more than one agent can be used for voltage regulating devices in the power system, and the reward of the actions will be used to control the actions of different voltage regulating devices. RL ensures a fast response for making decisions after training, compared to traditional optimization techniques that are based on objectives and constraints in the networks. However, to ensure accuracy in any of the RL algorithms, tuning of the parameters is required, and the training time is much longer. However, the use of the agent depends on the environment and the actions that need to be initiated. The use of RL for voltage management in power systems will involve the control of the voltage regulating devices, which can be discrete or continuous. Table 1 depicts the RL agents suitable for different applications in power system operational control based on the observed and action space.

Table 1. Different RL agents and the best observation and action space.

Agent	Observation Space	Action Space	Power Systems Applications
Q-Learning	Discrete	Discrete	OLTC tap position, Switch capacitor banks, Reactive power of DGs with step changes
DQN	Discrete	Discrete	OLTC tap position, Switch capacitor banks, and Reactive power of DGs with step changes.
DDPG	Continuous	Continuous	Active power and reactive power of PV smart Inverters, Reactive power of voltage regulating devices such as SVC, STATCOM, and Capacitor banks
TD3	Continuous	Continuous	Active power and reactive power of PV smart Inverters, Reactive power of voltage regulating devices such as SVC, STATCOM, and Capacitor banks
SAC	Continuous	Continuous	Active power and reactive power of PV smart Inverters, Reactive power of voltage regulating devices such as SVC, STATCOM, and Capacitor banks
PPO	Discrete/Continuous	Discrete/Continuous	Both Discrete and Continuous Variables
A3C/A2C	Discrete/Continuous	Discrete/Continuous	Both Discrete and Continuous variables

In many power systems, the voltage profile is discretized into three states: low, normal, and high voltage. This allows DQN to observe the voltage states in power systems. Other agents can observe the voltage profile as a continuous state, which tends to increase the training time of the RL agent. In addition, the major challenges of applying classical RL agents in power system operational control are the trial-and-error way of taking actions based on the reward functions and the penalty factors, which is a key challenge for application in power systems. This approach may not be safe for power system networks since violation of parameters tends to cause contingencies in the network. To address these problems, constrained RL (CRL) and hybrid approaches can be suitable to offer promising directions to ensure safe and reliable grid operation. In CRL, the reward function in classical RL is maximized subject to constraints on the variables in the networks instead of penalties that require tuning, which enables safe operation in the power system networks. In classic RL, the standard way to handle constraints (like voltage limits, capacity, or safety bounds) is to penalize constraint violations in the reward function. For example, if the voltage goes above 1.05 p.u., subtract 100 from the reward. This approach allows the agents to learn from the bad decision and take actions to correct the violations. Since this approach can work in an environment that is forgiving, that is, an environment where wrong actions during learning cannot affect the system. However, in power system networks, this approach is not suitable as violations can lead to catastrophic or unsafe grid conditions. Since the penalties only discourage violations, they do not strictly forbid them. The agent might still explore unsafe actions if it thinks it will get a higher long-term reward; this approach is like trial and error, which cannot be tolerated in power system networks. Furthermore, power systems often have strict voltage, current, thermal, frequency, or protection constraints. If an RL agent violates these during training or deployment, it could lead to unsafe or unrealistic actions. To avoid these issues during training for RL agents, a constraint RL is introduced to extend the classical RL agents to use mathematical optimization by maximizing or minimizing the reward function subject to constraints of variables such as voltage, which leads to safety for power system networks. This approach of constraint RL can be developed in the classical RL algorithm in the following ways: Lagrangian relaxation is the most common approach that turns the constrained problem into an unconstrained one using a Lagrange multiplier. The agents will learn both the policy and the Lagrange multipliers [86]. Constrained Reinforcement Learning (CRL) improves safety by incorporating operational limits into the learning process; however, it exhibits several limitations in complex power system applications. CRL is highly sensitive to penalty and constraint tuning, which can lead to conservative operation or instability during training. As system uncertainties increase due to dynamic line ratings, electric vehicle behavior, and high renewable penetration, the scalability and convergence of CRL deteriorate. Moreover, constraints in CRL are treated as static restrictions embedded in the reward function, limiting adaptability, interpretability, and long-term planning. Constraint violations may still occur during exploration, posing risks to grid security. To address this problem, a hybrid approach, such as Belief–Desire–Intention (BDI) within the RL framework, enhances decision-making by integrating cognitive reasoning with learning. The Belief component will improve state interpretation under uncertainty, the Desire component explicitly encodes system objectives, and the Intention component enforces feasible, goal-oriented actions. This structure reduces unsafe exploration, improves learning efficiency, and enables adaptive, interpretable, and constraint-aware control suitable for real-time power system operation.

4. Review of Reinforcement Learning-Based Control Strategies for Power Systems Under Uncertainty

In this section, we discuss the applications and limitations of various RL algorithms used for power system operational control to address the challenges introduced by uncertainty in the networks. Researchers have employed different reinforcement learning agents such as DQN, DDPG, distributed proximal policy optimization (DPO), multi-agent DDPG (MADDPG), multi-agent deep reinforcement learning (MADRL), evolutionary curriculum-based MADDPG (EC-MADDPG), attention-based MADDPG (AMADDPG), soft actor critic (SAC), multi-agent SAC (MASAC), multi-agent actor attention critics (MAAC), physics-Informed MADDPG (PI-MADDPG), distributed deep reinforcement learning agent (DRLA), and soft graph actor critic (SGAC), as intelligent strategies to tackle these issues, focusing on various objectives to control state actions for effective power system operation. The authors in reference [94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110] applied different RL agents for optimal coordination and control in power systems using a single reward function to ensure effective voltage regulation and power flow in the networks. In addition, References [111,112,113,114,115,116,117,118,119,120], adapts RL agents for optimal control and coordination based on multiple reward functions in addressing the problem of voltage regulation and problems introduced by fluctuation in DERs sources. The reward function of the RL depends on the objectives of the power system operators or the engineer. The control objectives in modern systems frequently include economy, such as cost minimization, and safety, such as voltage and frequency stability. These are often conflicting, as increasing safety or controllability can decrease profitability. More so, flexibility is also a key objective, particularly in environments with dynamic operating conditions or varying feedback. It is often framed as a distinct but complementary dimension to stability, enabling a system to handle uncertainty. More so, balancing weights is a critical step in multi-objective optimization to navigate trade-offs between conflicting goals. This must be ensured to avoid bias in the reward function. The common strategies include the weighted sum method, normalization, and Pareto analysis [121,122]. However, dynamic weight adaptation (DWA) and hypervolume-guided weighting are superior to the static methods because of their adaptation online during training to navigate trade-offs, preventing underperforming tasks from being ignored [123,124]. This approach is more appropriate for designing the multiple reward function in power systems when reinforcement learning agents are used.

The summary of the reviewed literature is presented in Table 2, based on the RL algorithm used, the reward objective, uncertainties considered, and the limitations of the study. The classification dimensions were divided based on the functional requirements of power system operations and the structural characteristics of RL algorithms. Uncertainty sources are divided based on physical uncertainty, such as loading DERs like wind and solar. The control objectives are categorized based on the dynamic operational goal, such as cost of energy, voltage regulation, frequency control, power losses reduction, and active power balancing. The learning architecture can be in two states: single learning and multi-agent learning, where an agent can learn locally with high-dimensional states.

Table 2. Summary of the review of the application of reinforcement learning in operational control of power systems.

RL Algorithm Used	Rewards Objective	Reference	Uncertainties	Learning Architectures	Limitations	Training Time	Research Gap
RL based on Q learning	Voltage deviation	[94]	DERs and Loading	Single agent learning	Linearizing power flow for the environments leads to loss of information and lacks constraints within the RL framework, leading to unsafe operational control.	Low	Incorporate nonlinear models and constraint-aware learning for safe operation
DQN	Voltage deviation	[95]	DERs and Loading	Single agent learning	The reward function is based on penalty factors, which can lead to unsafe decisions for the power system.	Medium	Need to incorporate constraint enforcement beyond reward shaping
DDPG	Voltage deviation	[96]	Loading	Single agent learning	The reward functions lack constraints, which leads to unsafe conditions in the network.	High	The need for constraint-guided continuous control frameworks
MADDPG	Voltage deviation	[97]	DERs and Loading	Multi-agent learning	The agents can initiate unsafe actions since the reward function is based on the penalty factor and reward.	High	Incorporate coordinated multi-agent constraint enforcement
MADDPG	Active power	[98]	Loading	Multi-agent learning	The centralized training with decentralized execution framework helps mitigate the non-stationarity issue in multi-agent environments; it still faces challenges in managing large agent populations and optimizing reward functions to promote efficient information exchange among agents.	High	There is a need to improve scalable coordination and communication strategies
AMADDPG	Voltage deviation	[99]	DERs and Loading	Multi-agent learning	The agent still relies on the reward function to encourage constraint satisfaction. Therefore, the approach can still lead to constraint violations if the reward and penalty are not tuned properly, which is not suitable for power system networks.	High	There is a need to design adaptive constraint satisfaction independent of reward tuning
MADRL	Energy Cost	[100]	Loading	Multi-agent learning	The agents will suffer from function approximation errors, which may result in suboptimal control policies. In addition, like many reward-driven RL methods, this approach does not inherently guarantee strict compliance with physical or operational constraints.	High	Improve robustness and constraint compliance mechanisms
DDPG	Voltage deviation	[101]	DERs and Loading	Single agent learning	Despite improving stability and reducing overestimation bias compared to standard DDPG, multi-agent TD3 still relies entirely on the reward and penalty structure to guide the agents’ behavior. This means it does not inherently guarantee that the learned policies strictly satisfy critical physical or operational constraints.	High	There is a need to ensure a safety-guaranteed policy learning under uncertainty
DDPG	Voltage deviation	[102]	Loading	Single agent learning	DDPG-based voltage regulation relies on well-shaped rewards; no guarantee that learned actions respect generator limits or grid constraints under rare scenarios, which can lead to unsafe violations of the power system.	High	There is a need to enhance its generalization under extreme scenarios
MADDPG	Voltage deviation	[103]	DERs and Loading	Multi-agent learning	MADDPG adopted is the computational burden during the training process, which requires much time for stability, and the agents can give actions that violate the operational limits in the network.	High	There is a need to reduce the computational training time
PI-MADDPG	Voltage deviation	[104]	DERs and Loading	Multi-agent learning	The physics-informed neural networks within the MADDPG agents address the computational burden in the training process of MADDPG. However, the physics-informed MADDPG still requires careful constraint design to guarantee voltage and power flow feasibility.	Medium	To enforce safety, the physics-based constraints need to be automated
EC-MADDPG	Voltage deviation	[105]	DERs and Loading	Multi-agent learning	Actions are penalty and reward-dependent, which can lead to unsafe decision variables	High	To ensure safe operation, the reward-based mechanisms for safety must be decoupled
DDPG	Voltage deviation	[106]	DERs and Loading	Single agent learning	The actions still depend on the reward function, which does not consider the limits in the network.	High	There is a need to embed physical limits directly in the decision process
DDPG	Voltage deviation,	[107]	DERs and Loading	Single agent learning	A single DDPG for multiple action spaces requires more time for hyperparameter tuning to avoid instability during training.	High	There is a need to improve training stability and efficiency
DDPG and TD3	Voltage deviation	[108]	DERs	Single agent learning	To ensure safe operation in the practical application of the DRL, a constraint needs to be added to the reward framework for effective operational control.	High	There is a need to develop inherent constraint-aware RL models
TD3	Voltage deviation	[109]	DERs	Single agent learning	The reward may not enforce strict, discrete device logic under uncertainty. Integration of discrete constraint solvers or hybrid rule-based checks within the TD3 action space will ensure safe operation.	High	Need to integrate hybrid discrete continuous control frameworks
SAC	Active power loss	[110]	DERs and Loading	Single agent learning	SAC provides fast control but relies fully on offline reward shaping; no explicit constraints to prevent actions that push voltage outside limits under abnormal events. This can be addressed by embedding constraints through safety layers in the SAC to ensure safe operational control.	Medium	There is a need to embed safety layers and real-time constraints
DQN	Voltage deviation and active power	[111]	DERs and Loading	Single agent learning	Relying solely on reward and penalty factors in reinforcement learning can be problematic because they do not guarantee strict adherence to critical physical constraints like voltage, frequency, or thermal limits.	Medium	Need to guarantee compliance with physical constraints
MADDPG	Voltage deviation and SOC	[112]	DERs and Loading	Multi-agent learning	The multi-agent DDPG controller used the reward, which depends on the voltage deviation and the state of the SVC at the point of connection. The approach can be improved by adding physical constraints within the reward function to ensure safe decisions.	High	Integrate physical constraints beyond reward design
MAAC	Voltage deviation and power loss	[113]	DERs and Loading	Multi-agent learning	This approach can lead to control conflict if not properly coordinated, and the decision process can be unsafe.	High	Need to develop coordinated, conflict-free, multi-agent control
MASAC	Active power losses and voltage deviation	[114]	DERs and Loading	Multi-agent learning	The two agents are trained at the same time, and the actions taken are based on the rewards and the penalties, which are a function of the voltage of all buses in the network, which will lead to unsafe operation in the network.	High	Need to introduce safety-driven multi-agent coordination
MADDPG	Voltage deviation and losses	[115]	DERs and Loading	Multi-agent learning	However, since the MADDPG relies on reward and penalty functions to guide reactive power control, there is no strict guarantee that the learned control actions will always satisfy voltage or inverter capacity constraints under all operating conditions, especially when the system faces unexpected scenarios.	High	There is a need to ensure robustness under the uncertainty of EV and DLR
MASAC	Cost of DGs and voltage instability	[116]	DERs	Multi-agent learning	It relies purely on the reward and penalty structure, mainly cost and voltage terms, which do not strictly guarantee that all operational and voltage constraints will be satisfied under unexpected scenarios.	High	Incorporate uncertainty-aware constraint handling
MADDPG	Voltage deviation and state of charge (SOC)	[117]	DERs and Loading	Multi-agent learning	MADDPG uses reward based only on voltage deviation and SOC; it may act sub-optimally in grids with multiple voltage control devices without explicit coordination.	High	Improve coordination among multiple control devices
DPO	Voltage deviation, reactive power	[118]	DERs and Loading	Single agent learning	The approach still depends on well-tuned rewards for enforcing operational constraints. It requires explicit constraint handling through Lagrangian methods or safe policy layers to ensure safe operational control for real-time applications.	Medium	Develop explicit constraint-handling frameworks
DRLA	Active power losses, OLTC, SCB, and reactive power of DG	[119]	DERs and Loading	Multi-agent learning	The optimal power flow model was used to ensure safe operation. However, this increases the training time of the RL agents. To reduce the training time due to the OPF model, a BDI can be integrated into the RL framework to ensure safe operation.	High	Reduce OPF dependency while maintaining safety
SGAC	Frequency deviation and cost of active power	[120]	Loading and DERs	Single agent learning	The issues of operational violations that can lead to unsafe conditions were addressed using an optimal power flow approach to model the power system environment, which increases the training time and can lead to an infeasible solution. To reduce the training time and avoid unfeasible solutions, BDI can be adopted within the RL framework.	High	Improve the efficiency of constraint integration without OPF

Existing literature on reinforcement learning (RL) for power system operational control primarily considers uncertainties from distributed energy resource (DER) variability and load demand. However, modern grids are increasingly affected by additional uncertainties, notably Dynamic Line Rating (DLR) and electric vehicle (EV) charging behavior, which critically impact network stability, operational flexibility, and energy management efficiency. Coordinated control of voltage-regulating devices under combined uncertainties from DERs, loads, and flexible line ratings remains underexplored. The omission of DLRs in RL frameworks introduces significant limitations. Conventional reliance on static line ratings (SLRs) derived from conservative assumptions leads to underutilization of transmission assets, constrains RL agents within restrictive operating boundaries, and reduces renewable integration. Ignoring DLRs restricts the state–action space, limits policy exploration, and decreases learning efficiency. Moreover, conventional penalty-based reward structures may allow transient violations of operational limits, compromising safety and control robustness. Future research should develop constraint-aware RL frameworks that explicitly incorporate DLRs, EV charging patterns, variable loads, and renewable generation. The use of DLRs as part of the uncertainties will require adaptive event-triggered mechanisms for nonlinear multi-agent systems, particularly output feedback control, which efficiently alleviates line congestion by reducing communication burdens and transmitting data only at critical bottlenecks [125]. This approach needs to be integrated into the dynamic line rating (DLR) framework to enhance stability and optimize resource usage. More so, integrating these uncertainties with structured approaches, such as Belief–Desire–Intention (BDI) frameworks, can improve training efficiency, policy adaptability, system reliability, interpretability, and practical deployment in modern power system operations.

5. Existing Challenges of Reinforcement Learning Agents and the Belief–Desire–Intention (BDI-RL) Framework as a Future Research Direction

Reinforcement learning agents in power systems suffer from safety, scalability, and constraint enforcement challenges due to their reliance on reward-driven learning. These limitations hinder their deployment in real-world grid operations and motivate the development of hybrid frameworks such as BDI-RL that incorporate reasoning and constraint-awareness into the learning process. The framework depicted in Figure 9 shows how BDI-RL agents are suitable in power systems for effective operational control actions in the presence of uncertainties. The integration of the Belief–Desire–Intention (BDI) architecture into the Reinforcement Learning (RL) control framework enhances the decision-making capability of intelligent agents operating in uncertain power system environments. In the proposed structure, in Figure 9, the power system environment, including distributed energy resources (DERs), on-load tap changers (OLTCs), STATCOMs, dynamic line ratings (DLR), electric vehicle (EV) behaviors, and variable loads, interacts continuously with the learning agent through observed states, actions, and rewards.

Figure 9. Framework for reinforcement application in the coordination and control of power system devices in the presence of uncertainties.

The Belief module serves as a perception and knowledge layer, processing observed variables such as bus voltages, DERs power outputs, losses, DLR states, and EV charging conditions. This module enables the agent to maintain an updated representation of the system’s operational status, reducing uncertainty and improving situational awareness. The Desire module translates system-level objectives such as voltage regulation, loss minimization, congestion avoidance, and renewable utilization into prioritized control goals. In Figure 9, the desire module can be aggregated as multiple objectives, such as voltage deviation, active power loss minimization, device control, such as minimization of the reactive power of the grid-connected inverter, and active power, with weight factors that reflect the relative importance of each operational objective from the desire module. These desires guide the learning process by shaping the reward structure and aligning RL exploration with operational requirements. Furthermore, constraints such as voltage and reactive power limits are incorporated alongside the reward formulation (as shown in Figure 9), ensuring that the learning process remains feasible and physically compliant. The Intention module acts as a deliberative planning layer that selects feasible control strategies consistent with both learned policies and physical constraints. By filtering RL actions through intentions, unsafe or infeasible control decisions are mitigated, addressing a key limitation of purely reward-driven RL approaches. Moreso, the proposed BDI structure, comprising Beliefs (environment model), Desires (goals), and Intentions (actions), provides inherent robustness to false data injection (FDI) attacks, which can be further enhanced by incorporating adaptive mechanisms. In the classical or standard RL approach, a direct injection of malicious data into state observations or rewards often leads to immediate policy failure (poisoning). However, the BDI framework acts as a cognitive filter. If an attacker injects false data into the observations, the BDI agent first updates its Beliefs. By employing a belief revision mechanism (e.g., comparing incoming perceptions against a prediction model), the system can detect anomalies that violate its previous, validated beliefs. If the inconsistency between perceived and expected data is high, the agent re-evaluates its Intentions (currently active plans/actions) to ensure that the desire (goal) is still pursued safely, rather than immediately acting on potentially falsified data. In addition, if the FDI attack is stochastic or dynamic, for example, V(t) = a(t) + n(t), where V is the observed voltage, a is the attack, and n is the noise, a compensator module within the RL agent. This compensator estimates the unknown attack parameters, similar to adaptive fault-tolerant control techniques based on an observer-based approach. The belief module specifically stores the estimates of the attack magnitude $\dot{A}$. The action A(t) generated by the RL policy is adjusted locally: A_safe(t) = A_learned(t) − $\dot{A}$. This ensures that the agent continues to function according to the intended goals rather than the falsified input data. Under FDI attack, the proposed approach does not fail completely; instead, the agent identifies a high error between perceived and expected environment states (Belief), and the agent activates a local adaptive compensation law that filters out the injected data or compensates for the attack signal. More so, the BDI framework enforces intentions that prioritize safety constraints, ensuring that, despite inaccurate information, the agent keeps the multi-agent system operating safely.

Classical RL methods struggle because the state and action spaces grow exponentially, resulting in slow convergence, poor scalability, and coordination challenges. The BDI-RL framework addresses this issue by introducing a structured, knowledge-driven decomposition of the decision process, which significantly improves scalability and control performance. First, in the Belief layer, the high-dimensional system is represented in a structured and reduced form by incorporating domain knowledge, network topology, and relevant state aggregation (e.g., clustering buses, prioritizing critical variables such as voltage violations or line congestion). Instead of learning from all raw variables, the agent focuses on informative and context-aware features, reducing the effective dimensionality and improving learning efficiency. Additionally, uncertainty (e.g., from renewables or dynamic line ratings) can be embedded probabilistically within beliefs, allowing the agent to reason under complex system conditions. Second, the Desire layer organizes multiple control objectives (e.g., voltage regulation, loss minimization, power transfer maximization) into a hierarchical or prioritized structure, rather than a flat reward function. This avoids the difficulty of balancing many competing objectives simultaneously and enables the system to focus on the most critical goals under specific operating conditions. As a result, the optimization problem becomes more manageable and interpretable, even as the number of variables increases. Moreso, the Intention layer plays a crucial role in reducing the action space and ensuring coordinated control. Instead of allowing the RL agent to explore all possible actions, intentions act as filters or decision rules that select only feasible and safe actions based on system constraints (e.g., voltage limits, line capacities, device constraints). This significantly reduces the combinatorial explosion of actions in large-scale systems. In multi-agent settings, intentions also enable coordinated decision-making, where agents align their actions based on shared beliefs and system-level objectives, thereby mitigating conflicts and improving scalability. Furthermore, BDI-RL supports modular and distributed control architectures, where different agents manage subsets of the network (e.g., microgrids or feeders) using local beliefs while still coordinating through shared intentions. This decentralization reduces computational burden and allows the framework to scale effectively to large power systems. Overall, BDI-RL transforms the problem from a purely data-driven, high-dimensional optimization task into a structured, knowledge-assisted decision-making process. By combining learning with reasoning, it reduces the effective dimensionality, improves coordination among multiple variables and agents, and ensures that control actions remain feasible and safe. This makes BDI-RL a highly suitable framework for optimal coordination and control in increasingly complex and variable-rich modern power systems. The uncertainty of the RL has not been explored within the RL framework. The Dynamic Line Rating (DLR) uncertainties can be effectively incorporated into a Belief–Desire–Intention Reinforcement Learning (BDI-RL) framework by embedding uncertainty-aware information and constraint reasoning across the belief (state), desire (objective/reward), and intention (action/constraint enforcement) layers. In the belief (state space), DLR uncertainty is represented by enriching the agent’s perception with both real-time and forecasted environmental variables, such as wind speed, ambient temperature, and solar irradiance, as well as derived quantities, such as estimated line ampacity and conductor temperature. This allows the belief layer to maintain a stochastic representation of network limits, rather than deterministic values, enabling the agent to reason under uncertainty. In the desire (reward design), the objective can be formulated to reflect risk-aware and reliability-driven goals, and maximizing power transfer can be incorporated as one of the objectives within a multi-objective reward function. This ensures efficient utilization of network capacity while simultaneously enforcing operational constraints such as voltage limits, thermal ratings, and system security requirements. As a result, the learning agent achieves a trade-off between performance and safety, particularly under uncertain operating conditions. In the intention (operational constraints and action selection), DLR uncertainty is enforced through constraint-aware decision-making mechanisms. The intention layer ensures that selected actions satisfy dynamic and uncertain line limits using techniques. Within BDI-RL, these constraints are not just external checks but are integrated into the reasoning process, where intentions are formed only if they are consistent with the beliefs (uncertain system knowledge) and operational safety requirements. Incorporating DLR uncertainties into BDI-RL transforms the control strategy into a knowledge-driven, risk-aware, and constraint-consistent framework, enabling more reliable and practical deployment of RL in power systems with high renewable penetration and dynamic operating conditions.

Overall, embedding BDI reasoning within the RL framework tends to improve interpretability, constraint compliance, and adaptability. This BDI-RL hybrid architecture supports more reliable and goal-directed coordination of voltage-regulating devices under multi-source uncertainties, making it well-suited for real-time operational control in modern power systems compared to the conventional RL framework that lacks interpretability. More so, BDI-RL will offer a robust, adaptive solution for managing uncertainties and maintaining efficient, reliable operation in modern power systems.

6. Practical Feasibility of BDI-RL

The integration of the Belief–Desire–Intention Reinforcement Learning (BDI-RL) paradigm provides a structured enhancement over conventional reinforcement learning (RL) by embedding knowledge-driven reasoning into the learning process. In this framework, the Belief module captures structured knowledge of system states, operational constraints, and network conditions, enabling the agent to maintain an informed representation of the power system environment. The Desire module translates system-level objectives such as voltage regulation, loss minimization, and efficient utilization of distributed energy resources into explicit and prioritized control goals. Meanwhile, the Intention module ensures that the selected actions are both feasible and safe by enforcing operational constraints during decision-making. This integration shifts RL from a purely reward-driven paradigm to a constraint-aware and reasoning-based framework, significantly improving its applicability in real-world power systems.

One of the key challenges in modern power systems is scalability, particularly in large-scale networks with high penetration of distributed energy resources (DERs). Conventional multi-agent RL approaches often face coordination complexity and communication overhead. The BDI-RL framework addresses this limitation by structuring decision-making through localized beliefs, allowing agents to operate with relevant and reduced information, while the intention layer facilitates coordinated and constraint-consistent actions. As a result, BDI-RL improves scalability and enables more efficient management of complex and distributed grid environments. Another critical issue is the high training time associated with deep RL methods such as DDPG and MADDPG, which require extensive exploration and large numbers of training episodes. In contrast, BDI-RL reduces unnecessary exploration by leveraging prior system knowledge embedded in the belief module. This guidance enables more directed learning, leading to faster convergence and a reduction in computational burden, which is essential for practical deployment. The data requirement of RL models also poses a significant limitation, as large datasets are often needed to achieve reliable performance. BDI-RL mitigates this challenge by incorporating domain knowledge into the learning process, thereby reducing reliance on purely data-driven exploration. This makes the framework more data-efficient and suitable for real-world applications where data availability may be limited or costly to obtain.

Importantly, system safety remains a major concern in RL-based control of power systems, as traditional RL does not inherently guarantee compliance with physical and operational constraints. The BDI-RL framework directly addresses this issue by embedding constraints within both the belief and intention modules. This ensures that all selected actions comply with critical requirements, such as voltage limits, power flow constraints, and equipment ratings. Consequently, BDI-RL enables safer and more reliable decision-making, even under uncertain and dynamic operating conditions. Overall, the proposed BDI-RL framework will effectively address key deployment challenges of conventional RL by integrating knowledge-driven reasoning with learning-based control. This results in a more scalable, data-efficient, and safety-aware approach, making it a practical solution for optimal coordination and control in modern power systems.

7. Conclusions

The growing integration of renewable energy sources, distributed generators, dynamic line ratings (DLRs), electric vehicle (EV) charging and discharging, and dynamic load profiles introduces significant uncertainties in modern power systems. Ensuring effective coordination and control of voltage-regulating devices under these uncertain conditions is critical for maintaining grid stability, reliability, and operational efficiency. This review highlights existing control strategies and identifies their limitations in achieving real-time adaptability and optimal performance. Reinforcement Learning (RL) has emerged as a promising solution due to its ability to make fast, data-driven decisions based on system states, overcoming the computational and modeling constraints of traditional optimization approaches. RL’s self-learning and adaptive capabilities facilitate improved voltage regulation and coordinated control in dynamic power networks. Nonetheless, RL faces challenges, including extensive training time, hyper parameter tuning, and computational complexity, which can affect convergence and limit large-scale implementation. Future research should focus on enhancing computational efficiency and robustness of RL through integration of BDI within the RL frameworks to ensure a scalable, explainable, and safety-aware control paradigm well suited for real-time power system operation.