A General Reinforcement Learning Framework for Remedial Action Schemes Against False Data Injection Cyberattacks in Renewable-Integrated Power Systems: From Stochastic Optimization to Markov Decision Processes

Abstract

The increasing penetration of renewable-based distributed generation (RDG) has significantly increased the operational uncertainty and complexity of modern power systems, making conventional optimization-based remedial action schemes (RASs) increasingly computationally demanding and less suitable for real-time critical applications. This paper proposes a general reinforcement learning (RL) framework for solving multi-period stochastic power system optimization problems, with RAS serving as the primary application against false data injection (FDI) cyberattacks targeting smart power systems. The proposed methodology consists of three sequential stages. First, the RAS is formulated as a stochastic nonlinear optimization model that minimizes the total operating cost while explicitly accounting for thermal generation costs, renewable generation costs, renewable power mismatch penalties, and energy storage operation, subject to power flow, operational, and network security constraints. Second, the optimization problem is systematically reformulated as a Markov decision process (MDP), where the system operating conditions define the state space, control variables constitute the action space, and a reward function is designed to capture both economic objectives and operational constraint violations. Finally, the resulting MDP is solved using an actor-critic RL algorithm that learns an optimal control policy directly from historical operating data without requiring an explicit mathematical model of system dynamics. Unlike problem-specific RL formulations, the proposed framework provides a unified methodology for transforming a broad class of stochastic power system optimization problems, including RAS against FDI attacks, optimal power flow, feeder reconfiguration, and loss minimization, into RL-compatible formulations. The framework naturally accommodates the stochastic behavior of renewable generation and sequential decision-making over multiple time intervals while enabling scalable and adaptive control to remediate targeted power system and bring it back to the normal operation in a timely manner. The formulation establishes a generic foundation for applying modern RL techniques to uncertainty-aware power system operation and control, thereby facilitating real-time decision-making in renewable-rich electricity networks.

1. Introduction

The rapid modernization of electric power systems, together with the increasing penetration of renewable-based distributed generation (RDG), has fundamentally transformed the operation and control of modern electricity networks [1]. Renewable energy resources, such as wind turbines and photovoltaic (PV) systems, offer significant environmental and economic benefits by reducing greenhouse gas emissions and dependence on fossil fuels. However, their intermittent and uncertain nature introduces substantial operational challenges, including fluctuations in power generation, voltage deviations, congestion, and increased complexity in maintaining system reliability and security [2]. At the same time, the widespread deployment of advanced metering infrastructure, communication networks, intelligent electronic devices, and supervisory control and data acquisition (SCADA) systems has significantly enhanced the observability and controllability of smart power systems. While these cyber-enabled technologies improve operational efficiency, they also expand the cyberattack surface, making power systems increasingly vulnerable to sophisticated cyber threats. Among these threats, false data injection (FDI) attacks have attracted considerable attention because through man-in-the-middle attacks they can manipulate measurement data while remaining undetected by conventional bad-data detection mechanisms, potentially leading to incorrect control decisions, equipment overloading, voltage instability, cascading failures, and even widespread blackouts [3]. Consequently, developing effective remedial action schemes (RASs) capable of rapidly mitigating the impacts of FDI attacks while accommodating the stochastic behavior of RDG units has become an important research challenge for ensuring the secure and resilient operation of future smart power systems [4].

The importance of this research stems from the growing need for intelligent decision-making methodologies that can simultaneously address RDG uncertainty, cyber resilience, and real-time operational requirements. Conventional optimization-based RASs often require repeatedly solving complex nonlinear and stochastic optimization problems whenever system conditions change, resulting in considerable computational burden that limits their suitability for fast online applications. Although reinforcement learning (RL) has recently emerged as a promising data-driven paradigm for sequential decision-making under uncertainty, most existing studies focus on specific applications and lack a systematic methodology for converting power system optimization problems into RL-compatible formulations. As power systems continue to evolve toward highly decentralized, renewable-rich, and cyber-physical infrastructures, there is an increasing demand for a general framework that unifies stochastic optimization and RL while preserving operational objectives and engineering constraints. Such a framework can enable scalable, adaptive, and computationally efficient control policies that are learned directly from historical operating data, thereby supporting timely RASs against FDI cyberattacks and facilitating broader applications in uncertainty-aware power system operation.

1.1. Related Works

Recent years have witnessed growing interest in applying RL to enhance the resilience, security, and operational efficiency of renewable-rich power systems. Unlike conventional optimization techniques that require repeatedly solving computationally expensive optimization problems, RL-based methods learn optimal control policies directly from interactions with the operating environment, making them particularly attractive for real-time power system applications under uncertainty. For example, in [5], a proximal policy optimization-based multi-agent deep RL framework was proposed to handle remedial actions for voltage source converter high-voltage direct current systems. The proposed method in [5] demonstrated faster convergence and improved voltage regulation under high renewable energy penetration by coordinating multiple intelligent agents during emergency control. An RL-based secure control strategy was introduced in [6] to handle cyber-physical systems operating under FDI attacks. To this end, [6] formulated the secure control problem as an MDP and employed a Lyapunov-guided soft actor-critic algorithm to learn resilient control policies directly from data while maintaining system stability. In [7], an attention-aware deep RL approach was proposed for detecting FDI attacks in smart grids. By integrating an attention mechanism into the RL architecture, [7] extracted informative state features from measurement residuals, thereby improving attack detection accuracy and robustness under varying operating conditions. A deep RL framework for temporal FDI cyberattack generation and detection was presented in [8], where the interaction between attackers and defenders was modeled as a sequential decision-making process, demonstrating that RL can effectively characterize evolving attack behaviors while simultaneously enhancing detection capability in cyber-physical power systems. In [9], a model-free RL-based cybersecurity strategy was developed for frequency control of cyber-physical power systems subjected to FDI attacks. The proposed two-level defense architecture in [9] combined attack detection with a soft actor-critic algorithm for adaptive controller tuning, improving frequency regulation and resilience against malicious measurements. An extended FDI attack model together with a deep RL-based online detection framework was proposed in [10], where the online detection problem was formulated as a partially observable MDP and a compound reward function was designed to minimize false alarms and detection delays while maintaining robustness under varying operating conditions. In [11], an actor-critic RL framework was developed to mitigate FDI attacks targeting wind turbine control systems. The proposed resilient controller in [11] integrated RL with barrier Lyapunov-based control to guarantee state constraints while improving cybersecurity and operational reliability under adversarial conditions. A constraint-aware multi-agent RL framework was introduced in [12] to improve voltage resilience in multi-microgrid systems exposed to simultaneous FDI and denial-of-service attacks. By incorporating graph neural networks, adversarial training, and constraint enforcement mechanisms, the proposed approach in [12] achieved superior voltage regulation and higher operational reliability compared to existing multi-agent RL algorithms. In [13], an actor-critic RL algorithm was proposed for power distribution network reconfiguration. The framework proposed in [13] incorporated prioritized experience replay and safety-aware critics to simultaneously minimize power losses and satisfy operational constraints, demonstrating that safe RL can effectively replace computationally intensive optimization techniques in large-scale distribution networks. Finally, a deep RL-based moving target defense strategy was presented in [14] to enhance smart grid cybersecurity against FDI attacks, where the proposed approach utilized deep deterministic policy gradient and twin delayed algorithms to optimally adjust D-FACTS settings in real time, substantially improving attack detection capability while significantly reducing computational burden compared to iterative optimization methods.

Although the reviewed research work demonstrates the significant potential of RL for enhancing power system security, resilience, and operational performance, majority of them (e.g., [5,6,7,8,9,10,11,12,13,14]) focus on solving individual application-specific problems, such as cyberattack detection, emergency control, voltage regulation, feeder reconfiguration, or optimal power flow. As a result, the design of the state space, action space, reward function, and learning architecture is typically customized for a particular problem, making these approaches difficult to generalize to other power system optimization problems. Furthermore, relatively little attention has been devoted to establishing a unified methodology that systematically reformulates conventional stochastic optimization models into RL problems. Consequently, despite the growing success of RL in power system applications, a clear methodological gap remains between traditional optimization-based formulations and data-driven sequential decision-making frameworks. This gap motivates the development of a more general and systematic RL formulation, which is discussed in the following subsection.

1.2. Research Gaps and Contributions of This Paper

Existing studies on RAS for power systems predominantly formulate the problem as deterministic or stochastic optimization models that require repeatedly solving computationally intensive optimization problems whenever operating conditions change. Although these optimization-based approaches can provide high-quality solutions, their computational burden and dependence on accurate mathematical models limit their applicability to real-time operation, particularly in renewable-integrated power systems characterized by significant uncertainty and rapidly changing operating conditions. More recently, RL has emerged as a promising alternative for power system control; however, most existing RL-based studies develop problem-specific formulations tailored to a single application, such as optimal power flow, voltage regulation, feeder reconfiguration, or even cyberattack mitigation. Consequently, there remains a lack of a systematic and general methodology that bridges conventional stochastic optimization models with RL by providing a principled transformation from optimization problems to MDPs. This gap restricts the broader adoption of RL in power system optimization, as each new application often requires a complete redesign of the state space, action space, reward function, and learning framework.

Although existing frameworks have addressed FDI attacks in power systems, many conventional approaches rely on predefined optimization formulations or application-specific control strategies, which may require substantial computational effort to obtain new control decisions under changing operating conditions. Similarly, existing RL-based approaches are often tailored to specific applications and lack a systematic procedure for transforming general stochastic optimization problems into MDPs. These limitations motivate the development of a unified and reusable framework that combines stochastic optimization with RL. Unlike application-specific formulations, the proposed framework systematically derives the MDP components from the underlying optimization model and employs an actor-critic algorithm to learn RAS for mitigating FDI attacks in renewable-integrated power systems. This approach provides a systematic foundation for extending RL-based decision-making to other power system optimization and control applications.

To address limitations referred to above, this paper proposes a general RL framework that systematically transforms a broad class of multi-period stochastic power system optimization problems into MDPs that can be efficiently solved using actor-critic RL. The proposed methodology is demonstrated through the design of a RAS against FDI cyberattacks in renewable-integrated power systems, while remaining sufficiently general to be extended to other optimization problems, including optimal power flow, feeder reconfiguration, and network loss minimization. Therefore, the main features of this paper are itemized hereunder.

• A unified three-stage framework is proposed to systematically reformulate multi-period stochastic power system optimization problems as RL problems, establishing a general bridge between mathematical optimization and sequential decision-making.
• A comprehensive stochastic optimization model is developed for RAS against FDI cyberattacks that simultaneously considers thermal generation costs, renewable generation costs, renewable generation mismatch penalties, energy storage operation, and network operational constraints under renewable uncertainty.
• A generic MDP formulation is introduced by systematically defining the state space, action space, state transition mechanism, and reward function directly from the optimization model, providing a reusable methodology that is not restricted to a specific power system application.
• An actor-critic RL algorithm is employed to learn optimal remedial control policies directly from historical operating data without requiring an explicit mathematical model of system dynamics, thereby enabling adaptive and scalable real-time decision-making.

Unlike existing application-specific RL formulations, the proposed framework establishes a generic foundation that facilitates the adoption of modern RL techniques across a wide range of uncertainty-aware power system operation and control problems.

1.3. The Remaining Sections of This Paper

The rest of the paper is organized as follows: Section 2 presents the proposed RL-based framework and the associated problem formulation, which includes three stages of presenting the problem as a stochastic non-linear optimization problem, representing the problem as an MDP, and solving the proposed MDP via RL technique. Section 3 provides case studies including the modified versions of IEEE 33-bus distribution system and IEEE 118-bus interconnected transmission network, initializations of both case studies covering three scenarios for each system, the simulation results, and the corresponding analyses and discussions. Finally, Section 4 concludes the paper.

2. Proposed Framework and Problem Formulation

An overview of the proposed RAS, which is oriented toward an RL approach, is displayed in Figure 1. The mathematical modelling of each stage is provided in the following subsections.

Figure 1. An overview of the proposed framework.

2.1. Stage 1 (i.e., Presenting the Problem as a Stochastic Non-Linear Optimization Problem)

The framework presented in Figure 1 provides a general procedure that can potentially be adapted to different multi-period stochastic optimization problems in power systems. This section proposes the formulation of Stage 1 (see Figure 1) as the first stage of solving the RAS introduced in [15]. In this regard, the main objective function of the problem is presented in (1) and (2), where the first term of (2) indicates the total cost associated with thermal units, the second term of (2) denotes the direct power generation cost for wind units, and the third term of (2) shows the direct power generation cost related to solar panels. It is noted that if the system does not encompass RDG units (e.g., wind turbine and PV panels), the second and third terms of the objective function should be removed.

min{f}

(1)

$$f={\displaystyle {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{i=1}^{N_G}{a}_i\times {\left(P_i^t\right)}^2+b_i}}\times \left(P_i^t\right)+c_i+\left|d_i\times \sin e_i\times \left(P_{i,\min }^t-P_i^t\right)\right|+{\displaystyle {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{j=1}^{N_W}{w}_j\times P_j^t}}+{\displaystyle {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^{N_S}{s}_k\times P_k^t}}$$

(2)

where f is the objective function to be optimized; T is the number of time intervals; N_G, N_W, and N_S are, respectively, the number of thermal units, wind turbines, and solar panels; P_i^t denotes the output active power of ith thermal unit at tth time slot; P_j^t and P_k^t are, respectively, the scheduled powers of jth wind turbine and kth solar panel at tth time interval; and w_j and s_k are constant coefficients associated with jth wind turbine and kth solar panel, respectively.

To address the intermittent nature of RDGs, the total cost related to wind turbines and solar panels needs to be divided into two parts. The first part, which is called the direct power generation cost, is modeled as the second and third terms of (2). The second part of the cost, called the mismatch cost, needs to be considered to address the power overestimation or underestimation. Hence, (3) and (4) present, respectively, the overestimation and underestimation power generation costs for jth wind turbine. Likewise, the overestimation and underestimation power generation costs associated with kth PV unit are, respectively, presented in (5) and (6) [16,17].

$$C_j^t\odot =E\left\{{\alpha }_j\times \left(P_j^t-P_{a,j}^t\right)\right\}$$

(3)

$$C_j^t\otimes =E\left\{{\beta }_j\times \left(P_{a,j}^t-P_j^t\right)\right\}$$

(4)

$$C_k^t\odot =E\left\{{\alpha }_k\times \left(P_k^t-P_{a,k}^t\right)\right\}$$

(5)

$$C_k^t\otimes =E\left\{{\beta }_k\times \left(P_{a,k}^t-P_k^t\right)\right\}$$

(6)

where E (.) is the expectation function; α and β are constant coefficients for overestimation and underestimation costs, respectively; P^t_a,j is the available power of jth wind turbine at tth time interval; and P^t_a,k is the available power of kth solar panel at tth time interval.

The objective function proposed in (1)–(6) needs to be minimized subject to satisfying a set of technical constraints, which are provided in (7)–(21).

$$P_m^t={\left(V_m^t\right)}^2\times G_{\mathit{mm}}+{\displaystyle \sum _{n=1}^{N_B}{V}_m^t\times V_n^t\times \left(G_{\mathit{mn}}\cos {\delta }_{\mathit{mn}}^t+B_{\mathit{mn}}\sin {\delta }_{\mathit{mn}}^t\right)};\forall m$$

(7)

$$Q_m^t=-{\left(V_m^t\right)}^2\times B_{\mathit{mm}}+{\displaystyle \sum _{n=1}^{N_B}{V}_m^t\times V_n^t\times \left(G_{\mathit{mn}}\sin {\delta }_{\mathit{mn}}^t-B_{\mathit{mn}}\cos {\delta }_{\mathit{mn}}^t\right)};\forall m$$

(8)

$$P_{\mathit{mn}}^t={\left(V_m^t\right)}^2\times G_{\mathit{mn}}-V_m^t\times V_n^t\times \left(G_{\mathit{mn}}\cos {\delta }_{\mathit{mn}}^t+B_{\mathit{mn}}\sin {\delta }_{\mathit{mn}}^t\right)$$

(9)

$$Q_{\mathit{mn}}^t=-{\left(V_m^t\right)}^2\times B_{\mathit{mn}}-V_m^t\times V_n^t\times \left(G_{\mathit{mn}}\sin {\delta }_{\mathit{mn}}^t-B_{\mathit{mn}}\cos {\delta }_{\mathit{mn}}^t\right)$$

(10)

$$P_{i,\min }\le P_i^t\le P_{i,\max };\forall i$$

(11)

$$Q_{i,\min }\le Q_i^t\le Q_{i,\max };\forall i$$

(12)

$$V_{m,\min }\le V_m^t\le V_{m,\max };\forall m$$

(13)

$$S_{\mathit{mn},\min }\le S_{\mathit{mn}}^t\le S_{\mathit{mn},\max }$$

(14)

$$\begin{array}{cc}{P}_j^t=\left\{\begin{array}{c}0\\ P_{j,\max }\times {\left({\displaystyle \frac{v^t}{v_{\mathit{rated}}}}\right)}^3\\ P_{j,\max }\end{array}\right.& \begin{array}{l}{v}^t\ge v_{c-o}\\ v_{c-i}\le v^t\le v_{\mathit{rated}}\\ v_{\mathit{rated}}\le v^t\le v_{c-o}\end{array}\end{array}$$

(15)

$${\left(P_j^t\right)}^2+{\left(Q_j^t\right)}^2\le S_j^2$$

(16)

$$\begin{array}{cc}{P}_k^t=\left\{\begin{array}{l}{\displaystyle \frac{{\left(I^t\right)}^2}{I_{\mathit{STD}}\times I_C}}\times P_{S,\mathit{rated}}\\ {\displaystyle \frac{{\left(I^t\right)}^2}{I_{\mathit{STD}}}}\times P_{S,\mathit{rated}}\end{array}\right.& \begin{array}{c}{I}^t\le I_C\\ I^t\ge I_C\end{array}\end{array}$$

(17)

$$P_k^t\le P_{k,\max };\forall k$$

(18)

$$P_x^{t+1}\Delta t-P_x^t\Delta t=\epsilon \times P_x^t;P_x^t\ge 0$$

(19)

$$P_x^{t+1}\Delta t-P_x^t\Delta t={\displaystyle \frac{P_x^t}{\varphi }};P_x^t\le 0$$

(20)

$$P_{x,\min }\le P_x^t\le P_{x,\max }$$

(21)

where P^t_m and Q^t_m are, respectively, the active and reactive power for bus m at tth time interval; V^t_m and V^t_n are the voltage magnitude of buses m and n at tth time interval; δ^t_mn is the voltage phase angle between buses m and n; G_mn and B_mn are, respectively, the conductance and susceptance of tie-line connecting buses m and n; P^t_mn and Q^t_mn are the active and reactive power flowing in the line connecting buses m and n, respectively; Q_i^t is the output reactive power of ith thermal unit at tth time slot; S^t_mn is the apparent power of the line connecting buses m and n; P_j,max is the rated power of jth wind turbine; v^t denotes the wind speed at tth time interval; v_c-i, v_c-o, and v_rated are the parameters of the wind turbines; Q^t_j is the reactive power for jth wind turbine at tth time slot; I^t is the solar irradiance at tth time interval; I_STD indicates the solar irradiance in the standard environment; P_S,rated is the rated power generated by solar panel; P_k,max is the maximum power of the PV unit; P^t_x and P_x,max are, respectively, the power of battery at tth time slot and the maximum limit on it; $P_x^t\Delta t$ and $P_x^{t+1}\Delta t$, respectively, denote the state-of-charge (SOC) associated with the xth storage system at the tth time interval; ε and ϕ are, respectively, the charging and discharging efficiencies.

Although the proposed formulation provides a general procedure for transforming multi-period stochastic optimization problems into an MDP and subsequently solving them using RL, the numerical validation in this work is specifically conducted for the RAS problem against FDI attacks. The same framework can potentially be adapted to other power system optimization problems, such as optimal power flow, feeder reconfiguration, and loss minimization, by appropriately redefining the state variables, control actions, constraints, and reward function according to the specific problem. However, since these structurally different applications are not investigated in the present study, their applicability is considered a potential extension rather than an experimentally validated capability of the proposed framework.

2.2. Stage 2 (i.e., Representing the Problem as an MDP)

Markov decision process (MDP) is a common technique in order to model RL problems [17,18]. Technically, the MDP is a combination of four different parts including the set of states (i.e., U), the set of actions (i.e., A), the probability of the transition from the current state to the next state (i.e., H) after taking action a^t at state u^t, and the set of rewards (i.e., R). To represent the problem (i.e., (1)–(21)) as an MDP, each part is elaborated as follows:

• The state u^t has eight components, which are P^t_D, Q^t_D, P_i^t, P_j^t, P_k^t, P_x^t, V_m^t, and S^t_mn, where P^t_D and Q^t_D are, respectively, the active and reactive power associated with demand at the tth time interval.
• The action a^t includes three components, which are P^t_m, Q^t_m, and Q_j^t. The action space defines the controllable operating decisions selected by the RL agent at each time interval. Specifically, the action a^t comprises the active and reactive power control variables referred to above. These variables are applied to the system model, and the resulting operating condition is evaluated using the network power flow equations and the operational constraints in (7)–(21). The active power dispatch of controllable generators is implemented through their corresponding active power setpoints, subject to their generation limits and the system power balance requirements. The BESS operation is represented by the signed active power variable P^t_x, which determines whether the storage unit charges or discharges. The sign of P^t_x is interpreted according to the convention adopted in the simulation model, and its magnitude is restricted by the maximum charging and discharging power limits. The resulting SOC is updated using the storage energy balance, accounting for the charging and discharging efficiencies and the associated SOC limits. Accordingly, generator dispatch and BESS operation are incorporated into the sequential decision-making process through the defined control variables and the system constraints, rather than being treated as independent decisions outside the MDP. Variables such as bus voltages and line power flows are determined by the network equations in response to the selected controls and the prevailing operating conditions. The reward function then evaluates the resulting operating cost and constraint violations, guiding the RL agent toward economically desirable and operationally feasible decisions.
• The probability h^t presents the probability of state transition from u^t to u^t+1, considering the randomness of the environment. It is noted that the proposed RL approach uses the historical data in order to learn the transition procedure (see Section 2.3).
• The single-step reward r^t after action a^t, which has been accomplished in state u^t, is defined in (22) and (23), where γ is the penalty applied while the voltage and/or reactive power exceed their limits; η is the penalty applied when the boundaries of energy storage system (e.g., battery in this paper) exceed the limits; and f is the main objective function (i.e., (2)).

r^t = f + γ^t + η^t

(22)

$$\begin{array}{cc}{\eta }^t=\left\{\begin{array}{l}-\kappa \times \left(P_x^t\Delta t-90\%\right)\\ 0\\ -\kappa \times \left(10\%-P_x^t\Delta t\right)\end{array}\right.& \begin{array}{l}{P}_x^t\Delta t\ge 90\%\\ 10\%\le P_x^t\Delta t\le 90\%\\ P_x^t\Delta t\le 10\%\end{array}\end{array}$$

(23)

It is important to distinguish the state variables, control actions, and endogenous system variables in the proposed MDP formulation. The state u^t represents the observable operating condition of the system at time t, including the demand, thermal generation, renewable generation, storage status, and relevant network operating variables. These variables provide the information required by the agent to characterize the current system condition. In contrast, the action a^t consists only of the controllable decision variables selected by the RL agent, namely the active and reactive power control variables defined above. Other quantities, such as bus voltages, line power flows, and the resulting renewable power outputs, are endogenous variables that are determined by the system equations and the selected action rather than being independent control decisions. With this definition, the Markov property is satisfied because, conditioned on the current state u^t and action a^t, the distribution of the next state u^t+1 depends only on the current operating condition and the stochastic evolution of demand and RDGs, rather than on the complete sequence of preceding states and actions.

Therefore, at each time interval (i.e., t), the agent needs to make the decision a^t (i.e., the action at the same time step) according to the observation of the state u^t resulting in the reward r^t. In the next step, the state u^t transfers to u^t+1 based on the MDP. To consider the uncertainty of the environment, the reward obtained by the agent after action a^t in state u^t is provided in (24), where λ is a discount factor that determines the relative importance of future rewards in the cumulative return. As a result, the main objective of RL in this paper is to learn a policy mapping state u^t to action a^t via maximizing the reward r^t.

$$r^t={\displaystyle \sum _{y=0}^{T-t}{\lambda }_y\times r^{t+y}}$$

(24)

The relationship between the original stochastic optimization problem and the RL formulation is established by defining the reward as the negative of the operational cost together with penalties associated with constraint violations. Specifically, since f in (2) represents the total operating cost to be minimized, maximizing the cumulative reward in the RL framework is equivalent to minimizing the corresponding cumulative operating cost when all constraints are satisfied. The penalty terms in (22) impose additional costs when the operating constraints, such as voltage, reactive power, or energy storage limits, are violated, thereby discouraging infeasible actions. The penalty coefficients are selected to be sufficiently large to prioritize constraint satisfaction while avoiding excessive distortion of the original economic objective. Furthermore, the discount factor λ determines the relative importance assigned to future rewards compared to immediate rewards and does not directly represent the uncertainty of the environment. The uncertainty associated with demand and RDGs is instead captured through the stochastic state transition process learned from historical operating data.

2.3. Stage 3 (i.e., Solving the Proposed MDP via RL Technique)

To effectively solve the problem introduced in (1)–(24), the multi-period optimization problem is reformulated as an MDP, which contains finite time intervals. Hence, the problem can be sequentially solved via RL algorithms. Adopted from [19,20], this section presents the actor-critic-based algorithm to solve the proposed MDP. Toward this end, the actor is a policy mapping state u^t to the action a^t. However, the critic is a value mapping state u^t to a number that measures the quality of the input state. The solution methodology is presented in (25)–(28). In (25), ∇r_π adjusts the strategy in such a way that the probability of action is enhanced with a higher value of reward. To increase the learning efficiency, r^t_z in (25) can be replaced with the temporal-difference error, which is presented in (26). This is due to the fact that the main parameter of the actor’s network (i.e., ∇r_π) is updated dynamically compared to updating only after an iteration is finalized. In (27) and (28), a minimization problem is introduced to optimize the parameters of value function (i.e., V(u^t,a^t)). It is noted that, in the proposed strategy, each batch of data is used to update π only one time.

$$\nabla r_{\pi }={\displaystyle \frac{1}{Z}}\times {\displaystyle \sum _{z=1}^Z{\displaystyle \sum _{t=1}^T{r}_z^t\times \nabla \mathit{lgh}\left(a_z^t|u_z^t\right)}}$$

(25)

V(u^t,a^t) = r(u^t,a^t) + λθ(u^t+1) − θ(u^t)

(26)

ω = E(θ(u^t) − r(u^t,a^t) + λθ(u^t+1))

(27)

min{ω}

(28)

where Z is the number of trajectories; $h\left(a_z^t|u_z^t\right)$ is the probability of taking action a^t in state u^t under the policy π; $\nabla \mathit{lgh}\left(a_z^t|u_z^t\right)$ indicates the direction enhancing the probability of selecting action a^t_z in state u^t_z; r^t_z is the reward showing the state improvement in probability; r(u^t,a^t) is the immediate reward; θ(u^t) and θ(u^t+1) are, respectively, the expected rewards that the agent can obtain after visiting states u^t and u^t+1; and ω is the critic network parametrized in order to map state u^t to θ(u^t). Figure 2 covers the entire framework for obtaining a holistic perspective.

Figure 2. The proposed RL-based approach including all three stages.

As illustrated in Figure 2, the input to the proposed actor-critic framework consists of power system operating information, including conventional generation outputs, RDG availability, BESS states, load demand, and relevant network operating conditions. These data are used to construct the state vector of the MDP, while the actor generates remedial control actions based on the observed state. The actions are applied to the power system environment, which evaluates the resulting operating conditions and provides the corresponding next state and reward to the agent. This feedback supports the iterative updating of the actor and critic networks during training. The directional arrows in Figure 2 therefore represent the sequential interaction between the agent and the environment rather than an uncontrolled or infinite process.

2.4. False Data Injection Attack Modeling

To evaluate the cyber resilience of the proposed RL-based RAS, an FDI attack model is incorporated into the MDP framework. Unlike random measurement errors, an FDI attack intentionally manipulates a subset of SCADA or phasor measurement unit (PMU) measurements before they are delivered to the control center. Consequently, the RL agent receives corrupted observations and must determine corrective control actions despite the presence of misleading system information.

Let $z_t={\left[z_{1,t},z_{2,t},z_{3,t},\dots ,z_{N_m,t}\right]}^T$ denote the vector of true measurements collected at time interval t, where N_m is the total number of available measurements, including bus power injections, branch power flows, voltage magnitudes, RDG outputs, and load demands. Under normal operating conditions, z_t can be expressed as (29), where x_t denotes the true system state at tth time interval, h(.) is the nonlinear measurement function, and e_t represents measurement noise at tth time interval.

z_t = h(x_t) + e_t

(29)

During an FDI attack, an adversary modifies only a subset of measurements by adding an attack vector (30) at time t resulting in the compromised measurement vector ${\widehat{z}}_t=z_t+a_t$.

$$a_t={\left[a_{1,t},a_{2,t},a_{3,t},\dots ,a_{N_m,t}\right]}^T$$

(30)

The attack vector satisfies (31), where Ω_s denotes the set of compromised measurements and ∆_s(t) is the injected false value.

$$a_{s,t}=\left\{\begin{array}{c}{∆}_s\left(t\right)\hspace{1em}s\in {\Omega }_s\\ 0\hspace{1em}\mathit{Otherwise}\end{array}\right.$$

(31)

The compromised measurements Ω_ss are then used to construct the observation received by the RL agent ${\widehat{u}}_t$ according to (32), where Γ(.) represents the mapping between the measurement vector and the state variables defined in Section 2.2. Therefore, instead of observing the true state u_t, the agent receives the corrupted observation ${\widehat{u}}_t$.

$${\widehat{u}}_t=\mathsf{\Gamma }\left({\widehat{z}}_t\right)$$

(32)

Hence, the FDI attack magnitude is modeled as (33), where ρ_s denotes the relative attack intensity applied to the sth measurement. Positive values of ρ_s correspond to measurement overestimation, whereas negative values represent measurement underestimation.

∆_s(t) = ρ_sz_s,t

(33)

To emulate realistic FDI cyberattacks, only a limited number of measurements are assumed to be compromised (i.e., |Ω_s|≪|N_s|), which reflects the practical limitation that an attacker can only gain access to a subset of communication channels or intelligent electronic devices. It is noted that N_s denotes the total number of measurements gathered via PMUs, distributed through both power transmission and distribution systems.

Finally, it is important to distinguish the evolution of the physical power system from the evolution of the observations available to the RL agent. The FDI attack directly modifies the measurement vector and, consequently, the observation constructed from that vector, as expressed in (29)–(33). It does not directly alter the actual physical system state. Instead, the physical system evolves according to the network equations and system dynamics under the control actions implemented at each time interval. Since these actions are selected using potentially corrupted observations, an FDI attack may indirectly affect the physical operating condition by inducing inappropriate control decisions. For example, compromised measurements may lead to unsuitable generator redispatch, BESS charging/discharging, or reactive power support, potentially resulting in deviations in bus voltages, power flows, and other operating variables.

Accordingly, the transition in (34) represents the evolution of the observations available to the RL agent under measurement corruption and should be distinguished from the underlying physical system transition. The attacked observation ${\widehat{u}}_t$ is used by the agent to select its control action, whereas the resulting physical operating condition is determined by applying that action to the power system model. This distinction is maintained during post-attack recovery, when the RL agent continues to make decisions based on the information available to it while the actual network responds to the implemented controls.

The recovery performance is assessed using the state variables and reference quantities. Therefore, the recovery metrics are calculated from the actual physical states generated by the power system simulation rather than directly from the compromised observations. This evaluation distinguishes successful restoration of the physical operating condition from an apparent improvement caused merely by changes in the corrupted measurements. The attack intensity, compromised measurement locations, and attack duration are varied during testing to assess the generalization of the proposed RAS to attack scenarios not encountered during training.

$${\widehat{u}}_{t+1}~H\left({\widehat{u}}_{t+1}|{\widehat{u}}_t,a_t\right)$$

(34)

To account for stealthy FDI attacks, the injected vector is designed to remain sufficiently consistent with the underlying measurement model and therefore avoid producing an abnormally large state-estimation residual. In particular, for a candidate attack vector a_t, the compromised measurement vector is required to satisfy a residual-based stealthiness condition, such that the residual associated with ${\widehat{\mathit{z}}}_{\mathit{t}}={\mathit{z}}_{\mathit{t}}+{\mathit{a}}_{\mathit{t}}$ remains below the detection threshold of the considered bad-data detection procedure. Thus, the attack is not treated merely as an arbitrary measurement error; instead, its magnitude and affected measurements are jointly selected to produce a corrupted observation while maintaining consistency with the system measurement model. During the evaluation, the compromised measurement set, attack intensity, and attack duration are also varied independently from the configurations used for training. Consequently, the testing scenarios include previously unseen attack locations, magnitudes, and durations, allowing the proposed RL-based RAS to be evaluated under FDI conditions that differ from those encountered during the learning process.

3. Case Study, Initialization, Simulation Results, and Analyses

3.1. Case Studies

To demonstrate the general applicability of the proposed framework, its effectiveness is evaluated across different test systems, including the modified versions of the IEEE 33-bus distribution system and the IEEE 118-bus interconnected transmission network to cover both transmission and distribution levels.

3.1.1. Test System 1: IEEE 33-Bus Power Distribution System

The first case study is conducted on the modified IEEE 33-bus distribution system, which has been extensively adopted as a benchmark for evaluating optimization, renewable integration, distribution automation, and cyber-physical power system security strategies [21]. The original system consists of 33 buses and 32 distribution branches operating at a nominal voltage level of 12.66 kV, supplying multiple load points. Owing to its realistic operating characteristics and moderate computational complexity, this benchmark provides an appropriate platform for validating sequential decision-making algorithms under uncertain operating conditions. To emulate the characteristics of a modern active distribution network, the original test system is enhanced by integrating RDGs, including PV units and wind turbines, together with corresponding BESSs. Furthermore, time-varying load demand, RDG uncertainty, and forecast errors are considered over multiple scheduling intervals to capture the stochastic nature of practical distribution system operation. Within this environment, FDI cyberattacks are assumed to compromise selected measurement signals associated with system states and RDGs, potentially misleading the control center and degrading the system security. The proposed RL-based RAS (see Figure 2) is then employed to determine optimal corrective control actions that minimize the operating cost while maintaining voltage profiles, line loading, power balance, renewable utilization, and energy storage constraints under both normal and cyberattack conditions.

3.1.2. Test System 2: IEEE 118-Bus Power Transmission System

To demonstrate the scalability and generality of the proposed framework (i.e., Figure 2) beyond distribution networks, a second case study is performed on the IEEE 118-bus transmission system, which is one of the most widely accepted benchmark systems for transmission-level operation and control studies [22]. The benchmark represents a practical large-scale interconnected power network consisting of 118 buses, 186 tie-lines, numerous generating units, transformers, and load buses, thereby providing a significantly higher-dimensional optimization problem than the distribution case. The original system is modified to include utility-scale RDGs, such as wind farms and large PV plants, together with grid-scale BESSs to reflect the operational characteristics of contemporary renewable-rich transmission networks. Multi-period load profiles and renewable generation scenarios are incorporated to account for temporal variability and stochastic uncertainty throughout the scheduling horizon. In addition, coordinated FDI attacks targeting critical measurements used by state estimation and energy management systems are simulated to evaluate the resilience of the proposed methodology under large-scale cyber-physical disturbances. The resulting operating conditions require the RAS to simultaneously regulate generator outputs, reactive power support, renewable dispatch, energy storage operation, and network security constraints while preserving system stability and minimizing the total operating cost.

By validating the proposed RL framework on both the IEEE 33-bus distribution system and the IEEE 118-bus transmission system, the proposed methodology, depicted in Figure 2, demonstrates its applicability across different power system architectures, network scales, and operational environments, thereby highlighting its capability as a unified framework for solving a broad class of stochastic power system optimization problems in the digital era.

3.2. Initialization and Simulation Scenarios

3.2.1. Initialization

To evaluate the effectiveness and generality of the proposed RL framework, comprehensive simulations are conducted on two benchmark power systems, introduced in Section 3.1. The simulations are performed over a 24-h scheduling horizon with one-hour time intervals, allowing the proposed framework to capture the temporal variations in load demand, RDGs, and BESS operation. Historical load, wind speed, and solar irradiance profiles are employed to construct the stochastic operating environment used for RL training. The renewable power outputs are calculated according to the wind turbine and PV models presented in Section 2, while BESSs operate under charging/discharging limits and SOC constraints. Thermal generators are subjected to their generation capacity, reactive power capability, and operational limits. All power flow calculations satisfy the network constraints presented in (7)–(21).

For the cybersecurity assessment, FDI attacks are generated by manipulating selected measurements transmitted to the control center while maintaining physically feasible operating conditions. The injected false measurements affect the operator’s perception of system states, potentially leading to improper dispatch decisions and violations of operational constraints. Different attack intensities are considered by varying the magnitude, duration, and number of compromised measurements. During the learning process, the RL agent interacts with the stochastic environment without prior knowledge of the system transition model. At each decision interval, the agent observes the current operating state, determines the remedial control actions through the actor network, receives the corresponding reward according to (22)–(24), and updates both actor and critic networks using the temporal difference learning strategy described in Section 2.3. The learning process continues until the cumulative reward converges and the obtained policy satisfies the desired operational performance. To ensure statistical reliability, each simulation is repeated using multiple random seeds, and the reported results correspond to the average performance over all independent training runs.

The proposed actor-critic algorithm is implemented using a multilayer perceptron architecture, with two fully connected hidden layers of 128 neurons each for both the actor and critic networks. The actor and critic learning rates were set to 1 × 10⁻⁴ and 1 × 10⁻³, respectively. The model is trained for 2000 episodes, with a maximum of 500 steps per episode. The discount factor is set to 0.99, and the replay buffer capacity is set to 100,000 transitions, with a minibatch size of 64. An exploration strategy based on Gaussian noise is also adopted to encourage action exploration, with the noise variance gradually reduced during training. The network parameters are updated using the Adam optimizer (i.e., a gradient-based optimization algorithm used to train deep learning models) to promote stable and efficient convergence.

3.2.2. Simulation Scenarios

Scenario D1: Normal Operating Conditions in the IEEE 33-Bus Distribution System: The first scenario evaluates the performance of the proposed RL-based RAS under normal operating conditions without any cyberattack. The modified IEEE 33-bus distribution system operates over a 24-h scheduling horizon considering time-varying load demand, wind generation, PV generation, and BESS operation. The objective of this scenario is to validate the effectiveness of the proposed optimization-to-MDP transformation under normal system operation while demonstrating that the RL agent can successfully learn an optimal control policy that minimizes the total operating cost subject to all network constraints. The obtained results, including voltage profiles, SOC for BESSs, renewable power utilization, active power losses, and cumulative operating cost, are compared with those obtained using the original stochastic optimization model to verify that the proposed RL framework achieves comparable operational performance while significantly reducing the computational burden associated with repeated online optimization.

Scenario D2: Localized FDI Attack in the IEEE 33-Bus Distribution System: The second scenario investigates the capability of the proposed framework to mitigate a localized FDI cyberattack targeting the active distribution network. In this scenario, an attacker compromises a limited number of measurement devices associated with selected load buses #7, #14, #24, and #30, RDGs located at buses #18 and #33, and bus voltage measurements at buses #6, #18, #25, and #33, thereby introducing false information into the system monitoring process. It is noted that the attacked measurements were selected to represent a realistic localized cyberattack affecting electrically diverse regions of the feeder. Load buses #7, #14, #24, and #30 capture upstream, mid-feeder, and downstream demand centers, while RDG measurements at buses #18 and #33 represent geographically dispersed distributed energy resources. Voltage measurements at buses #6, #18, #25, and #33 provide observability across the feeder, enabling the injected false data to influence both state estimation and dispatch decisions without assuming a system-wide sensor compromise. The manipulated measurements mislead the control center regarding the actual operating conditions, resulting in improper dispatch decisions if no remedial action is implemented. Upon observing the abnormal system state, the RL agent determines the optimal corrective actions by coordinating thermal generation, renewable power dispatch, BESS charging and discharging, and reactive power support to restore secure operation. The effectiveness of the proposed RAS is evaluated through comparisons of voltage regulation, operating cost, renewable curtailment, network losses, and recovery time before and after the implementation of the RL-based control policy.

Scenario D3: Coordinated Multi-Point FDI Attack in the IEEE 33-Bus Distribution System: The third scenario considers a coordinated FDI cyberattack in which multiple measurement devices distributed throughout the network are simultaneously compromised. Unlike the previous scenario, the attacker manipulates several load measurements at buses #6, #10, #14, #19, #24, #30, and #32 together with RDG outputs at buses #12, #18, #25, and #33 and branch power flow measurements in distribution branches (1–2), (6–7), (13–14), (17–18), (24–25), and (32–33) to create a significantly distorted system operating condition. The compromised measurements were selected to emulate a coordinated FDI cyberattack targeting multiple layers of the distribution monitoring infrastructure. Load measurements were chosen from electrically diverse buses spanning the upstream, mid-feeder, and downstream regions to maximize the impact on system power balance. RDG measurements were compromised at geographically dispersed distributed energy resources to distort available generation estimates and interfere with economic dispatch decisions. In addition, branch power flow measurements along major feeder sections were manipulated to corrupt network observability and congestion assessment. This coordinated FDI attack represents a substantially more challenging threat model than the localized attack considered in Scenario D2 and enables a comprehensive evaluation of the robustness, adaptability, and resilience of the proposed RL-based RAS under severe cyber-physical disturbances. Such a coordinated attack increases the likelihood of voltage violations, branch overloading, unnecessary renewable curtailment, and improper battery scheduling. This scenario is designed to evaluate the robustness and adaptability of the proposed RL framework under severe cyber-physical disturbances. The simulation results demonstrate how the learned policy sequentially performs corrective actions over multiple time intervals to restore normal operation while satisfying all operational constraints. The overall performance is assessed in terms of cumulative reward, operating cost, voltage deviation, line loading, and system recovery trajectory.

Scenario T1: Normal Operating Conditions in the IEEE 118-Bus Transmission System: The fourth scenario extends the validation of the proposed framework to the modified IEEE 118-bus transmission system to demonstrate its scalability and applicability to large-scale power networks. Similar to the distribution case, the transmission system is simulated over a 24-h operating horizon with stochastic load demand, utility-scale wind farms, PV plants, and grid-scale BESSs. The RL agent determines the optimal dispatch of conventional generators, renewable resources, reactive power support, and BESS operation while maintaining transmission security constraints, including bus voltage limits, transmission line thermal limits, and generator operating limits. The obtained operating cost, network security indices, RDG utilization, and computational efficiency are compared with those of the stochastic optimization model to verify that the proposed MDP formulation remains effective as the dimensionality and complexity of the optimization problem increase.

Scenario T2: Coordinated False Data Injection Attack in the IEEE 118-Bus Transmission System: The fifth scenario evaluates the resilience of the proposed framework against coordinated FDI attacks targeting the transmission-level energy management system. The attacker manipulates several critical measurements used by state estimation, including bus power injections at buses #17, #30, #49, #66, #80, and #100, tie-line power flows in lines (8–30), (30–38), (49–54), (65–68), (80–81), and (100–103), and generator output measurements at buses #10, #25, #54, #65, #87, and #111, thereby creating incorrect estimates of the system operating state. The compromised measurements were selected to emulate a coordinated transmission-level cyberattack targeting geographically dispersed substations and critical monitoring points within the IEEE 118-bus system. Bus power injection measurements were manipulated at electrically distributed buses to introduce widespread state estimation errors, while tie-line power flow measurements along major transmission corridors were falsified to distort congestion assessment and inter-area power transfer estimates. Generator output measurements at multiple generating stations were simultaneously compromised to induce incorrect economic dispatch and reserve allocation decisions. This coordinated measurement selection provides a realistic and challenging attack model that significantly degrades energy management system observability while remaining consistent with practical assumptions regarding limited attacker access to critical measurement devices. Without appropriate remedial actions, these manipulated measurements result in generator redispatch errors, tie-line congestion, voltage instability, and increased operating costs. After detecting the altered operating conditions through the observed system states, the RL agent performs sequential remedial actions by redispatching generation resources, coordinating BESSs, regulating reactive power support, and adjusting RDG schedules. The performance of the proposed framework is evaluated using voltage security indices, line loading, generation cost, cumulative reward, and computational time, demonstrating its capability to rapidly restore secure operation under large-scale cyberattacks.

Scenario T3: Extreme Cyber-Physical Operating Conditions in the IEEE 118-Bus Transmission System: The final scenario presents an extreme operating condition designed to evaluate the robustness of the proposed RL framework under simultaneous cyber and physical disturbances in a larger power system. In this scenario, the transmission system experiences a coordinated FDI attack during a period of high system loading, large RDG forecast errors, and limited BESS availability. The simultaneous occurrence of these events creates a highly uncertain operating environment that challenges both system security and economic operation. The proposed RL-based RAS continuously observes the evolving system states and learns an adaptive sequence of corrective actions to minimize the overall operating cost while maintaining voltage stability, preventing transmission line overloads, maximizing RDG utilization, and respecting all operational constraints. The results obtained from this scenario provide a comprehensive assessment of the proposed framework under realistic emergency conditions and demonstrate its suitability for real-time decision-making in future renewable-integrated cyber-physical power systems.

3.3. Obtained Simulation Results

The proposed RL-based RAS was evaluated using the modified IEEE 33-bus distribution system and the modified IEEE 118-bus transmission system under the six operating scenarios introduced in Section 3.2.2. In all simulations, the proposed actor-critic algorithm was trained using historical operating data while considering stochastic load demand, RDG uncertainty, BESS operation, and the possibility of FDI cyberattacks. The RL performance was compared with the stochastic optimization model formulated in Stage 1 of Figure 1 and Figure 2 to evaluate both the quality of the obtained solutions and the computational efficiency of the proposed optimization-to-MDP transformation.

The convergence behavior of the proposed RL algorithm is illustrated in Figure 3, which depicts the average cumulative reward during the training process for both benchmark systems, as presented in (25)–(28).

Figure 3. Convergence of the proposed actor-critic RL algorithm on both test systems.

As observed in Figure 3, the cumulative reward increases rapidly during the early training episodes as the agent explores different operating strategies. After approximately 400 episodes, the learning process gradually stabilizes, indicating that the actor and critic networks successfully converge toward an optimal policy. The relatively smooth convergence curves demonstrate that the proposed reward formulation (see the objective function (28), elaborated in Section 2.3) effectively balances economic objectives and operational security constraints while avoiding unstable learning behavior. Moreover, the convergence characteristics, displayed in Figure 3, remain similar for both the distribution and transmission systems despite the significantly larger state and action spaces associated with the IEEE 118-bus network compared to the IEEE 33-bus network, confirming the scalability of the proposed RL framework.

The economic performance of the proposed framework under normal operating conditions is presented in Figure 4, where the total operating costs obtained using the stochastic optimization model and the proposed RL approach are compared for both test systems. The results indicate that the RL agent successfully learns dispatch policies that closely approximate the optimal solutions obtained by stochastic optimization while requiring considerably lower online computational effort. The minor differences observed between the two approaches originate primarily from the exploration process inherent to RL and diminish as additional training episodes are performed. These results, depicted in Figure 4, verify that the proposed optimization-to-MDP transformation preserves the original optimization objective while enabling efficient real-time decision-making.

Figure 4. Comparison of total operating cost obtained by stochastic optimization and the proposed RL framework for both test systems.

To provide a better perspective, the computational performance of the proposed methodology is summarized in Figure 5, which compares the average execution time required by the stochastic optimization model and the proposed RL framework during online operation. While the optimization model repeatedly solves a nonlinear stochastic optimization problem whenever operating conditions change, the trained RL agent determines the corresponding control actions through a simple forward evaluation of the actor network. Consequently, the proposed framework achieves a considerable reduction in online computational time while maintaining solution quality that closely approximates the optimization-based benchmark. The results depicted in Figure 5 demonstrate the practical suitability of the proposed methodology for real-time RAS in renewable-integrated smart power systems, where rapid decision-making is essential for maintaining system security during cyberattacks.

Figure 5. Comparison of online computational time between stochastic optimization and the proposed RL framework on both test systems.

As shown in Figure 4 and Figure 5, the proposed RL-based approach effectively handled the operating conditions while achieving nearly identical total operating costs to those obtained using the stochastic optimization algorithm (see Figure 4). Furthermore, the proposed framework reduced the computation time by 99.6% for the IEEE 33-bus system and 99.8% for the IEEE 118-bus system (refer to Figure 5). These results underscore the significance of the proposed RL-based methodology, illustrated in Figure 2, as power system operators require fast and reliable decision-making tools to manage power grids efficiently on a daily basis.

The voltage profiles obtained before and after implementing the proposed RAS under the localized FDI attack of Scenario 2D are illustrated in Figure 6.

Figure 6. Bus voltage profiles before and after applying the proposed remedial action scheme in Scenario 2D on the IEEE 33-bus distribution system.

From Figure 6, prior to applying the RL-based corrective actions, several buses experience noticeable voltage deviations due to manipulated measurement data, resulting in violations of the allowable operating limits. After the proposed RL policy is activated, the voltage magnitudes are restored within their acceptable ranges through coordinated adjustments of RDG, reactive power support, and energy storage dispatch. As an example, the voltage magnitude at the end of the IEEE 33-bus system increased from 0.924 p.u. to 0.965 p.u., representing a 4.44% improvement and demonstrating a significant enhancement in the voltage profile at the end of the feeder (i.e., Bus #33). This improved voltage regulation demonstrates the capability of the learned policy to effectively compensate for the adverse effects of the FDI attacks without requiring repeated online optimization.

The effectiveness of the proposed framework in relieving network congestion is demonstrated in Figure 7, which presents the line loading of test system 1 in Scenario D3.

Figure 7. Distribution line loading under normal and attack conditions related to Scenario D3 on the IEEE 33-bus distribution system.

According to Figure 7, before the implementation of the proposed RAS, several branches operate near or above their thermal limits (as highlighted by the dark red circles enclosed within the green dashed box in Figure 7) because the compromised measurements lead to improper power flow distributions. After the RL agent performs sequential control actions, the loading levels of the critical distribution branches decrease significantly, indicating that the proposed policy successfully redistributes power flows while satisfying all network security constraints. Similar improvements are observed for the IEEE 118-bus transmission network, confirming that the proposed methodology remains effective regardless of network topology or system scale.

The dynamic behavior of the BESSs throughout the 24-h scheduling horizon is shown in Figure 8 for test system 2 (i.e., IEEE 118-bus transmission network). During periods of high renewable generation, the batteries absorb surplus energy and increase their SOC, whereas they discharge during peak-demand intervals and immediately after the FDI cyberattack to support voltage regulation and reduce generation costs. The resulting charging/discharging schedules remain within the operational limits of the storage units, demonstrating that the proposed RL framework successfully coordinates BESSs with conventional generation and RDGs while respecting all technical constraints included in the optimization model.

Figure 8. SOC for BESSs over the 24-h scheduling horizon associated with Scenario T2 on the IEEE 118-bus transmission network.

According to Figure 8, it can be concluded that the SOC decreases gradually from 50% at 00:00 to a minimum of 35% at 07:00, reflecting BESS utilization during the early hours when renewable generation is limited. During the designated charging period (09:00–15:00) and around 1:00 PM, the SOC rises rapidly from 55% to approximately 90%, indicating efficient storage of surplus renewable energy while remaining within the prescribed operational limit of 20–90% (see constraint (21)). Following the FDI cyberattack event at 16:00 (see the red dashed line in Figure 8), the BESS transitions into a sustained discharge mode, with the SOC decreasing from 84% to 34% by 21:00 to supply the increased evening demand and mitigate peak loading. Despite the FDI attack, the SOC exhibits a smooth and controlled trajectory without abrupt fluctuations, suggesting that the BESS management strategy maintains stable operation and preserves sufficient energy reserves for system support. By the end of the daily cycle, the SOC recovers slightly to 48% at 23:00, indicating readiness for the subsequent operating period while consistently adhering to the established safety limits.

To complete the results shown in Figure 8 and also obtain a deeper perspective on the available and scheduled renewable dispatches, the power utilization achieved by the proposed framework is illustrated in Figure 9, where the available renewable generation is compared with the scheduled dispatch throughout the scheduling horizon.

Figure 9. Available renewable generation versus scheduled renewable dispatch associated with Scenario T2 on the IEEE 118-bus transmission network.

Based on Figure 9, the RL agent effectively adapts to the stochastic behavior of wind and solar resources by continuously adjusting the dispatch decisions according to the observed system conditions. Consequently, renewable curtailment remains limited even during periods of significant uncertainty and cyber disturbances, while power balance and network security are maintained. These results, shown in Figure 8 and Figure 9, demonstrate that the proposed reward function appropriately captures both the economic value of renewable generation and the penalties associated with renewable mismatch.

An important observation from Figure 9 is that the proposed RL-based energy management strategy maintains high renewable energy utilization throughout the 24-h scheduling horizon, even in the presence of the FDI attack that occurred at 16:00. The available renewable generation increases from 18 MW at 00:00 to a peak of 78 MW at 12:00, while the scheduled renewable dispatch closely follows this profile, reaching 75 MW at the same time. The maximum difference between the available and dispatched renewable power is only 3 MW, occurring during the period of highest renewable production, indicating minimal renewable curtailment. After the FDI cyberattack, the dispatch continues to closely track the available generation, decreasing smoothly from 61 MW to 19 MW between 15:00 and 23:00 without noticeable deviations or abrupt changes. This behavior suggests that the proposed control framework successfully mitigates the impact of manipulated data by preserving accurate renewable scheduling, maximizing renewable energy utilization, and preventing unnecessary curtailment. Consequently, the results shown in Figure 9 indicate that the RL-based strategy enhances both the operational efficiency and cyber resilience of the power grid by ensuring stable renewable power integration under FDI attack conditions.

The spatial distribution of compromised measurements under the coordinated FDI attack scenario (see Scenario T2 elaborated upon in Section 3.2.2) is presented in Figure 10. This figure highlights the measurement locations intentionally manipulated by the attacker, including bus injections, renewable generation outputs, line flows, and load measurements. Although the injected false data substantially distort the perceived operating conditions, the proposed RL framework remains capable of identifying appropriate corrective actions based solely on the observed system states and the learned interaction with the environment. This observation confirms that the proposed methodology does not require explicit knowledge of the underlying cyberattack model or system transition probabilities on a relatively large-scale IEEE 118-bus power transmission system. The sequence of remedial actions selected by the RL agent following the occurrence of a severe FDI attack under Scenario T3 is presented in Figure 11.

Figure 10. Locations of compromised measurements under FDI attacks in Scenario T2.

Figure 11. Sequential remedial control actions selected by the proposed RL agent in Scenario T3.

Immediately after the FDI attack is detected through abnormal state observations, which is not in the scope of this paper, the RL policy performs coordinated adjustments of thermal generation, RDG dispatch, BESS charging/discharging, and reactive power support. Hence, according to Figure 11, one can infer that as the system gradually returns to secure operation, the magnitude of the corrective actions decreases until the operating point converges to a new stable equilibrium. The sequential nature of these decisions clearly demonstrates one of the primary advantages of the proposed MDP formulation, namely its ability to continuously adapt the control policy according to the evolving system conditions rather than solving each operating point independently. A key insight from Figure 11 is that the RL agent responds immediately after the FDI attack by coordinating multiple remedial actions rather than relying on a single control variable. The corrective actions are strongest immediately after the attack at 16:00, including a 25 MW BESS discharge, 18 MW thermal generation redispatch, 15 MVAR reactive power support, and a 10 MW reduction in renewable dispatch, before gradually decreasing to zero as the system recovers. This sequential reduction in corrective actions indicates that the RL controller effectively mitigates the cyberattack’s negative impact while avoiding excessive control effort, enabling the power grid to return to the normal operation in a stable and efficient manner.

To complement this observation, the associated transient recovery characteristics of the proposed framework are illustrated in Figure 12, where the evolution of key operational variables is monitored following the initiation of the severe FDI attack in Scenario T3.

Figure 12. Recovery trajectory of the power system following the severe FDI attack introduced in Scenario T3 in the IEEE 118-bus transmission system.

According to Figure 12, it can be perceived that voltage deviations, branch loading, and power imbalance gradually decrease as the RL agent repeatedly interacts with the environment and applies the learned remedial actions (see Figure 2). Compared to the system response obtained without implementing the proposed RAS, the recovery process is significantly faster and exhibits substantially smaller oscillations, thereby improving both operational security and system resilience under cyber-physical disturbances.

Specifically, Figure 12 demonstrates that the proposed RL-based RAS significantly enhances the transient recovery of the power grid following the coordinated severe FDI attack associated with Scenario T3. Immediately after the FDI attack at 1 s, the proposed RL-based RAS (see the blue solid lines in Figure 12) suppresses voltage deviation, branch loading deviation, and power imbalance more rapidly than the conventional scheme without RAS (refer to the red dashed lines in Figure 12), resulting in smaller oscillations and faster convergence to steady-state conditions. These results indicate that the coordinated RL-based corrective actions effectively improve system stability, reduce post-attack disturbances, and accelerate recovery, thereby enhancing the resilience of the IEEE 118-bus system against coordinated FDI attacks.

It is important to distinguish between the decision-making timescale of the proposed MDP-based RL framework and the timescale used to illustrate the system recovery following an FDI attack. The MDP formulation in Section 2.2 and Section 2.3 represents the multi-period operational decision-making process, in which the RL agent observes the current system state, selects a corrective action, and receives a reward based on the resulting operating condition and constraint violations. Following the initiation of the coordinated FDI attack in Scenario T3 at t = 1 s, the RL-based RAS determines corrective actions according to the observed system conditions and the learned policy. These actions modify the system operating point, and the resulting changes in voltage deviations, branch loading, and power imbalance are monitored to assess the recovery process. Accordingly, the second-level trajectories in Figure 12 represent the temporal evolution of these performance indicators following the attack and the implementation of corrective actions, whereas the MDP provides the sequential decision-making framework governing the selection of those actions. The interpretation of these trajectories therefore depends on the simulation model used to represent the system response and the timing with which corrective actions are applied. This distinction clarifies the relationship between the multi-period operational optimization framework and the post-attack recovery results presented in Figure 12.

To sum up, Table 1 summarizes the principal quantitative results obtained from the six simulation scenarios. The results demonstrate that the proposed RL-based RAS maintains operating performance close to the stochastic optimization benchmark while providing substantial reductions in online computational requirements. Under the localized FDI attack, the voltage magnitude at Bus #33 improves from 0.924 p.u. to 0.965 p.u., while under the transmission-level attack, the maximum difference between available and scheduled renewable generation is only 3 MW at the peak renewable generation period. The BESS and coordinated remedial control results further demonstrate that the proposed framework utilizes multiple control resources to support system recovery while maintaining the prescribed operational limits.

Table 1. Summary of the quantitative performance of the proposed RL-based RAS.

Test System/ Scenario	Performance Metric	Stochastic Optimization/Before RAS	Proposed RL-Based RAS/After RAS	Improvement/Observation
IEEE 33-bus/D1	Total Operating Cost ($/day)	1842.60	1861.03	1.00% higher cost; comparable solution quality
IEEE 118-bus/T1		18,450.00	18,634.50
IEEE 33-bus/D1	Online Computational Time (s/decision)	2.50	0.010	99.6% reduction
IEEE 118-bus/T1		15.00	0.030	99.8% reduction
IEEE 33-bus/D2	Bus #33 Voltage Magnitude (p.u.)	0.924	0.965	4.44% improvement; voltage profile restored
IEEE 33-bus/D3	Critical Branch Loading (%)	112.0	94.0	18 percentage-point reduction; loading restored below the thermal limit
IEEE 118-bus/T2	BESS SOC at 08:00 (%)	—	35	Within the prescribed 20–90% operating range
	BESS SOC During Charging (%)	—	55 → 90	Surplus renewable energy stored during charging
	BESS SOC After FDI Attack (%)	—	84 → 34	Supports post-attack recovery while respecting SOC limits
	Available Renewable Generation at 12:00 (MW)	78	75 scheduled	3.0 MW of renewable curtailment
	Maximum Renewable Curtailment (MW)	78 available	75 dispatched	3.0 MW, approximately 3.85% of available generation
	FDI Attack Time	—	16:00	Renewable dispatch continues to track available generation
IEEE 118-bus/T3	BESS Corrective Action at Attack (MW)	—	25 MW discharge	Immediate remedial response
	Thermal Generation Redispatch (MW)	—	18 MW	Coordinated generation adjustment
	Reactive Power Support (MVAR)	—	15 MVAR	Immediate voltage support
	Renewable Dispatch Adjustment (MW)	—	−10 MW	Temporary reduction to support secure operation
	Post-Attack Recovery Time (s)	20.0	8.0	60% reduction in recovery time

It should be noted that the present study primarily benchmarks the proposed RL-based RAS against the stochastic optimization model and the system response without remedial control, rather than against multiple alternative RL or safe-RL algorithms. This comparison was selected because the stochastic optimization model represents the original decision-making formulation from which the proposed MDP is constructed, thereby providing a direct benchmark for evaluating whether the optimization-to-MDP transformation preserves solution quality while reducing online computational requirements. In addition, the computational results reported in Figure 5 specifically quantify the online inference advantage of the trained RL policy; however, this metric does not include the offline training cost, which is incurred only during the policy development stage. The training process is therefore performed offline using historical operating scenarios and does not directly affect real-time remedial decision-making. A systematic comparison with representative model-free RL and safe-RL algorithms, including their training time, sample efficiency, convergence characteristics, constraint-violation frequency, and inference time, will provide a broader assessment of the proposed methodology (reserved for the future step of this research in large-scale distribution network including multi-microgrid systems). Similarly, detailed ablation studies separating the contributions of the reward formulation, BESS coordination, renewable dispatch, reactive power support, and multi-action remedial control will further quantify the contribution of each component. These evaluations constitute important extensions of the present work; nevertheless, the current results demonstrate that the proposed framework maintains solution quality close to the stochastic optimization benchmark while achieving 99.6% and 99.8% reductions in online computational time for the IEEE 33-bus and IEEE 118-bus systems, respectively.

3.4. Discussion

The simulation results obtained from both the IEEE 33-bus distribution system and the IEEE 118-bus transmission system consistently demonstrate the effectiveness, robustness, and scalability of the proposed RL framework, introduced in Figure 2. In this regard, the convergence characteristics verify that the optimization-to-MDP transformation enables stable policy learning despite the stochastic behavior of renewable generation and the high dimensionality of modern power systems. Under both localized and coordinated FDI cyberattacks, the learned remedial action policy successfully restores secure operating conditions by coordinating conventional generation, renewable resources, reactive power support, and BESSs while satisfying all operational constraints. Furthermore, the close agreement between the RL solutions and those obtained from the stochastic optimization model confirms that the proposed framework preserves the original optimization objectives while substantially reducing the computational burden associated with repeated online optimization. These findings indicate that the proposed methodology is not limited to RASs against cyberattacks but provides a general and scalable framework for transforming a broad class of multi-period stochastic power system optimization problems into RL formulations suitable for real-time operation in renewable-rich cyber-physical power systems.

In addition to the cyberattacks considered in this study, other factors, such as actuator faults and faulty communication networks, can also affect the stability and consensus performance of multi-agent systems. For instance, [23] investigated consensus control under actuator faults and network imperfections. These issues highlight the importance of developing robust and adaptive control strategies to enhance the reliability of multi-agent systems under various adverse conditions. Investigating the combined effects of different cyberattacks and physical or communication faults represents a promising direction for future research.

4. Conclusions

This paper presents a general RL-based framework for solving multi-period stochastic power system optimization problems, with a particular focus on the application of a RAS for enhancing the resilience of smart power systems against FDI cyberattacks. The simulation results demonstrate that the proposed RL-based RAS provides an effective and computationally efficient solution for enhancing the cyber resilience of renewable-integrated smart power systems under FDI attacks, validated over two different IEEE test systems with different scales. Specifically,

• Across the modified IEEE 33-bus distribution system and IEEE 118-bus transmission system, the actor-critic RL framework successfully learned operating policies that closely approximated the stochastic optimization benchmark while reducing the online computation time by 99.6% and 99.8%, respectively, making the proposed methodology well suited for real-time applications.
• Under cyberattack conditions, the proposed framework restored the voltage magnitude at Bus 33 from 0.924 p.u. to 0.965 p.u., corresponding to a 4.44% improvement, while simultaneously relieving line congestion through coordinated redispatch of generation, reactive power support, and BESSs.
• The BESS scheduling strategy maintained the SOC within the prescribed 20–90% operating limits throughout the 24-h horizon, and the renewable dispatch closely tracked the available renewable generation with a maximum mismatch of only 3.0 MW, indicating minimal renewable curtailment despite cyber disturbances.
• The sequential corrective actions enabled rapid post-attack recovery by reducing voltage deviations, branch loading, and power imbalance with significantly smaller oscillations than the conventional approach.

Overall, these results confirm that the proposed optimization-to-MDP transformation preserves the quality of optimization-based solutions while providing the computational speed, adaptive decision-making capability, and resilience required for secure operation of modern power systems with high renewable penetration under cyber-physical threats.

Future steps of this research will focus on extending the proposed RL-based RAS of this paper toward more sophisticated cyber-physical power system models and practical experimental validation. One promising direction is the investigation of multi-agent and hierarchical RL frameworks capable of coordinating distributed controllers across interconnected transmission and distribution networks while considering communication delays, partial observability, and decentralized decision-making. The proposed approach in this work can also be enhanced by incorporating graph neural networks and transformer-based architectures to improve scalability for even larger-scale power systems with thousands of buses and heterogeneous renewable resources. From a cybersecurity perspective, future steps should consider adaptive and stealthy cyberattacks evading the detection stage, simultaneous combinations of FDI, denial-of-service, replay, and topology attacks, and adversarial RL techniques to improve robustness against intelligent attackers. In addition, uncertainty modeling can be strengthened through probabilistic forecasting of renewable generation, demand response, electric vehicle charging, and extreme weather events. Finally, the proposed framework in this paper should be validated using hardware-in-the-loop (HIL) and real-time digital simulator (RTDS) platforms, microgrid laboratory testbeds, and utility-scale datasets to assess implementation feasibility, communication requirements, computational performance, and operational reliability under realistic field conditions, thereby facilitating the transition of the proposed methodology from simulation studies to practical deployment in future resilient smart grids.