Advancements and Future Directions in the Application of Machine Learning to AC Optimal Power Flow: A Critical Review

Abstract

Optimal power flow (OPF) is a crucial tool in the operation and planning of modern power systems. However, as power system optimization shifts towards larger-scale frameworks, and with the growing integration of distributed generations, the computational time and memory requirements of solving the alternating current (AC) OPF problems can increase exponentially with system size, posing computational challenges. In recent years, machine learning (ML) has demonstrated notable advantages in efficient computation and has been extensively applied to tackle OPF challenges. This paper presents five commonly employed OPF transformation techniques that leverage ML, offering a critical overview of the latest applications of advanced ML in solving OPF problems. The future directions in the application of machine learning to AC OPF are also discussed.

1. Introduction

Over the past few decades, the optimal power flow (OPF) concept has garnered significant attention in the field of power system operation [1]. First introduced by Carpentier [2], OPF is widely acknowledged as a vital tool for efficiently planning and enhancing the operation of electric power systems. Its primary objective is to determine the optimal or most secure operating point, known as control variables, while ensuring compliance with system constraints and optimizing specific objective functions. However, several challenging developments have emerged in recent years, such as the advent of smart grid (SG) technologies, microgrid systems, distributed energy resources (DERs), and the integration of large-scale renewable energy sources (RESs). These developments have introduced three distinct characteristics for OPF in the future grid.

Firstly, the optimization of power systems is anticipated to shift towards a large-scale framework. Simultaneously, the advanced metering infrastructure (AMI) and two-way communication systems have placed higher demands on real-time performance for power system optimization tasks. The efficiency of conventional optimization tools has demonstrated limitations in meeting these requirements [3].

Secondly, the widespread integration of distributed generations (DGs) such as RES, energy storage systems (ESSs), and fuel cells in power systems has introduced significant security constraints that impede the accuracy of conventional optimization tools. For instance, the Midcontinent Independent System Operator (MISO) oversees one of the world’s largest electricity markets, encompassing over 45,000 buses and 1400 generation resources. Its OPF model encompasses extensive resource-level constraints and system-wide constraints [4]. To simplify the large-scale OPF, MISO has implemented Lagrangian Relaxation (LR), decomposing the problem into smaller subproblems to enhance efficiency [5,6]. With notable advancements in mixed-integer programming (MIP) solvers, MISO has transitioned from LR to MIP in its market clearing engines [7]. The complete OPF model comprises tens of thousands of buses, transmission lines, and thousands of post-contingency scenarios, making it impossible to solve within the market clearing time constraints. Consequently, MISO resorts to solving a reduced OPF model for its daily operations. Furthermore, Chen [8] has highlighted that MIP solvers may not quickly solve the problem, even with an excellent initial solution, as determining optimality for existing commitment solutions can be time-consuming. Hence, there is an urgent need for faster and more accurate OPF solution methods.

Thirdly, OPF problems are typically solved with sampling intervals of a few hours or minutes. Consequently, OPF solutions encompass a multi-period framework that incorporates inter-temporal constraints, such as ramping up and down, minimum on/off time, and others, to ensure a smooth and realistic operational trajectory over time [9,10]. Thus, the inclusion of time-coupling constraints in large-scale problems results in complex, multi-period, mixed-integer programming optimization problems [11,12]. Additionally, RESs exhibit distinct characteristics compared to conventional generating resources, characterized by high stochasticity and intermittency in their production output [13]. The incorporation of RESs into power systems brings about considerable uncertainty in short-term and real-time operations due to its deep integration. Consequently, it is necessary to formulate OPF problems as dynamic/stochastic problems rather than static problems. However, traditional optimization methods have limited efficiency when addressing these multi-period dynamic/stochastic OPF problems [14].

Several classical (deterministic) and recent (nondeterministic) heuristic optimization techniques have been proposed to solve the OPF problem. Classical methods include Newton method network flow programming, linear programming, nonlinear programming, quadratic programming, and the interior point [15,16,17,18]. However, it should be noted that the nonlinearity of OPF problems can cause classical methods to converge to local optima. As a result, metaheuristic optimization techniques are widely employed to address OPF problems. These optimization techniques can be categorized based on their inspirations, such as nature-swarm-inspired methods, human-inspired algorithms, evolutionary-inspired algorithms, physics-inspired algorithms, and artificial neural networks (ANNs).

The advancements in deep architectures of ANNs, particularly in representation learning and data handling capabilities, have contributed to significant progress in various fields, including natural language processing [19,20,21,22], computer vision [23,24,25,26], and power system operation [27,28,29,30,31]. These advancements have enabled the development of data-driven solutions that provide highly accurate numerical solutions in shorter time durations. For example, Nair et al. [32] aimed to incorporate machine learning (ML) techniques into two main sub-tasks of an MIP solver: generating a high-quality joint variable assignment and reducing the objective value gap between this assignment and an optimal solution. Their approach was validated using various real-world datasets, most of which contained between ${10}^3$ and ${10}^6$ variables and constraints. By comparing the performance of SCIP (a solver) in terms of the primal–dual gap within the given time constraints, they demonstrated that one of the best learning-augmented SCIP variations achieved a ${10}^4$ improvement in the gap. Additionally, Xia et al. [33] proposed a bi-projection neural network (BPNN) for solving a class of constrained quadratic optimization problems, which was proven to be globally stable in the sense of Lyapunov. Numerical results showed that BPNN outperformed conventional algorithms in terms of speed. Apart from ANNs, there are various ML algorithms indirectly used for solving the OPF problem, such as support vector machine (SVM) [34,35], random forest [36,37], K-means [38,39], and others.

This article offers a comprehensive review of ML techniques applied to solve AC OPF problems. It introduces how to apply ML into AC OPF problems from five perspectives based on the general mathematical form of AC OPF: the power flow equation, the active constraint of AC OPF, the Markov form of AC OPF, the AC OPF warm-start point prediction, and the optimization process of traditional optimization methods. The techniques are thoroughly compared and discussed, providing researchers with a better understanding of how advanced ML techniques are applied in AC OPF and how they can enhance the efficiency, accuracy, and scalability of AC OPF solutions. Furthermore, in consideration of the future development of power systems, particularly with the emergence of SG, DER, and RES, this review provides an overview of the future development of ML in addressing open problems in AC OPF while acknowledging current limitations, including the need for a hierarchical solution framework to ensure mathematical guarantees, exploring unsupervised learning approaches for enhanced training efficiency, discussing the integration of first principles into the attention mechanism, and utilizing large models.

2. Problem Formulation

The objective of solving the OPF problem is to optimize a specific objective function by making optimal adjustments to power system control variables, while also satisfying several equality and inequality constraints. The mathematical representation of this optimization problem is typically as follows:

$$MinF(x,u)$$

(1)

Subjected to

$$\begin{array}{c}{g}_j(x,u)=0j=1,2,\dots ,m\end{array}$$

(2)

$$\begin{array}{c}{h}_j(x,u)\le j=1,2,\dots p\end{array}$$

(3)

where F acts as the objective function, x is a vector representing the dependent variables (state variables), u is a vector representing the independent (the control) variables, $g_j$ and $h_j$ represent equality and inequality constraints, respectively. m and p are the number of equality and inequality constraints, respectively.

The dependent variables (x) in the power system can be described as follows:

$$x=[P_{G,1},V_{L,1},\dots V_{L,NPQ},Q_{G,1},\dots Q_{G,NG},S_{TL,1},\dots S_{TL,NTL}]$$

(4)

where $P{G}_1$ is the slack bus power, $V_L$ is the voltage of load bus, $Q_G$ is the generator reactive power output, $S_{T}L$ is the apparent power flow in the transmission line, $NPQ$ is the number of load buses, $NG$ is the number of generation buses, and $NTL$ is the number of transmission lines.

The independent variable u of the power system can be described as follows:

$$u=[P_{G,2},\dots P_{G,NG},V_{G,1},\dots V_{G,NG},Q_{C,1},\dots Q_{C,NG},T_1,\dots T_{NT}]$$

(5)

where $P_G$ is the output active power of the generator, $V_G$ is the voltage of the generation bus, $Q_C$ is the injected reactive power of the shunt compensator, T is the tap setting of the transformer, $NC$ is the number of shunt compensator units, and $NT$ is the number of transformers.

2.1. Objective Functions

2.1.1. Quadratic Fuel Cost

The objective function is the quadratic equation of the total generation fuel cost which is formulated as follows:

$$F=\sum _{i=1}^{N}G{F}_i\left(P_{Gi}\right)=\sum _{i=1}^{NPV}(a_i+b_i{P}_{Gi}+c_i{P}_{Gi}^2)$$

(6)

where $F_i$ is the fuel cost of the ith generator. $a_i$, $b_i$, and $c_i$ are the cost coefficients of the ith generator.

2.1.2. Real Power Loss Minimization

The required objective function is minimizing the active power loss, which can be formulated as follows:

$$P_{loss}=\sum _{i=1}^{N}TL{G}_{ij}(V_i^2+V_j^2-2V_i{V}_{j}cos{\delta }_{ij})$$

(7)

where $G_{ij}$ the conductance of a transmission, $NTL$ is the number of transmission lines, and ${\delta }_{ij}$ is the phase difference of voltages.

2.2. Constraints

The transmission system has several constraints which can be categorized as follows:

2.2.1. Equality Constraints

The equality constraints represent the balanced load flow equations as follows:

$$\begin{array}{c}{P}_{Gi}-P_{Di}=|V_i|\sum _{j=1}^{NB}\left|V_j\right|(G_{ij}cos{\delta }_{ij}+B_{ij}sin{\delta }_{ij})\end{array}$$

(8)

$$\begin{array}{c}{Q}_{Gi}-Q_{Di}=|V_i|\sum _{j=1}^{NB}\left|V_j\right|(G_{ij}cos{\delta }_{ij}+B_{ij}sin{\delta }_{ij})\end{array}$$

(9)

where $P{G}_i$ and $Q{G}_i$ are the generated active and reactive power at bus i, respectively. $P{D}_i$ and $Q{D}_i$ are the active and reactive load demand at bus i, respectively. $G_{ij}$ and $B_{ij}$ are the conductance and susceptance between bus i and bus j, respectively.

2.2.2. Inequality Constraints

The inequality constraints can be classified as follows:

• Generator active power output

  $$P_{Gi}^{min}\le P_{Gi}\le P_{Gi}^{max}i=1,2,\dots ,NG$$

  (10)
• Generator bus voltage

  $$V_{Gi}^{min}\le V_{Gi}\le V_{Gi}^{max}i=1,2,\dots ,NG$$

  (11)
• Generator reactive power output

  $$Q_{Gi}^{min}\le Q_{Gi}\le Q_{Gi}^{max}i=1,2,\dots ,NG$$

  (12)
• Transformer tap settings

  $$T_i^{min}\le T_i\le T_i^{max}i=1,2,\dots ,NT$$

  (13)
• Shunt VAR compensator

  $$Q_{Ci}^{min}\le Q_{Ci}\le Q_{Ci}^{max}i=1,2,\dots ,NC$$

  (14)
• Apparent power flow in transmission lines

  $$S_{Li}\le S_{Li}^{max}i=1,2,\dots ,NTL$$

  (15)
• Voltage magnitude of load buses

  $$V_{Li}^{min}\le V_{Li}\le V_{Li}^{max}i=1,2,\dots ,NPQ$$

  (16)

3. Transformation of AC OPF Formulation for Machine Learning

Based on the form of OPF problems in Section 2, ML technology usually requires a certain form of transformation for OPF problems, with the following five transform forms:

Transform form 1: Direct utilization of the equality constraints according to (8). In this approach, Baker [40,41,42] showed that given $P_{Di}$, $Q_{Di}$, $G_{ij}$, and $B_{ij}$, $P_{Gi}$ and $V_i$ can be obtained according to (8). Therefore, the general process involves using traditional optimization algorithms to obtain a large number of pairs $[(P_{Di},Q_{Di}),(P_{Gi},V_i)]$. Subsequently, an ANN is constructed to fit those pairs, as illustrated in Figure 1.

Transform form 2: Formulating OPF problems into a Markov decision process (MDP). The OPF problem can be described as an MDP with four components $(S,A,P,R)$. Here, S represents the state space, where $s_t\in S$ denotes the state at time t. The state includes variables such as $P_{Di}$, $Q_{Di}$, $G_{ij}$, and $B_{ij}$. A represents the action space, where $a_t\in A$ represents the action at time t. The actions include variables like $P_{Gi}$ and $V_i$. $P\left(s_{t+1}\right|s_t,a_t)$ represents the state transition function, which indicates the probability of transitioning from state $s_t$ to $s_{t+1}$ when action $a_t$ is taken. It corresponds to the simulation model of the power grid. R represents the reward function, where $r_t=R(s_t,a_t,s_{t+1})$ represents the immediate reward obtained when action $a_t$ is taken in state $s_t$. The reward consists of objective functions, inequality constraints, and other factors. Therefore, the general process involves modeling the power grid and developing the power grid environment. Then, an ANN-based agent is designed to interact with the power grid environment, capturing the state, the action, and the rewards triad $(s_t,(a_t,r_t))$. Finally, the agent is trained based on the rewards $r_t$ to update the agent parameters, as illustrated in Figure 2.

Transform form 3: Predicting the active constraints in the OPF problem. Recognizing the active and inactive sets before solving the optimization problem can greatly reduce the problem’s complexity. This approach involves two stages: learning and predicting. In the learning stage, sampling is performed to obtain all possible activity sets, and the optimal activity set is identified from these sets. Subsequently, a classifier is trained to determine the optimality of an activity set. In the predicting stage, the classifier is used to evaluate the optimality of an activity set, as depicted in Figure 3.

Transform form 4: Predicting the warm-start point of OPF. Since improving the warm-start point could have convergence and computational speed benefits, the benefits of a warm start are solver dependent [43]. This approach involves generating a large number of datasets using traditional optimization algorithms, and forming state-optimal solution pairs. ML algorithms are then employed to learn the mapping relationships, and the learned model is used to predict the optimal solution in practical scenarios. However, at this stage, the optimal solution serves as the initial solution for the OPF optimization problem.

Transformation form 5: Learning the solving process. Baker [41] highlighted that several Quasi-Newton methods have been developed to find solutions to optimality conditions by utilizing an approximate Jacobian matrix. Firstly, the solver records the optimal solution data pairs $[x_k,x_{k+1}]$ at each step. Since $x_{k+1}=x_k-\alpha J^{-1}\left(x_k\right)d\left(x_k\right)$, Baker [41] proposed applying an ANN to fit $F_R\left(x_k\right)$, which represents $x_k-\alpha J^{-1}\left(x_k\right)d\left(x_k\right)$, where $d(·)$ and $J^{-1}(·)$ are a vector and the inverse Jacobian matrix of the KarushKuhn–Tucker (KKT) conditions, respectively. These data pairs are then utilized as input and output for training an ANN. Convergence is considered achieved when the difference $x_k-x_{k+1}$ is smaller than a predefined threshold ($ϵ$). The prediction process involves using the ANN to approximate fast iterations and obtain the best results, followed by solving the power flow to restore feasibility, as illustrated in Figure 4. Some research further incorporates gradients, including the Jacobian matrix, to guide ANN training [44,45,46].

4. Machine Learning Applications in OPF

Based on the aforementioned understanding, existing applications in OPF can be categorized into the direct mapping of OPF variables, prediction of active constraints, learning control policy for OPF, predicting warm-start points, and learning solving processes, which are summarized in

Table 1. Machine learning methods applied in optimal power flow (OPF).

Method	Strength	References
Direct Mapping of OPF Variable	Decrease computational time	[47]
	Incorporate physics-based rules	[30,48,49,50]
	Simplify the learning process	[51]
	Consider new energy unit	[29,52]
	Guarantee the solution feasibility	[34,53,54]
	Combine the DC OPF problem and convex ANN	[30,55,56]
Predicting Active Constraints	Explore simplified OPF version	[57,58]
	Classify active and inactive constraints	[40,58,59,60]
	Consider the uncertainty	[57,61]
	Explore the relationships among sets of con-currently active constraints	[62,63]
Learning Control Policy for OPF	Apply RL into PS operation	[64,65,66,67,68,69]
	Consider multi-object	[70]
	Apply RL into microgrid operation	[71,72,73]
	Develop model-based RL	[45,74,75]
	Guarantee the solution feasibility	[76,77]
Predicting Warm-Start Points	Explore simplified OPF version	[78,79,80,81,82,83]
	Enhance interpretability	[78]
Learning solution process	Approximate Jacobian matrix	[41,42,84,85,86,87,88,89]

.

4.1. Direct Mapping of OPF Variables

Zamzam and Baker [47] tried to learn a mapping between the system loading and optimal generation values, enabling the ANN to find near-optimal and feasible AC OPF solutions. It can bypass solving the traditionally nonconvex AC OPF problem, resulting in a significant decrease in computational burden for grid operators. However, the above application of ML techniques in OPF has faced several challenges, including the high demand for quality and quantity of training data, the generation of physically infeasible solutions, and the limited generalizability and interpretability of ML models. Zhou et al. [48] embedded the discrete topology representation into the continuous admittance space and trained a deep neural network (DNN) to learn the mapping from load and admittance to the corresponding OPF solution. They then employed the trained DNN to solve AC OPF problems over any power network with the same bus, generation, and line capacity configurations but different topology and/or line admittances. This method has two advantages. It beds the topology of the line as prior information into the neural network, which can greatly improve the interpretability of the neural network, reduce the dependence on a large amount of training data to a certain extent, and have good generalization performance for the case of topology changes.

Nellikkath et al. [30] also used a similar idea by incorporating physics-based rules or laws into ANN to overcome the above obstacles. This method is named the physics-informed neural network (PINN), and Huang and Wang [49] provided a detailed overview of the application of PINN in power systems. Specifically, Nellikkath et al. introduced the physical equations in the form of the AC OPF KKT conditions inside the neural network training. By doing that, the neural network can reduce its dependency on the size and quality of the training dataset, and it can determine its optimal parameters based on the actual equations that it aims to emulate. Falconer et al. [50] embedded physical information into a neural network from another perspective. Falconer pointed out that the majority of work in this area typically employs fully connected neural networks (FCNN), but graph neural networks (GNN) perform with superiority in embedding topological information within the power grid into the neural network architecture. Thus, Falconer et al. employed GNN to model the transmission line, bus, node, and so on, and the result outperformed both FCNN and convolution neural network models. Additionally, they further demonstrated the marginal utility of applying GNN architectures compared to FCNN for a fixed grid topology and found that GNN models are able to straightforwardly take the change of topological information into account.

In order to simplify the learning process, Lei et al. [51] introduced the Lagrange multipliers as intermediate variables to assist the extreme learning machine (SELM) for the OPF problem. The Lagrange multipliers cover the allocation of energy in the optimal power operating points, which is highly related to the AC OPF solutions. Hence, it can perform with significant efficiency for learning the AC OPF solution.

With the significant increase in new energy units, the problem makes decisions on the output of new energy units. Differently from traditional thermal power units, new energy units do not have the constraint of climbing rate, only the maximum power output at the current time. Christian et al. [52] incorporate new energy units into the OPF optimization problem by learning a mapping among the nodal loads and photovoltaic active powers and optimal photovoltaic reactive powers obtained. It first solved a standard AC OPF and then replaced AC OPF with a less computationally expensive ANN to perform centralized control of reactive power in photovoltaic systems. Further, they utilized shapley additive explanations, an explainable technique, to provide further insights into the behavior of the centralized ANN controller.

ANN has many parameters that need to be adjusted. This means that since parameter size, optimization problem complexity, and data set size need to be adapted to the problem, the parameters often need to be adjusted repeatedly. As compared to deep learning algorithms, SELM offers a rapid training speed and eliminates the need for time-consuming parameter-tuning processes. Lei et al. [29] introduced a data-driven approach for OPF utilizing the stacked SELM framework. Nonetheless, Lei et al. also highlighted that employing SELM directly for OPF poses challenges due to the intricate correlation between the system’s operational state and OPF solutions. They followed a similar idea in [51] and further decomposed the OPF model features into three stages by introducing the internal variables, which greatly simplified the training process.

Although the above methods have great advantages in terms of efficiency, the problem of unfeasibility often occurs. Lotfi et al. [34] firstly classify the feasible and infeasible AC OPF problems according to the load configuration where the feasible and infeasible labels come from the traditional optimization method. Then, the classification results will support the system operators to acknowledge the feasibility of the current load configuration before feeding it to ANN to predict system variables such as generator voltages and active power output. Thirdly, another DNN is used to predict the desired output of the OPF problem for each feasible pair of active and reactive power demand points. Instead of liking the work in [34], Minas et al. [53] adopted the idea of division and governance, decomposed the electric network, first learning the current and voltage from the coupling area, and then predicting each decomposed sub-network. They pointed out that the predictions can only result in minor violations only for the operational bound constraints. Xiang et al. [54] emphasized the importance of inequality constraints, arguing that inequality constraints should be satisfied in the solution of OPF based on ML. Therefore, they employed a penalty approach with a zero-order gradient estimation technique in the training process toward guaranteeing the inequality constraints.

Direct current (DC) OPF, due to its extensive simplification of the OPF problem, is essentially a convex problem, and its generalization performance cannot be guaranteed when the neural network is used for training and solving DC OPF. Ling et al. [55] utilized the convexity of the DC OPF problem and trained an input convex neural network. Further, in [30], they constructed the training loss based on Karush–Kuhn–Tucker optimality conditions. By combining these two techniques, the trained model has provable generalization properties. The aforementioned methods primarily concentrate on the static system topology, implying that modifications to the system’s structure (initiated by the system operator) necessitate retraining the DNN. This process incurs substantial training overhead and demands a substantial amount of training data specific to the new system topology. Chen et al. [56] employed meta-learning for DNN-based OPF predictor training, which can find a common initialization vector that enables fast training for any system topology.

4.2. Predicting Active Constraints

Solving the OPF problem directly using ML methods poses challenges, leading researchers to explore simplified versions of the original OPF to alleviate solver complexity. One area of current focus is predicting active constraints, which involves identifying and eliminating non-active constraints beforehand, thereby simplifying the optimization problem. Misra et al. [57,58] emphasize that utilizing active sets as features preserves the underlying system physics, enables interpretable models, considers important safety constraints, and facilitates straightforward representation and encoding. Additionally, it is possible to obtain the optimal solution by solving the reduced OPF problem for each important active set and checking the resulting solutions for feasibility.

Instead of like the work in [34] classifying the feasible and infeasible AC OPF problem, Baker and Bernstein [40] tried to eliminate zero-probability events such as inactive constraints to transform the joint chance constraints into deterministic constraints. Specifically, they use the forecasted load and solar at each node as the features within the SVC used to classify overvoltage constraints at those nodes. Yeesian et al. [59] provided an affine control policy. By utilizing the same set of active constraints, they proposed an ensemble control policy that combines several basis policies to improve performance. Despite the exponential growth in the number of potential bases relative to the system’s size, their research revealed that only a small subset of these bases is pertinent to system operation. However, this method can only be appied in DC OPF due to its convex characteristic. Misra et al. [58] tried to classify the active and inactive constraints, and then recognize the important active sets. Then, the ML-based classifier is constructed and trained. It should be noted that the proposed active margin functions as continuous indices instead of binary active or inactive to better quantify how likely each security constraint will be active. Further, Hasan et al. [60] used a mixture of classification and regression learners to predict active and inactive inequality constraints. Specifically, they first trained a regression learner to complete the direct mapping of OPF variables’ task. Subsequently, the variable was used for constrained classification.

With extensive access to new energy sources, the impact of input uncertainty on the active set becomes significant. Deepjyoti and Sidhant [57] employed the ML-based classifier to learn the mapping between the uncertainty inputs and the active set of constraints at optimality, thus further enhancing the computational efficiency of the real-time prediction. Similarly, Liu et al. [61] used historical data and simulation data about power system conditions (e.g., load fluctuation, temperature, uncertainties of solar and wind generation) and security-constrained optimal power flow (SCOPF) calculation results to prepare input features and output labels.

Zhang et al. [62] discovered that the sets of inactive line flow constraints identified by previously proposed constraint screening methods were actually observed to be active across various scenarios. As a result, they investigated the line characteristics commonly associated with active flow constraints and explored the relationships among sets of concurrently active line flow constraints. Liu et al. [63] proposed active likelihood functions to measure the conditions of transmission line capacity constraints. Then, a DNN model was developed to predict active likelihood functions and then predict an active constraint set.

4.3. Learning Control Policy for OPF

Reinforcement learning focuses on how to make decisions based on the context to maximize the expected benefits [90]. In this framework, a basic reinforcement learning agent interacts with its environment in discrete time steps. At each time step t, the reinforcement learning agent receives the current state $S_t$ and reward $R_t$. It then selects an action $A_t$ from the available set of actions and sends it to the environment. The environment will run this action and transit it to a new state $S_{t+1}$, and the associated reward $R_{t+1}$ for the transition $(S_t,A_t,S_{t+1})$ is determined. The primary goal of the agent is to maximize the anticipated cumulative reward. In the practical context of power system operation, the OPF problem is addressed by utilizing real-time system state detection to determine the most favorable operational actions. These actions are then executed through the corresponding actuators, while the system state is monitored at the subsequent time step. This process bears similarity to the general framework of reinforcement learning.

Woo et al. [64] employed the twin delayed deep deterministic policy gradient approach to improve the computational performance of AC OPF. Specifically, they designed an appropriate reward function in the training process considering cost functions, equality constraints, and inequality constraints. It means that if the agent outputs an unreasonable action, the reward will be very low or even negative. Then, this low reward will instruct the agent not to take this action. By contrast, if the reward is high, the agent will continue to output a similar action the next time. Furthermore, since the action space is continuous and generally extremely high, they also employed random exploration into the training process. Additionally, random Gaussian noise was also added to the individual net loads to represent uncertainty characteristics introduced by renewable energy sources. Similar work can be seen in [65,66,67]. Wang et al. [68] further proposed a novel human–machine collaborative (HMC) framework for line flow control. Specifically, HMC will analyze the actions produced by a reinforcement learning agent and human, respectively, to determine which action the power system should take. Zeng et al. [69] employed RL to devise an adaptive policy for selecting penalty parameters in the AC OPF problem, which is solved using the alternating direction method of multipliers. The objective of their approach is to minimize the number of iterations required for convergence.

In the face of global warming and the ongoing depletion of fossil energy resources, the power system is compelled to adopt an operational and developmental approach that minimizes carbon emissions. Various methods exist to reduce carbon emissions from both the production and consumption perspectives, including the utilization of renewable energy alternatives and the aggregation of distributed resources. Qin et al. [70] presented a novel approach for solving the multi-objective optimal carbon emission flow problem using reinforcement learning. Their method considers both the economic indicators traditionally considered in OPF and the reduction of unnecessary carbon emissions during the electricity transmission process.

In abnormal grid conditions, the microgrid operates in the islanded mode for providing uninterrupted supply to loads and stability improvement with power resilience. This islanded operation depends on the effective operation of connected distributed renewable energy sources (DRESs). In [71,72,73], deep learning reinforcement agent was employed to provide optimum power dispatch, and accurate control of connected DRES enabled the grid to restore service.

The challenges of SCOPF mainly come from the contingency constraints. Conventional supervised learning and reinforcement learning do not directly consider the constraints in the learning process [74]. Recent research shows that model-based RL can outperform the model-free methods in domains that require precision by incorporating a model into the training procedure. Therefore, utilizing primal-dual methods with deep deterministic policy gradient (DDPG) [74,75] to replace the agent cost provides potential alternatives to address the challenges of SCOPF problems. Inspired by this work, instead of building reward critic networks and cost critic networks via interacting with the environment (i.e., power flow equations), Yan and Xu [45] proposed solving KKT conditions of the Lagrangian as the actor gradients. Specifically, with the formulated sparse Jacobians of constraints and sparse Hessians of Lagrangians, the interior point method is incorporated in DDPG to derive the parameter updating rule of the DRL agent.

Despite deep reinforcement learning, the AC OPF problem itself may not be feasible due to the numerous security constraints on the power system. Zhou et al. [76] designed an ancillary classifier to identify the feasibility of the AC OPF problem before conducting a DRL agent. In order to accelerate the training process, they also adopted a supervised-learning method from deep learning to generate good initial weights for neural networks, and then the proximal policy optimization algorithm is applied to train and test the artificial intelligence agents for stable and robust performance. Sayed et al. [77] proposed a convex-constrained soft actor–critic (CC-SAC) deep reinforcement learning algorithm for the AC OPF problem. CC-SAC is a combination of data-driven and physics-driven approaches. The data-driven part speeds up the solution time by predicting near-optimum control actions. The physics-driven part effectively guarantees the solution feasibility.

4.4. Predicting Warm-Start Points

The quality of the starting point greatly influences the result and convergence efficiency of the optimization algorithm, especially for the non-convex and constrained alternating current OPF problem [78,79,80,81,82]. Forecasting warm start points is similar to directly mapping OPF variables. Baker [83] used OPF data to train a random forest to predict solutions for future AC OPF problems. The inputs (features) to the random forest are simply the loads at each bus. The outputs of the model are the optimal generation values and the voltage magnitudes.

Cao et al. [78] pointed out that simply utilizing black-box models lacked interpretability. Therefore, they proposed a fast and explainable warm-start point learning method based on the multi-target binary decision tree with a postpruning module. A set of detailed decision rules for selecting warm-start points was generated after the learning process. The generated rules assist the power system operators in identifying important loads and thereby provide the model interpretability.

4.5. Learning Solution Process

While the ML-based OPF solution method mentioned above has received considerable attention in terms of solution feasibility and optimality, it still lacks theoretical and mathematical guarantees. As a result, numerous studies have focused on the optimization method’s solution process, aiming to utilize ML methods to reveal gradient information and ensure solution feasibility.

The Jacobian matrix is the core part of power flow analysis, which is the basis for power system planning and operations. Chen et al. [84] introduced a measurement-based approach for calculating the power flow Jacobian matrix. By leveraging this matrix, they were able to extract valuable insights regarding the system topology in near real-time. Zeng et al. [85] proposed a GPU-based sparse modified Newton’s method by the introduction of a fixed Jacobian matrix, which integrates vectorization and parallelization techniques to accelerate power flow calculations. He et al. [86] estimated the Jacobian matrix in high dimensional space by the least square method and ANN. They found that the ANN-based method is sensitive to up-to-date topology parameters and state variables.

Calculating the Jacobian and its inverse is expensive and time-consuming, which may not be appropriate for large problems or fast-timescale optimization. Baker [41,42] proposed a Quasi-Newton method, in which Baker designed an ANN with a proper choice of weights and activation functions to approximate Jacobian calculating and guarantee convergence. The radial basis function ANN was also used to solve nonlinear equation load flow analysis and avoid calculating partial derivatives and an inverse Jacobian matrix [87,88]. Veerasamy et al. [89] proposed a novel generalized linear Hopfield neural network-based power flow analysis technique using the Moore–Penrose Inverse (MPI) to solve the nonlinear power flow equations (PFEs). The Hopfield neural network (HNN) with linear activation function augmented by a feed-forward layer is used to compute the MPI. In this work, the inverse of the Jacobian matrix in solving the PFEs is determined by including a feed-forward network along with the feedback network.

5. Limitations and Path Forward

The successful applications of ML in AC OPF reveal the potential of data-driven solutions from various perspectives. This section aims to provide a comprehensive and general outlook for the future development of ML in addressing open problems in AC OPF, while also highlighting current limitations. The advancement of ML in AC OPF should align with the modern and future development of power systems. It is important to note that Section 4 presents extensive applications of ML in AC OPF, particularly with the emergence of SG, DER, and RES. However, the AC OPF problem has not yet been entirely solved through ML due to the lack of mathematical guarantees. Therefore, a common approach is to establish a hierarchical solution framework. The first layer employs a data-driven scheme to quickly obtain an initial solution, while the second layer utilizes a math-driven scheme to refine the initial solution, ensuring adherence to the constraints and a continuous improvement of solution quality.

Furthermore, a substantial portion of power flow data is expressed in complex numbers, including voltage, power, and impedance. Incorporating this valuable information effectively into ANNs and devising an unsupervised learning approach are effective means of improving training efficiency. Simultaneously, both power flow and OPF rely on power flow equations. Hence, leveraging general first principles based on power flow equations can serve as further guidance for PINNs. Additionally, apart from linearity, other convexities observed in OPF research can be integrated into PINNs, thereby expanding their capabilities [49].

Moreover, the grid topology is subject to variations caused by various external and internal factors. Therefore, it is crucial to develop robust reinforcement learning agents and leverage meta-learning techniques to enable ML algorithms to adapt quickly to new tasks based on past learning experiences. Meta-learning algorithms leverage common features and patterns across tasks to improve inference and generalization to new tasks. Transfer learning is also highly valued in the deep learning community as it significantly enhances sample efficiency and training efficiency, particularly for problems with a series of similar tasks.

Finally, the attention mechanism, which has shown success in natural language processing and has recently been applied in power systems [91,92], is considered one of the most popular ML architectures. Exploring the integration of first principles into the attention mechanism presents an exciting avenue for future research. With the rapid advancement of attention mechanism-based large models like GPT-4 [93], they can serve as valuable references for designing large models to address OPF problems. This direction holds immense potential and is quite exciting.

6. Conclusions

This article provides a comprehensive review of machine learning (ML) techniques applied to solve AC OPF problems. It summarizes and discusses five specific techniques based on the general mathematical form of AC OPF: the power flow equation, the active constraint of AC OPF, the Markov form of AC OPF, the AC OPF warm-start point prediction, and the optimization process of traditional optimization methods. This review aims to assist researchers in quickly understanding the application of advanced ML technology in AC OPF problems and provide a framework for improving ML techniques based on the specific characteristics of AC OPF. By deepening the understanding of AC OPF and the aforementioned techniques, researchers can conduct more efficient and reliable research on AC OPF solution technologies.

Furthermore, considering the future development of power systems and the state-of-the-art ML techniques, this review explores potential research directions for ML in AC OPF. These include establishing mathematical guarantees, improving training efficiency, integrating first principles, and utilizing large models. These advancements have the potential to significantly enhance the speed and reliability of AC OPF solutions, supporting the safe and economically stable operation of future power systems.