Reinforcement Learning for Integrated MPPT and Battery Management in Photovoltaic Systems: A Systematic Review

Abstract

This work provides a systematic literature review on reinforcement learning (RL) for integrated maximum power point tracking (MPPT) and battery management in photovoltaic (PV) systems. As PV installations increasingly incorporate battery energy storage, the traditional objective of maximizing instantaneous power extraction is no longer sufficient on its own, since control decisions also affect battery state of charge, efficiency, degradation, and load support. Although RL has shown strong potential for sequential decision making in energy systems, most existing studies still treat MPPT and battery management as separate or only loosely coordinated problems. This review examines this issue by systematically examining how RL, particularly continuous-action methods, has been applied to coupled PV–battery control. The analysis highlights the shortcomings of discrete-action formulations in power-electronic systems and emphasizes the advantages and limitations of actor–critic approaches such as DDPG, TD3, PPO, and SAC for directly optimizing continuous-control variables. Across the reviewed literature, RL is found to be used predominantly at the supervisory energy management level, with PV generation often treated as exogenous rather than as an explicit control decision. This review therefore identifies a persistent structural separation between converter-level PV control and storage-aware energy management within a common learning and evaluation framework. It identifies continuous-action RL as a candidate formulation for unified PV–battery optimization while highlighting important challenges in constraint handling, state representation, sample efficiency, stability, and hardware validation.

1. Introduction

The rapid expansion of PV systems, coupled with the increasing deployment of battery energy storage, has significantly transformed the control and optimization requirements of modern energy systems. While early research primarily focused on maximizing instantaneous power extraction through MPPT, contemporary PV–battery architectures demand coordinated control strategies that account for system-level dynamics, operational constraints, and long-term performance objectives.

Conventional MPPT techniques, such as Perturb and Observe (P&O) and Incremental Conductance (INC), remain widely used due to their simplicity and low computational cost. These methods iteratively adjust the operating point of the PV array to track the maximum power point under varying environmental conditions [1,2]. However, their reliance on local perturbation mechanisms leads to well-known limitations, including steady-state oscillations, sensitivity to measurement noise, and degraded performance under rapidly changing irradiance [3,4]. Under partial shading, where multiple local maxima arise, these methods often converge to sub-optimal operating points [5,6]. Further, when considered as an isolated control objective, conventional MPPT generally focuses on instantaneous PV power extraction and does not by itself represent the effects of present operating-point decisions on future battery and system states.

To address these limitations, data-driven approaches based on artificial neural networks (ANNs) and deep learning (DL) have been introduced. These methods approximate the nonlinear relationship between PV measurements and optimal operating points, enabling improved tracking accuracy and reduced oscillations compared with classical algorithms [1,7]. Despite these advantages, many supervised learning-based MPPT formulations depend on offline training data and may exhibit limited adaptation when operating conditions differ from those represented during training. Moreover, such approaches typically learn static input–output mappings and do not explicitly account for sequential decision making or long-term system objectives.

Reinforcement learning has gained attention as an alternative approach by formulating MPPT as a sequential decision-making problem. Early RL-based approaches employ discrete-action formulations, where the control problem is modeled as a finite Markov Decision Process (MDP) and solved using algorithms such as Q-learning or SARSA. These methods demonstrate improved adaptability and reduced oscillatory behavior compared with perturbation-based techniques [8,9,10]. Extensions based on deep Q-networks (DQNs) further enhance scalability by approximating the action-value function with neural networks, enabling improved performance under complex operating conditions [11,12].

Even so, discrete-action RL can introduce limitations when inherently real-valued control variables are represented using a finite action set. Coarse discretization may restrict control resolution and can produce quantization effects or oscillatory behavior, whereas finer discretization increases the size of the state–action space and may aggravate the curse of dimensionality [13]. These trade-offs are particularly relevant when the RL action represents a real-valued converter command, such as a duty-ratio, voltage, current, or power reference. However, discrete actions remain appropriate for operating-mode selection, switching decisions, and explicitly enumerated feasible actions.

It is also important to distinguish the RL action representation from the physical switching process. A duty ratio may be represented as a continuous policy output, while its implementation through PWM remains a discrete high-frequency switching process.

Continuous-action RL avoids explicit discretization by directly parameterizing real-valued control commands. Actor–critic methods, including deep deterministic policy gradient (DDPG), Twin Delayed DDPG (TD3), Proximal Policy Optimization (PPO), and Soft Actor–Critic (SAC), have reported favorable and, in some studies, superior performance relative to the selected baselines in MPPT applications, including faster convergence, reduced steady-state oscillations, and high tracking efficiency under the tested dynamic conditions [14,15,16,17]. Unlike static supervised mappings, these methods optimize a cumulative reward through interaction with the modeled environment and can therefore support adaptation to nonlinear and time-varying dynamics. These reported advantages are dependent on the action definition, reward design, training conditions, constraints, and comparison baseline and do not establish general superiority over discrete, mixed-action, or conventional control approaches.

Despite these advances, the majority of existing RL-based MPPT studies focus on standalone PV systems. In practical deployment, PV arrays are increasingly integrated with battery storage and variable loads, where control decisions influence not only instantaneous power extraction but also the evolution of battery state of charge (SoC), system efficiency, and component lifetime. In such systems, the traditional objective of maximizing instantaneous PV power may conflict with broader operational goals, such as preventing battery over-charge, minimizing degradation, or satisfying load demand. Consequently, the MPPT objective or its resulting operating reference may need to be coordinated with battery and system-level objectives, particularly when storage or converter constraints become active. Such coordination may be implemented through hierarchical, cascaded, supervisory, or joint-decision architectures.

This interaction motivates an examination of how PV operating-point control and battery-aware objectives are represented across the existing literature. Within the corpus reviewed in this article, many RL formulations optimize supervisory battery, grid, or load decisions while leaving MPPT to a separate controller. Relatively few reviewed studies include converter-level PV operating-point decisions and battery-aware long-horizon objectives within the same learning and evaluation framework.

To examine these questions, this article presents a structured review of RL methods related to MPPT and battery management. This review maps the reported architectures, action variables, control layers, constraint treatments, and validation approaches and summarizes the study-specific evidence reported for discrete-, continuous-, and mixed-action spaces. Particular attention is given to the assumptions, limitations, and validation requirements associated with each formulation.

The contribution of this work is twofold:

• A critical analysis of discrete- and continuous-action RL approaches, highlighting their limitations and suitability for power-electronic control;
• A structured review of integrated PV–battery control strategies, identifying open challenges and future research directions for continuous-action RL.

The remainder of this paper is structured as follows: Section 2 examines the relationship between MPPT and battery management in PV–battery systems and motivates the need for integrated control formulations. Section 3 describes the PRISMA-guided review methodology adopted in this study. Section 4 presents the results of the systematic review and synthesizes the literature in relation to the research questions. Section 5 discusses the major research gaps and future opportunities for continuous-action RL in integrated PV–battery control. Finally, Section 6 concludes the paper.

2. Integrated MPPT and Battery Management

The integration of battery energy storage fundamentally alters the objective of MPPT in photovoltaic systems. While conventional MPPT maximizes instantaneous PV power, PV–battery systems introduce additional constraints related to SoC, charging limits, efficiency, and lifetime. Consequently, MPPT may no longer be treated as an isolated control task but ought to be embedded within a broader system-level optimization framework.

2.1. Coupling Between MPPT and Battery Dynamics

PV–battery systems may be implemented using DC-coupled or AC-coupled architectures, but the coupling between MPPT and storage dynamics is particularly strong in DC-coupled configurations, where the PV array and battery share a common DC bus [18]. In such systems, the selection of the PV operating point directly affects battery charging behavior, so enforcing strict MPPT without accounting for storage constraints may lead to excessive charging currents, unnecessary curtailment, inefficient energy utilization, or accelerated battery degradation [19,20].

Conventional control strategies typically adopt a hierarchical structure in which MPPT maximizes PV power at a lower level, while a supervisory controller independently manages battery charging and load supply. Although this decoupled architecture simplifies implementation, it implicitly assumes that maximizing instantaneous PV power is consistent with broader system-level objectives. In practice, this assumption may fail, particularly when the battery approaches its operational limits, leading to sub-optimal energy routing, unnecessary cycling, or increased stress on storage components [21].

To mitigate these limitations, several coordinated architectures and supervisory strategies have been proposed. An example of such architecture is shown in Figure 1, illustrating a representative DC-coupled PV–battery system in which the PV array and battery energy storage system exchange power through a shared DC bus. Although the PV-side and battery-side converters are often controlled independently, their operation remains physically coupled through the common DC link. As a result, the operating point selected for the PV array directly influences battery charging power, state-of-charge evolution, and the amount of energy available for load support. Consequently, decisions traditionally associated with MPPT can have significant implications for battery utilization, efficiency, and degradation.

While the topology shown employs separate DC–DC converters for the PV array and battery, similar interactions exist in alternative architectures, including non-inverting four-switch buck–boost converters, bidirectional buck–boost interfaces, and integrated multiport converter topologies. Likewise, though the coupling of these subsystems is particularly evident in DC-coupled configurations, similar dependencies arise in many alternative architectures, including AC-coupled ones. Although these configurations differ in terms of power-flow flexibility and converter integration, they share the fundamental characteristic that PV generation and battery behavior cannot be considered entirely independent control problems.

From a control perspective, this architecture highlights a structural mismatch that persists throughout much of the literature. The physical system is inherently coupled, yet MPPT and battery management are frequently formulated as separate optimization tasks, even if sometimes coordinated. This observation motivates integrated control approaches in which the PV operating point is treated as an explicit decision variable alongside battery charging and discharging actions, enabling system-level objectives such as efficiency, battery lifetime, and load support to be considered jointly rather than sequentially.

Multiport converters, such as three-port DC–DC converters, enable unified coordination of PV, battery, and load interfaces, reducing redundant conversion stages and improving energy routing flexibility [22]. Likewise, SoC-aware supervisory strategies can dynamically adjust power references to prevent over-charge and over-discharge while reducing cycling stress [21,23]. However, even in these settings, many studies still optimize battery scheduling while assuming that MPPT operates independently. For example, Voigt et al. [24] improved storage scheduling through machine learning-based load forecasting and optimization, yet the PV operating point was not included among the storage-scheduling decision variables.

With that, it can be seen that the intrinsic coupling of PV operating-point selection, future SoC evolution, and load demand remains underexploited in many existing control frameworks. These limitations motivate integrated control strategies that explicitly account for the interaction among PV generation, storage dynamics, and system-level objectives.

Figure 1. Representative DC-coupled PV–battery microgrid architecture (adapted from [25]). The PV array is interfaced through a DC-DC converter implementing MPPT control (using a given algorithm like P&O), while the battery is connected via a bidirectional DC-DC converter to a shared DC bus. The shared DC link illustrates the coupling between PV operating-point decisions and battery charging dynamics.

Figure 1. Representative DC-coupled PV–battery microgrid architecture (adapted from [25]). The PV array is interfaced through a DC-DC converter implementing MPPT control (using a given algorithm like P&O), while the battery is connected via a bidirectional DC-DC converter to a shared DC bus. The shared DC link illustrates the coupling between PV operating-point decisions and battery charging dynamics.

2.2. Limitations of Decoupled MPPT and Energy Management

Established PV–battery control systems frequently distribute MPPT, battery regulation, DC-bus control, and energy management functions across hierarchical or cascaded layers. Such structures simplify implementation, support time-scale separation, and can provide effective coordination when supervisory logic modifies operating references or control modes in response to SoC, load, or power-balance conditions.

In typical cascaded implementations, an MPPT or supervisory controller generates the operating reference, while faster inner voltage and current loops track that reference and regulate the converter and DC bus. A limitation arises when the MPPT and battery-management objectives are specified independently, such that maximizing instantaneous PV power is assumed to remain consistent with broader system-level objectives. This assumption may fail near battery-SoC limits or under variable load conditions, potentially leading to unnecessary curtailment, inefficient cycling, or increased battery stress [21]. Because present PV operating-point and storage decisions influence subsequent SoC evolution and constraint activation, PV–battery systems exhibit temporal interdependence. This motivates explicit coordination between MPPT and storage management, whether through hierarchical, cascaded, supervisory, or joint-decision control formulations.

Although RL is especially well suited to sequential PV–battery optimization, earlier ANN-based extensions also attempted to incorporate storage-aware behavior into MPPT-related control. Such approaches reported improvements in local dynamic performance, including smoother charging behavior and improved DC-link regulation in standalone or grid-integrated settings [25,26,27]. However, these methods remained fundamentally supervised and largely static, typically coordinating battery regulation and PV control without explicitly formulating the long-horizon coupling of PV operating-point selection, future SoC evolution, and system-level objectives. As a result, they only partially address the limitations of decoupled MPPT and energy management.

A representative non-RL example illustrating converter-level joint regulation is the DG–PV–BES islanded system proposed by Jha et al. [28]. In that architecture, the bidirectional buck–boost converter performs the dual function of PV MPPT and battery charging/discharging current regulation, thereby avoiding a fully decoupled arrangement in which MPPT and storage control are handled by separate converters or independent supervisory loops. The proposed scheme is particularly relevant because it demonstrates that PV operating-point regulation and battery protection can be coordinated within a shared electrical control structure while also being validated on an experimental prototype in variable-irradiance and -fault scenarios. Although the method is not learning-based, it provides an important benchmark for this review by showing that unified PV–battery coordination is technically achievable at the converter layer and need not be restricted to supervisory scheduling formulations.

Figure 2 conceptually summarizes the changes from decoupled MPPT and battery-management architectures toward increasingly integrated control formulations. Most conventional PV–battery systems remain close to the left side of the spectrum, where MPPT and energy management are treated as distinct tasks. Coordinated approaches introduce supervisory layers that exchange information between subsystems but still preserve separate optimization objectives. In contrast, fully integrated formulations treat PV operating-point selection, battery charging/discharging decisions, and grid interaction as components of a single control problem. This distinction is particularly important because the degree of integration determines whether long-term interactions between energy harvesting, storage utilization, and future system states can be explicitly considered during decision making. Even so, these categories describe how objectives and decision variables are allocated; they do not represent a ranking of technical maturity or performance. Coordinated multi-layer architectures preserve dedicated inner control loops while exchanging references, measurements, and constraints across supervisory and converter levels. Joint-decision architectures instead determine multiple PV- and battery-side variables within a shared optimization or control framework. The appropriate structure depends on the application, time scales, safety requirements, model availability, and implementation constraints.

Figure 2. Different distributions of control authority and decision variables of PV–battery control architectures, from decoupled MPPT and battery management to integrated optimization. The leftmost panel represents conventional architectures in which MPPT and battery/EMS functions are treated independently. The middle panel illustrates coordinated or layered approaches where a supervisory controller influences both PV and battery operation while preserving separate control loops. The rightmost panel represents integrated formulations in which a unified controller, such as a reinforcement learning agent, jointly determines PV operating point, battery charging/discharging power, and grid interaction based on system-wide objectives and future-state considerations. The horizontal arrow indicates an increasing degree of shared decision making and does not necessarily imply a progression in technical maturity, stability, or performance.

2.3. Reinforcement Learning for Joint Optimization

Reinforcement learning provides one possible framework for representing PV–battery operation as a sequential decision-making problem. By modeling the system as an MDP, RL enables direct optimization of long-term objectives that incorporate PV power extraction, battery-SoC evolution, and operational constraints.

Several studies demonstrate the effectiveness of RL in energy management in PV–battery systems. Härtel and Bocklisch [29] applied a PPO-based controller for energy cost minimization, achieving performance comparable to model predictive control while outperforming rule-based strategies. Similarly, Xu et al. [30] employed model-based RL to optimize energy scheduling in residential PV–battery systems, improving self-consumption and operational efficiency.

Nevertheless, many works treat PV generation as an exogenous input, assuming that MPPT operates independently of storage management. This separation limits the ability of RL to fully exploit the coupling between PV operating-point selection and battery dynamics.

2.4. Continuous-Action Reinforcement Learning for MPPT

Continuous-action RL formulates MPPT as a sequential control problem in which the agent directly optimizes the converter duty cycle to maximize cumulative performance over time. The distinction between discrete- and continuous-action RL concerns the decision variable produced by the policy, rather than the binary switching state of the semiconductor devices. For example, a continuous-action policy may generate a real-valued duty-ratio command, whereas a discrete-action policy selects that command from a predefined finite set. In both cases, the commanded duty ratio is physically implemented through discrete high-frequency PWM switching. Accordingly, the discrete nature of PWM does not by itself determine the appropriate RL action-space formulation. Discrete actions may be suitable for switching modes or predefined feasible commands, whereas continuous actions may be useful when fine adjustment of a duty-ratio or electrical reference is required. In fact, recent advancements increasingly favor parameterized or hybrid action spaces that combine the deterministic stability of discrete switching with the precision of continuous tuning. Therefore, rather than declaring one approach universally superior, a system-specific evaluation of stability and execution constraints is essential.

Beyond that, and unlike supervised ANN- or deep learning-based approaches that learn a static mapping from measurements to operating points, RL learns a policy through interaction with the PV–converter environment, typically modeled as an MDP. State representations commonly include PV voltage, current, power, and duty-cycle history, while actions are expressed as continuous duty-cycle values or increments. Reward functions generally favor high power extraction while penalizing oscillations, excessive control effort, or other undesirable dynamics [31].

Among continuous-action methods, DDPG has been widely adopted because its actor–critic structure enables direct continuous control. Artetxe et al. [14] reported faster convergence and reduced steady-state oscillations than P&O in both simulation and real-time experiments. However, vanilla DDPG is often sensitive to hyperparameters and may exhibit unstable learning dynamics [32]. To address these limitations, more robust actor–critic variants have been explored, including TD3, which mitigates value overestimation through twin critics and delayed updates; PPO, which improves policy update stability through clipped objectives; and SAC, which enhances exploration robustness through entropy regularization. These structural modifications are intended to reduce particular training pathologies, and reported results indicate reduced settling time and steady-state efficiency exceeding 99% under dynamic irradiance conditions [17]. Rostamipour et al. [33] further proposed an SAC-based MPPT approach augmented with transfer learning, enabling faster adaptation across irradiance and shading scenarios while reducing ripple compared with DDPG and P&O. Building on these continuous-action formulations, Figure 3 presents a generalized RL-based MPPT control architecture in which measured PV variables are processed into a state representation, the agent outputs a continuous duty-cycle increment, and the resulting duty cycle is applied to the converter through the PWM stage.

Figure 3. Continuous-action reinforcement learning architecture for the MPPT control of a PV-fed DC–DC converter. The environment consists of the PV array, the DC–DC converter, and the load, while the MPPT controller receives the measured PV variables $V_{PV}$, $I_{PV}$, and $P_{PV}$. These measurements are processed into a state vector $s_t$, which is supplied to a continuous-action RL agent. The agent outputs a duty-cycle increment $a_t=\Delta d_t$, which is integrated and saturated to generate the converter duty cycle $d_t\in [0.05,0.95]$. The reward function $r_t$ promotes increased PV power extraction while penalizing excessive duty-cycle variation and oscillatory behavior. The RL agent may be implemented using actor–critic algorithms such as DDPG, TD3, PPO, and SAC. The continuous duty-ratio representation shown at the policy output should be distinguished from the discrete switching events generated by the PWM implementation.

2.4.1. Reward Design and Stability Considerations

Reward design critically shapes continuous-action RL behavior in MPPT. Typical formulations maximize extracted PV power while penalizing excessive current, abrupt duty-cycle variation, or in storage-integrated settings, constraint violations related to battery operation [34]. Because actions influence future system states, the reward implicitly encodes long-term objectives rather than instantaneous optimization alone.

However, actor–critic methods generally provide limited formal stability guarantees. Prior studies report sensitivity to hyperparameters and reward scaling, particularly in DDPG-based implementations [32], and stability assessment remains largely empirical [35]. Robust convergence through principled reward shaping, regularization, and constraint-aware policy design therefore remains an open challenge.

2.4.2. Simulation-to-Real Transfer and Hardware Validation

Most RL-based MPPT studies rely on MATLAB/Simulink environments with simplified PV and converter models. Although such environments are suitable for controlled evaluation, they often neglect sensor noise, switching losses, parameter drift, and other non-idealities, contributing to a persistent simulation-to-reality gap [36]. Hardware validation is typically limited to laboratory prototypes or hardware-in-the-loop platforms [37], and systematic sim-to-real methodologies remain underdeveloped.

For example, Karaki et al. [38] reported improved disturbance resilience using a deep-RL-based MPPT strategy in a fault-tolerant control framework, but validation remained primarily simulation-based. Bridging this sim-to-real gap remains a major barrier to practical deployment in PV systems.

2.4.3. Critical Assessment

Continuous-action RL might provide a suitable formulation when the selected decision variable is represented as a real-valued voltage, current, power, or duty-ratio command. In such cases, actor–critic methods avoid the need to define a finite action grid and can represent fine variations in the commanded reference. Several reviewed MPPT studies report favorable tracking performance, reduced steady-state oscillations, or improved adaptation under nonlinear and time-varying operating conditions. These reported benefits are relevant to applications in which control resolution is important, but they remain dependent on the selected baseline, reward design, training procedure, and evaluation conditions.

This consideration is also relevant in PV–battery systems, where decision variables such as PV voltage or duty ratio, battery charging/discharging power, and current references may be represented as continuous quantities. Compared with finite discretizations of these variables, continuous-action methods can avoid explicit command-level quantization and may scale more conveniently when several real-valued actions are optimized jointly [39]. For example, Jin et al. [40] reported lower operating cost and faster convergence for SAC than for the DQN in the microgrid energy management problem considered in their study. This result supports the potential value of continuous actions in that particular architecture and comparison setting, rather than establishing a general advantage over all discrete-action methods.

Discrete actions nevertheless remain appropriate for operating-mode selection, switching decisions, and explicitly enumerated feasible commands, while mixed-action spaces may be preferable when categorical decisions must be combined with real-valued references. Continuous policies also introduce important challenges, including training instability, sample inefficiency, sensitivity to reward and state design, constraint enforcement, limited formal guarantees, computational requirements, and hardware validation.

Accordingly, continuous-action RL may offer a useful representation for PV–battery control problems involving multiple real-valued decision variables, but it should not be regarded as a universally superior solution or as a replacement for established conventional controllers. Its engineering value must be established through controlled comparisons with discrete-action, mixed-action, and conventional methods under common architectures, constraints, operating conditions, and performance metrics.

2.5. Continuous-Action RL in Integrated-Control Studies

Continuous-action RL is capable of being particularly well suited for integrated MPPT and battery management, as key control variables, such as converter duty cycle and battery charging power, are inherently continuous. Actor–critic methods such as DDPG, PPO, and SAC enable direct optimization of these variables without requiring discretization.

Zhou et al. [41] demonstrated that DDPG-based continuous control improves coordination between PV generation, battery dispatch, and load demand, achieving better supply–demand balance compared with rule-based approaches. Similarly, SAC-based methods have been shown to enhance robustness and convergence in stochastic environments, while supporting constraint-aware optimization [42,43].

Compared with finite discrete-action formulations, continuous RL can offer finer control resolution, reduce command-level quantization effects, and support smoother coordination across coupled subsystems under the evaluated conditions.

Recent non-RL work also illustrates a stronger form of cross-layer coordination than is typically observed in the RL literature. Rizki et al. [44] proposed a standalone PV–battery DC microgrid framework that combines hybrid TCN–LSTM forecasting, predictive energy management, hierarchical load shedding, and converter-level double-integral sliding-mode control within a single coordinated architecture. In that formulation, forecasts of irradiance, temperature, and load are used to generate PV and battery reference trajectories proactively, while the boost and bidirectional converters track those references through robust low-level control. This type of design is important for the present review because it occupies an intermediate position between fully decoupled EMS formulations and truly unified single-controller PV–battery co-control: PV is not treated as merely exogenous, yet converter-level execution remains layered beneath a supervisory predictive EMS. As such, it provides a useful comparator for assessing how far existing integrated control architectures have progressed, even outside RL.

Additional non-RL studies further reinforce this point, although more selectively. Dagal et al. [45] addressed a PV–battery system under partial shading using a novel gray-wolf-optimization-based MPPT strategy that explicitly incorporates battery state-of-charge considerations into the control objective, reporting high tracking efficiency and reduced ripple in simulation. Likewise, Tavoosi et al. [46] considered a hybrid PV/battery switching system in which separate control inputs regulate the PV-side boost converter and the battery-side converter, using an evolutionary-algorithm-tuned sliding-mode controller to improve transient and steady-state behavior. These studies are relevant because they show that non-RL intelligent or nonlinear control methods can coordinate PV-side and battery-side actuation more tightly than most supervisory EMS formulations; however, they remain primarily converter-control papers rather than long-horizon sequential optimization frameworks.

2.6. Challenges and Research Opportunities

Despite its advantages, integrated RL-based control for PV–battery systems faces several challenges:

• Constraint handling: Battery-SoC limits, power bounds, and safety constraints must be explicitly enforced.
• Training complexity: Learning in high-dimensional, coupled environments increases sample requirements.
• Sim-to-real gap: Most studies rely on simulation, with limited hardware validation.
• State representation: Accurate estimation of battery states (e.g., SoC and SoH) is critical to reliable policy behavior.

These challenges indicate that while continuous-action RL provides a potentially promising framework for unified MPPT and energy management, further research is required to ensure robustness, scalability, and practical deployability. All the same, the reviewed evidence does not establish that an RL controller should necessarily replace mature cascaded, sliding-mode, MPC, droop-based, or other nonlinear control methods.

A practically defensible extension of continuous-action RL-based MPPT is to incorporate battery-aware objectives through coordinated supervisory decision making or reference generation. Under uncertain, multi-objective, and temporally coupled operating conditions, an RL policy may determine PV and battery references, while established voltage, current, DC-bus, and protection loops retain responsibility for fast and safety-critical regulation. In this way, continuous-action RL can be investigated as a cross-layer coordination mechanism rather than as a direct replacement for mature converter controllers. Hybrid implementations may further combine the learned policy with model-based constraints, safety filters, or fallback control laws.

This extension remains to be validated through controlled comparisons that determine whether and under which conditions the inclusion of battery-aware, continuous decision variables provides measurable benefits over separate RL formulations, discrete- or mixed-action methods, and conventional hierarchical control. Such comparisons should use common system architectures, operating scenarios, constraints, time scales, baselines, and performance metrics, followed where possible by hardware-in-the-loop or experimental validation.

Such a shift may reinforce an eventual premise of continuous-action RL as a central framework for real-time, system-level optimization in renewable energy systems.

3. Methodology

This section presents the systematic literature review conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines [47]. No review protocol was registered. The objective of this review is to ensure a transparent, reproducible, and comprehensive identification of relevant research related to: (i) photovoltaic (PV) and PV–battery energy systems; (ii) maximum power point tracking (MPPT), energy management, and power-electronic control strategies; and (iii) intelligent and learning-based control methods.

With an initial search strategy capturing a broad range of AI-based MPPT and optimization approaches to position the state of the art, the final synthesis prioritizes RL-based formulations, particularly those addressing sequential decision making, continuous-control actions, and battery-integrated PV systems.

3.1. Systematic Review Questions (PRQs)

To guide the systematic review and ensure alignment with the objectives of this work, the following PRISMA research questions (PRQs) were defined:

• PRQ1: In PV–battery systems with variable loads, what limitations are reported for decoupled control architectures where MPPT and battery/EMS control are designed independently?
• PRQ2: What control variables and actuation levels are used in learning-based approaches for PV–battery systems (e.g., PV duty cycle/operating point, battery current/power, and DC-bus voltage), and how often is MPPT treated as a controllable decision rather than an exogenous process?
• PRQ3: How are reinforcement learning formulations constructed for joint PV–battery optimization (state, action space, reward design, and constraint handling), and what evidence comparing discrete-action and continuous-action control in this context exists?
• PRQ4: What practical implementation considerations are addressed in the literature for RL-based PV–battery control (training environments, constraint enforcement mechanisms, observability/SoC estimation, robustness, and validation beyond simulation)?

The PRQs define the coding structure and analytical lens of the systematic review. The evidence synthesized in response to PRQ1–PRQ4 is used to map the reported control architectures, identify comparatively underrepresented combinations, and define requirements for future comparative and experimental evaluation. These findings also inform broader research questions and may motivate the future formulation and assessment of continuous-action RL frameworks for integrated PV–battery control.

3.2. Search Strategy

The literature search was conducted across three major scientific databases: IEEE Xplore, ScienceDirect, and SpringerLink. Google Scholar was used as an indexing database to ensure broad coverage and cross-validation of records retrieved from publisher-specific sources. Google Scholar was additionally used for backward and forward citation tracking to identify relevant studies not captured in the initial database queries.

Four groups of keywords were defined to represent the main conceptual domains of this review:

• PV, battery, and hybrid energy systems: “photovoltaic”, “PV system”, “solar energy”, “solar power”, “PV–battery”. “PV–BESS”, “energy storage”, “battery energy storage system”, “BESS”, “hybrid renewable system”, “battery management”, “state of charge”, “SoC”.
• MPPT, energy management, and power electronic control:
  “MPPT”, “maximum power point tracking”, “energy management”, “power management”, “energy management system”, “EMS”, “DC–DC converter”, “buck converter”, “boost converter”, “bidirectional converter”.
• Machine learning and deep learning methods:
  “machine learning”, “deep learning”, “neural network”, “artificial neural network”, “ANN”, “CNN”, “RNN”, “LSTM”, “autoencoder”, “transformer”, “data-driven model”, “fuzzy logic”, “genetic algorithm”, “GA”, “particle swarm optimization”, “PSO”.
• Reinforcement learning and policy-based control:
  “reinforcement learning”, “deep reinforcement learning”, “actor–critic”, “policy gradient”, “deep Q-network”, “DQN”, “DDPG”, “TD3”, “SAC”, “continuous action”.

Multiple search strings were formed using partial and complete combinations of the above keyword groups to avoid over-restrictive filtering. The following search string illustrates the primary query used in one of the databases (adapted as required for the other databases):

(("photovoltaic" OR "PV system" OR "solar energy") AND ("battery" OR "energy storage" OR "BESS") AND ("MPPT" OR "energy management" OR "EMS") AND ("machine learning" OR "deep learning" OR "neural network" OR "fuzzy logic" OR "genetic algorithm" OR "reinforcement learning" OR "deep reinforcement learning" OR "DQN" OR "DDPG" OR "TD3" OR "SAC"))

Table 1 summarizes the database-specific search strings and limits applied in this review. As noted above, the search strategy combined photovoltaic-domain terms with control, energy management, and artificial intelligence keywords, with minor adaptations to match the syntax and filtering options of IEEE Xplore, ScienceDirect, and SpringerLink.

Table 1. PRISMA search keywords by database.

Database	Search Set	Search String/Keywords Used
IEEE Xplore	Broad intelligent control	(“photovoltaic” OR “PV system” OR “solar energy”) AND (“MPPT” OR “maximum power point tracking” OR “energy management” OR “power management”) AND (“machine learning” OR “deep learning” OR ”neural network“ OR “reinforcement learning” OR “intelligent control”)
IEEE Xplore	PV + battery systems	(“photovoltaic” OR “PV”) AND (“battery” OR “energy storage” OR “BESS”) AND (“energy management” OR “EMS” OR “power control”) AND (“machine learning” OR “deep learning” OR “reinforcement learning” OR “GA” OR “genetic algorithm” OR “particle swarm optimization” OR “PSO” OR “LSTM” OR “RNN” OR “CNN” OR “ANN” OR “artificial neural network” OR “fuzzy logic”)
IEEE Xplore	Converter-level control	(“DC-DC converter” OR “buck converter” OR “boost converter” OR “bidirectional converter” OR “data-driven model”) AND (“photovoltaic” OR “PV”) AND (“machine learning” OR “deep learning” OR “reinforcement learning”)
ScienceDirect (Elsevier)	AI-based Energy Management for PV	TITLE(“photovoltaic” OR “PV”) AND (“energy management” OR “MPPT” OR “control strategy”) AND ABSTRACT(“machine learning” OR “deep learning” OR “reinforcement learning” OR “neural network”)
ScienceDirect (Elsevier)	PV–Battery Intelligent Energy Management	(“PV battery” OR “hybrid PV” OR “PV-BESS” OR “photovoltaic battery”) AND TITLE(“power management” OR “energy management” OR “battery management” OR “EMS” OR “SoC” OR “state of charge”) AND ABSTRACT(“machine learning” OR “deep learning” OR “reinforcement learning”)
ScienceDirect (Elsevier)	Converter-Level Intelligent Control	TITLE(“DC-DC converter” OR “power converter” OR “bidirectional converter”) AND TITLE(“photovoltaic” OR “PV”) AND ABSTRACT(“machine learning” OR “neural network” OR “reinforcement learning”)
ScienceDirect (Elsevier)	Intelligent/Optimization-Based Control for PV Systems	TITLE(“photovoltaic” OR “PV”) AND (“energy management” OR “optimization” OR “intelligent control”) AND ABSTRACT(“neural network” OR “deep learning” OR “reinforcement learning”)
ScienceDirect (Elsevier)	MPPT-Focused AI Control	TITLE(“MPPT” OR “maximum power point tracking”) AND (“photovoltaic” OR “PV”) AND ABSTRACT(“machine learning” OR “deep learning” OR “reinforcement learning” OR “neural network”)
SpringerLink	Learning-based PV control	(“photovoltaic” OR “solar”) AND (“control” OR “energy management” OR “MPPT”) AND (“machine learning” OR “deep learning” OR “reinforcement learning” OR “actor-critic” OR “policy gradient” OR “deep Q-network” OR “DQN” OR “DDPG” OR “TD3” OR “SAC” OR “continuous action”)
SpringerLink	Power electronics focus	(“DC-DC converter” OR “power electronic”) AND (“photovoltaic”) AND (“intelligent control” OR “learning-based”)
All databases	PV–battery separate/supervisory control	(“PV battery” OR “PV-battery” OR “PV-BESS”) AND (“MPPT” OR “maximum power point tracking”) AND (“BMS” OR “Battery management” OR “BESS”) AND (“hierarchical” OR “decentralized” OR “separate” OR “supervisory”)
All databases	PV–battery unified/integrated control	(“PV battery” OR “PV-battery” OR “PV-BESS”) AND (“MPPT” OR “maximum power point tracking”) AND (“BMS” OR “Battery management” OR “BESS”) AND (“unified control” OR “integrated control” OR “coordinated control” OR “joint control”)

The search was restricted to peer-reviewed journal articles and conference papers published in English between 2015 and 2026. This time window reflects the increased adoption of modern data-driven, deep learning, and RL techniques in power electronics and energy system control during the last decade.

The initial search strategy was deliberately broad and included AI/ML, deep learning, and RL terms in order to capture the full methodological landscape of intelligent control in PV and PV–battery systems and to reduce the risk of prematurely excluding relevant studies. During screening and full-text assessment, it became clear that RL was the most relevant class of methods for addressing the review questions related to sequential decision making, integrated control, and long-horizon optimization. Accordingly, the final analytical synthesis prioritized RL-based studies, while non-RL studies were retained primarily for background framing and contextual comparison; this comparator set includes established hierarchical, cascaded, predictive, nonlinear, and optimization-based control approaches that provide relevant engineering context. Standalone PV MPPT studies were captured during the broader search and screening process because they were useful for establishing the methodological differentiations from conventional and supervised approaches toward RL-based control. However, during full-text eligibility assessment and final coding for synthesis, studies focused exclusively on standalone PV MPPT without storage interaction or without relevance to the sequential PV–battery coordination problem were excluded from the core analytical synthesis. Some such studies were retained only as contextual background where they helped position RL within the broader MPPT literature.

3.3. Inclusion and Exclusion Criteria

Studies were included in the review if they satisfied the following criteria:

• Application of ML, DL, RL, DRL, or hybrid data-driven and optimization-based approaches to PV or PV–battery energy systems.
• Focus on MPPT, PV energy optimization, battery management, or power-electronic control.
• Presentation of simulation and/or experimental results with quantitative performance metrics.
• Consideration of continuous or discrete control actions applied to power converters (e.g., buck, boost, or bidirectional DC–DC converters).
• Relevance to PV–battery systems or hybrid renewable energy systems with storage interaction.

Studies were excluded if they met any of the following conditions:

• Purely theoretical discussions of AI methods without application to energy systems.
• Generic ANN, fuzzy logic, or optimization algorithms without relevance to PV, energy storage, or power-electronic control problems.
• PV modeling studies without any optimization, control, or decision-making component.
• Large-scale grid or market-level energy management studies unrelated to standalone or hybrid PV systems.
• Insufficient technical detail, absence of quantitative results, lack of relevance to the defined scope, or lack of peer-reviewed publication status.
• Studies that focus exclusively on standalone PV MPPT without consideration of storage interaction or sequential decision-making effects were excluded from the core analytical synthesis.
• Supervised learning approaches (e.g., ANN, CNN, and LSTM) used solely for static MPP estimation or forecasting, without closed-loop control or policy-based decision making, were excluded from the final reinforcement learning-focused analysis.

Borderline studies were classified according to their primary methodological and application relevance to the review questions. Where a study addressed PV, storage, or intelligent control but did not fully satisfy the core synthesis criteria, it was assessed based on whether it contributed directly to at least one of the following dimensions: (i) sequential decision making, (ii) reinforcement learning-based control, (iii) discrete- versus continuous-action formulation, or (iv) explicit PV–battery coupling. Studies not meeting these criteria were excluded from the core synthesis, although some were retained for contextual background.

3.4. Study Selection Process and PRISMA Flow Diagram

The study selection followed the standard PRISMA workflow consisting of four sequential phases:

• Identification: The initial database search returned 5120 records. After the removal of duplicate entries, 3667 unique studies remained for screening.
• Screening: Titles and abstracts of the 3667 records were screened against the predefined inclusion and exclusion criteria. This step resulted in 2147 studies selected for full-text assessment, after 370 not retrieved.
• Eligibility: Full-text evaluation was conducted to assess methodological relevance, technical depth, and applicability to PV systems, energy storage, and intelligent control. Following this phase, 101 studies were retained for further analysis, 44 of which were retained as contextual non-RL comparators.
• Included: The final set of 57 publications meeting all eligibility requirements was included in the systematic literature review and used for the thematic and comparative analysis presented in the subsequent literature review sections specifically addressing RL-based control of PV or PV–battery systems, identified for deeper analytical synthesis. This hierarchical classification enabled differentiation between general AI-based MPPT approaches and sequential decision-making frameworks.

Duplicate records were identified using Zotero and removed manually by the author after import from the searched databases. Title/abstract screening was conducted by a single reviewer (the author), with auxiliary organizational support tools. Full-text eligibility assessment was likewise conducted by the author alone. Because screening and full-text assessment were performed by a single reviewer, no inter-reviewer disagreements arose, and no formal arbitration procedure was required.

The screening and eligibility assessment were conducted by the author in accordance with the defined criteria. The complete study selection process is summarized in the PRISMA 2020 flow diagram shown in Figure 4, which illustrates the identification, screening, eligibility, and inclusion stages of the systematic review. To minimize selection bias, eligibility decisions were based strictly on the predefined inclusion and exclusion criteria.

Figure 4. PRISMA flow diagram for the systematic literature review.

4. Results of Systematic Review

4.1. Overview of Included Studies

The PRISMA-guided screening process identified a final corpus of studies for detailed review, with RL papers forming the core analytical set used to address PRQ1–PRQ4. In addition, a secondary set of non-RL studies was retained as a structured comparative reference set. Accordingly, the results are presented in two layers: (i) a descriptive comparison between RL and non-RL studies across architecture, MPPT treatment, actuation, method, and control layer; and (ii) a PRQ-driven synthesis focused on the RL corpus.

In total, $n=57$ RL studies were included in the core synthesis, while $n=44$ non-RL studies were retained for contextual comparison. This comparative coding does not broaden the analytical scope of the review beyond RL; rather, it is used to clarify where RL-based work is positioned within the wider PV–battery control literature and to make visible the structural differences between RL and non-RL approaches.

The comparative analysis shows that the literature is not divided simply by algorithmic family but also by control layer, actuation level, and treatment of MPPT. In particular, RL studies are concentrated at the EMS/supervisory layer and often treat PV generation as exogenous, whereas non-RL studies more frequently engage converter-level PV control or explicitly coordinated MPPT structures. These patterns are summarized in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 and are subsequently used to contextualize the RL-focused PRQ synthesis.

Figure 5 shows the distribution of studies by publication year for the RL corpus and the non-RL comparator set. The temporal pattern indicates that RL-based work is concentrated in more recent years, with a marked increase from 2023 onward, whereas the non-RL literature is distributed more broadly across earlier years. This pattern is consistent with the recent rise of RL in PV–battery energy management and supports the interpretation of RL as a comparatively newer methodological strand within the broader intelligent control literature.

Figure 5. Publication-year distribution of the reviewed literature, comparing RL studies with the non-RL comparator set. The RL corpus is concentrated in recent years, indicating the comparatively recent emergence of reinforcement learning within PV–battery control research.

Figure 5. Publication-year distribution of the reviewed literature, comparing RL studies with the non-RL comparator set. The RL corpus is concentrated in recent years, indicating the comparatively recent emergence of reinforcement learning within PV–battery control research.

Figure 6 shows a clear difference in publication format between the two bodies of literature. The non-RL comparator set is dominated by journal articles (34 studies), with a smaller number of conference papers (10), whereas the RL corpus contains a larger share of conference papers (32), alongside 24 journal articles and 1 master’s thesis. This pattern suggests that reinforcement learning-based work in PV–battery control is still evolving as a comparatively newer and faster-moving research strand, with many contributions appearing first in conference venues, while the non-RL literature appears more established and consolidated in archival journal publications. In the context of this review, the figure complements the broader temporal trend already observed in the manuscript by showing that the recent growth of RL research is accompanied not only by increased volume but also by a publication profile that is more conference-oriented than the non-RL comparator set.

Figure 6. Distribution of studies by document type, comparing the RL corpus and the non-RL comparator set. The RL literature is concentrated in conference papers and journal articles, whereas the non-RL literature is dominated by journal articles, with only a small number of thesis represented.

Figure 6. Distribution of studies by document type, comparing the RL corpus and the non-RL comparator set. The RL literature is concentrated in conference papers and journal articles, whereas the non-RL literature is dominated by journal articles, with only a small number of thesis represented.

Figure 7 presents the top publication venues for the RL corpus. The studies are distributed across a diverse set of journals and conference proceedings, with IEEE Access and the IEEE Internet of Things Journal contributing the largest numbers of papers, while most other venues contain only one or two studies. This pattern suggests that RL-based PV–battery research is dispersed across multiple adjacent engineering and energy domains rather than concentrated in a single dedicated outlet.

Figure 7. Distribution of RL studies by publication venue (top 20 venues shown). The RL literature is spread across a broad range of journals and conference proceedings, indicating a fragmented but multidisciplinary publication landscape.

Figure 7. Distribution of RL studies by publication venue (top 20 venues shown). The RL literature is spread across a broad range of journals and conference proceedings, indicating a fragmented but multidisciplinary publication landscape.

With that, Figure 8 illustrates a VOSviewer keyword co-occurrence map of the studies retained after the PRISMA screening process. The visualization reveals several strongly interconnected thematic clusters, showing that the literature is organized around a few central concepts. Terms such as algorithm, performance, model, energy management, system, and microgrid occupy the most central positions, indicating that the reviewed studies are largely focused on optimization and control of PV–battery and microgrid energy systems. The map also highlights the prominence of reinforcement learning-related terms, including deep RL, agent, state, PPO, TD3, and deep deterministic policy gradient, which confirms the growing role of RL-based approaches in this research area. At the same time, the dense links between these terms and words such as battery, load, grid, energy cost, and HEMS suggest that most of the literature emphasizes supervisory energy management and battery dispatch rather than fully unified converter-level MPPT and battery co-control. Overall, the figure supports the interpretation that the field is evolving toward more intelligent and integrated decision-making frameworks while remaining structurally centered on system-level energy management.

Figure 8. Visualization of co-occurrence patterns among keywords in the studies retained after the PRISMA screening process, produced using VOSviewer 1.6.20.

Figure 8. Visualization of co-occurrence patterns among keywords in the studies retained after the PRISMA screening process, produced using VOSviewer 1.6.20.

4.2. Comparative Study Characteristics: RL and Non-RL Literature

To enable consistent cross-study comparison, each included article was coded using a structured classification framework. The framework captures the degree of integration of PV control, MPPT, battery control, and supervisory energy management functions, as well as the methodological approach, control layer, and relevance to the primary review questions. Each study was assigned the category that best reflects its dominant control structure and contribution.

The control-layer classification was used as an analytical coding dimension rather than as an eligibility criterion. Studies were not excluded merely because their control authority was located at the EMS, reference-generation, or converter-control layer. Moreover, these layers were not treated as mutually exclusive: coordinated and hierarchical architectures spanning multiple layers were explicitly identified and coded. The resulting distributions should therefore be interpreted as a descriptive mapping of the reviewed corpus rather than as proof that the broader literature is divided into isolated or competing control domains.

Table 2 summarizes the coding framework used to classify the selected studies.

Table 2. Classification criteria used for coding the reviewed articles.

Dimension	Code	Definition
Architecture	A1	Separated architecture: PV MPPT and battery/EMS are largely treated as distinct subsystems with little direct coordination.
	A2	Coordinated/layered integrated architecture: multiple controllers or loops are present but explicitly coordinated through supervision, hierarchy, or shared system structure.
	A3	Strongly unified architecture: one controller or one decision framework directly co-determines both PV and battery behavior.
MPPT treatment	M0	MPPT not mentioned explicitly.
	M1	MPPT assumed implicitly, or PV treated as exogenous, fixed, or uncontrollable.
	M2	Conventional external MPPT: A separate standard MPPT loop exists and is not subordinated to system-level objectives.
	M3	Supervisory-coordinated MPPT: A conventional MPPT loop exists but can be overridden, constrained, or coordinated by battery or system objectives.
	M4	MPPT is itself a reinforcement learning decision.
	M5	MPPT and battery control are unified within the same converter/control scheme or decision framework.
PV actuation	B0	No meaningful PV actuation.
	B1	PV treated as exogenous or influenced only indirectly.
	B2	PV actively controlled, but in a separate or loosely coupled manner.
	B3	PV control is strongly integrated into the central decision or control structure.
Battery actuation	C0	No battery included.
	C1	Battery present but not meaningfully controlled.
	C2	Battery actively controlled.
Method	D0	No AI or advanced optimization beyond standard control.
	D1	Rule-based, heuristic, or deterministic scheduling/control.
	D2	Classical control, PI, MPC, or optimization without learning-based control.
	D3	Intelligent non-RL method, such as ANN, ANFIS, fuzzy logic, PSO, or GWO.
	D4	Reinforcement learning.
Layer	E1	Planning, forecasting, day-ahead, or economic scheduling layer.
	E2	EMS, supervisory control, or dispatch layer.
	E3	Converter-level, fast control, or directly electrical control layer.

4.2.1. Architecture of PV–Battery Coordination

Figure 9 compares the architectural structure of RL and non-RL studies. Three broad classes were coded: separated architectures (A1), coordinated/layered integrated architectures (A2), and strongly unified architectures (A3). A clear contrast emerges. The non-RL comparator set is dominated by coordinated/layered architectures, whereas the RL set contains a larger proportion of separated architectures. Strongly unified architectures remain rare in both groups.

Figure 9. Comparison of architectural integration between RL studies ($n=57$) and non-RL studies ($n=44$). A1 denotes separated architectures, A2 coordinated/layered architectures, and A3 strongly unified architectures.

Figure 9. Comparison of architectural integration between RL studies ($n=57$) and non-RL studies ($n=44$). A1 denotes separated architectures, A2 coordinated/layered architectures, and A3 strongly unified architectures.

This distribution is important because it indicates that even when RL is used, most studies do not yet formulate PV and battery behavior within a single tightly unified decision framework. Instead, RL is commonly inserted into an already layered control structure, often at the supervisory or scheduling level. Conversely, non-RL studies more often reflect coordinated multi-loop architectures in which PV, battery, and converter behavior are linked through explicit engineering structure, even if not optimized through a unified learning framework.

This pattern is also visible in specific representative studies. Jha et al. [28] provide a clear example of a strongly unified non-RL converter-level architecture in which MPPT and battery-current regulation are carried out through a shared bidirectional converter. By contrast, Rizki et al. [44] represent a coordinated but layered architecture in which forecasting, a predictive EMS, and converter-level control are tightly linked but remain structurally differentiated across supervisory and execution layers. Dagal et al. [45] and Tavoosi et al. [46] further illustrate converter-oriented PV–battery coordination using metaheuristic and nonlinear control, respectively, but without extending the formulation to a broader sequential decision-making EMS framework. Together, these studies help clarify that stronger PV–battery integration is already present in parts of the non-RL literature, particularly at the converter layer, whereas RL-based work remains concentrated at supervisory levels with PV often treated as exogenous.

4.2.2. Treatment of MPPT

The treatment of MPPT is one of the most revealing dimensions in the dataset. As shown in Figure 10, the RL literature is heavily concentrated in categories where MPPT is either not explicitly addressed, assumed implicitly, or left external to the learning problem. Only a very small number of RL studies formulate MPPT itself as an RL decision or embed MPPT and battery control in the same unified decision framework.

Figure 10. Comparison of MPPT treatment between RL studies ($n=57$) and non-RL studies ($n=44$). The coding distinguishes whether MPPT is absent, implicit, externally conventional, supervisory-coordinated, explicitly an RL decision, or unified with battery control.

Figure 10. Comparison of MPPT treatment between RL studies ($n=57$) and non-RL studies ($n=44$). The coding distinguishes whether MPPT is absent, implicit, externally conventional, supervisory-coordinated, explicitly an RL decision, or unified with battery control.

By contrast, the non-RL comparator set contains more studies in which MPPT is explicitly coordinated or implemented through a separate conventional loop that remains visible in the system formulation. This difference supports one of the central arguments of the review: much of the RL literature in PV–battery systems focuses on battery dispatch, energy management, or scheduling while leaving PV operating-point selection outside the learning loop. In other words, the key gap is not merely the absence of learning, but the persistent separation between energy harvesting and energy management.

4.2.3. PV Actuation and Battery Actuation

Figure 11 and Figure 12 compare how directly RL and non-RL studies control PV-side and battery-side variables. The RL literature is strongly concentrated in categories where PV is treated as exogenous or only indirectly represented, whereas the non-RL comparator set contains many more studies with active PV-side control at the reference or converter level. This reinforces the view that RL-based PV–battery work has largely evolved as EMS-oriented research rather than as converter-level PV-control research.

Figure 11. Comparison of PV actuation between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are concentrated in categories where PV is treated as exogenous or only indirectly controlled, while non-RL studies more often include active PV-side control.

Figure 11. Comparison of PV actuation between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are concentrated in categories where PV is treated as exogenous or only indirectly controlled, while non-RL studies more often include active PV-side control.

Figure 12. Comparison of battery actuation between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are strongly concentrated in active battery-control categories, highlighting the battery-centric nature of most RL formulations.

Figure 12. Comparison of battery actuation between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are strongly concentrated in active battery-control categories, highlighting the battery-centric nature of most RL formulations.

The opposite pattern is observed on the battery side. RL studies overwhelmingly involve active battery dispatch or charging/discharging decisions, confirming that storage is the main actuation target in RL-based formulations. Non-RL studies, in contrast, more frequently operate closer to charging-control or converter-reference implementations. Taken together, these two figures show that the dominant RL paradigm in this domain is battery-centric supervisory control with exogenous PV, rather than genuinely integrated PV–battery co-control.

4.2.4. Methodological Distribution

Figure 13 summarizes the methodological distribution across the two sets. As expected, the RL corpus is dominated by reinforcement learning-based and hybrid formulations, while the non-RL comparator set is concentrated in classical control, optimization-based approaches, and intelligent non-RL methods such as ANN, fuzzy, or metaheuristic-based control.

Figure 13. Methodological distribution across RL studies ($n=57$) and non-RL studies ($n=44$). The non-RL set includes conventional, rule-based, optimization-based, and intelligent non-RL methods, while the RL set includes reinforcement learning and hybrid approaches.

Figure 13. Methodological distribution across RL studies ($n=57$) and non-RL studies ($n=44$). The non-RL set includes conventional, rule-based, optimization-based, and intelligent non-RL methods, while the RL set includes reinforcement learning and hybrid approaches.

This figure is less important as evidence of the research gap by itself, since the classification partly reflects the sampling logic of the review. However, it remains useful because it shows that the contrast between the two groups is not simply “RL versus not RL” in name only; it also corresponds to broader methodological tendencies. The non-RL set provides a meaningful baseline for comparison because it includes multiple established alternatives that have historically addressed PV control, MPPT coordination, converter regulation, and battery management without RL.

4.2.5. Control Layer

The clearest structural distinction appears in the control-layer comparison shown in Figure 14. RL studies are concentrated at the EMS/supervisory and scheduling layers, whereas non-RL studies have substantially stronger representation at the converter and directly electrical control layer. This pattern is consistent with the findings above on MPPT treatment and PV actuation. It indicates that most RL-based PV–battery studies do not formulate the problem at the electrical actuation layer where converter duty cycle, voltage references, or MPPT-related operating points are directly optimized.

Figure 14. Comparison of control-layer emphasis between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are concentrated at the EMS/supervisory and scheduling layers, while non-RL studies are more strongly represented at the converter-level control layer.

Figure 14. Comparison of control-layer emphasis between RL studies ($n=57$) and non-RL studies ($n=44$). RL studies are concentrated at the EMS/supervisory and scheduling layers, while non-RL studies are more strongly represented at the converter-level control layer.

This layer separation helps explain why integrated MPPT and battery control remains uncommon in the RL literature. It is not only that MPPT is often omitted; rather, the majority of RL work is posed at a control layer where PV generation enters as an input, forecast, or upper bound, not as a directly optimized electrical decision variable. Consequently, the converter-level and EMS-level strands of the literature remain only weakly connected.

4.3. Implications of the Comparative Coding

Taken together, the comparative coding reveals a consistent structural pattern. The RL literature is primarily organized around storage dispatch, supervisory control, and scheduling, with PV generation often treated as fixed, forecasted, or implicitly MPPT-operated. The non-RL literature, by contrast, more frequently engages the converter layer and explicit PV control, including externally implemented or supervisory-coordinated MPPT structures. Strongly unified PV–battery control remains uncommon in both sets.

These observations are important because they sharpen the interpretation of the RL corpus reviewed in this article. The research gap is not simply that RL has not yet been used widely enough; rather, the gap is that RL has been used predominantly at higher supervisory abstraction levels for fully integrated PV–battery optimization. The central observation is not that EMS-level RL is incomplete or should be replaced by converter-level learning. The reviewed EMS studies make substantial contributions within battery dispatch, scheduling, economic, grid-interaction, and uncertainty-management objectives. A different and comparatively underrepresented combination is the explicit connection of converter-level PV operating-point decisions with battery-aware long-horizon objectives within the same RL formulation or coordinated benchmark.

4.4. Summary of Findings

The reviewed literature demonstrates significant progress in the application of artificial intelligence and RL to PV–battery systems, particularly at the energy management and scheduling layer. RL has been successfully applied to battery dispatch, grid exchange optimization, EV coordination, multi-agent energy trading, and degradation-aware scheduling under uncertainty. Continuous actor–critic methods, hybrid RL–optimization architectures, and structured constraint-handling mechanisms are increasingly common.

However, this methodological maturity is concentrated at the supervisory EMS level. PV generation is treated in the overwhelming majority of studies as an exogenous input or as an upper-bounded dispatchable source with implicit MPPT, rather than as a controllable operating-point decision. A small number of recent studies do move closer to integrated control by embedding RL-based MPPT and battery charging/discharging within a single controller; however, these efforts remain small-scale and primarily converter-centric, rather than establishing a broadly adopted unified framework for joint PV operating-point optimization and storage-aware EMS across realistic microgrid settings [48,49]. As a result, most existing formulations optimize energy allocation and curtailment rather than energy harvesting itself, even though the converter-control literature now clearly shows that DRL can act directly on duty cycle, switching, and MPPT objectives when the problem is posed at the power-electronics layer [50].

This persistent separation between PV operating-point control and battery dispatch highlights a structural gap in the literature: sequential decision-making frameworks rarely integrate converter-level PV actuation with storage-aware energy optimization. The synthesis in the following section therefore examines this gap systematically through PRQ1–PRQ4 and establishes the foundation for a unified continuous-control framework.

4.5. Synthesis of Evidence by PRQ

This subsection synthesizes the evidence to address PRQ1–PRQ4. Across the reviewed studies, RL is predominantly applied at the energy management and scheduling layer, including grid exchange, battery dispatch, and load or EV coordination, as listed in Table A1. In contrast, inverter-level PV operating-point control (i.e., MPPT treated as an explicit decision variable) is typically assumed, abstracted, or implemented via conventional control schemes external to the learning problem. At the same time, several non-RL studies, as listed in Table A2, show that this separation is not inevitable. Jha et al. [28] demonstrate that MPPT and battery-current regulation can be unified at the converter layer through a shared bidirectional converter, while Rizki et al. [44] show that PV references, battery references, and converter control can be coordinated through a predictive architecture spanning forecasting, an EMS, and low-level execution. Dagal et al. [45] and Tavoosi et al. [46] provide additional evidence that non-RL metaheuristic and nonlinear-control approaches can directly actuate both PV-side and battery-side dynamics. These studies are therefore useful because they indicate that the main limitation in much of the RL literature is not the presence or absence of coordination in principle, nor whether supervisory or converter-level control is preferable. It is whether the learning formulation directly includes PV operating-point decisions, battery-side objectives, or both and how coordination between the associated control layers is evaluated.

4.5.1. PRQ1: Limitations of Decoupled Architectures (MPPT vs. Storage/EMS)

Across the reviewed literature, PV generation is modeled in almost all cases as an exogenous input to the EMS, implicitly assuming independent MPPT or a fixed PV yield model. Recent converter-oriented DRL studies confirm that RL can, in principle, control MPPT and converter duty-cycle decisions directly, but this strand of work remains methodologically separate from most PV–battery EMS formulations [43,50,51]. RL actions regulate battery and grid power, while PV enters the state as measured or forecast output without a controllable operating-point variable [29,42,52,53,54,55,56,57]. When PV “control” is included, it is typically expressed as curtailment or scaling at an aggregated DER level rather than converter duty-cycle or $V_{PV}$/$I_{PV}$ regulation [58]. The separation between energy harvesting and energy allocation is therefore preserved. A representative example is the PSCAD-based PV–battery hybrid system by Go and Choi [59], in which the PV converter performs P&O-based MPPT, the BESS regulates the DC-link through a bidirectional converter, and the inverter controls system output power. Although this constitutes a genuinely integrated PV–battery control architecture at the hardware and converter level, the control objectives remain partitioned across separate subsystems rather than being learned or optimized jointly within a unified policy.

A limited exception is provided by recent small-scale PPO-based studies that jointly control PV MPPT, BESS charging/discharging, and load-side voltage regulation within a centralized RL controller, using duty-cycle and modulation actions rather than dispatch-only decisions. These studies are important because they demonstrate feasibility of integrated learning-based control, yet they remain focused on compact DER test systems rather than the broader uncertainty-aware, degradation-aware, or market-coupled EMS formulations that dominate the literature [48,49].

This abstraction spans diverse system types, including wind–PV–storage scheduling [60], residential PV–battery EMSs [61,62], EV charging stations [63,64], multi-asset microgrids [65,66], and actor–critic-based energy trading [67,68]. Even in degradation-aware, forecast-integrated, hybrid RL–optimization, Lyapunov-stable, or robust $H_{\infty }$-inspired formulations, PV power remains a disturbance or upper-bounded source rather than an electrically optimized variable. The MPPT–EMS separation is thus primarily architectural rather than algorithmic: converter-level RL for MPPT and duty-cycle control clearly exists, but it is rarely co-developed with storage-aware supervisory optimization in a unified formulation [48,50].

Practical implications of layered design are illustrated by Qiu et al. [69], who show that legacy charge-control logic can become mismatched under heterogeneous storage chemistries, leading to inefficiencies and premature limiting. Although focused on batteries, this result highlights a broader limitation: embedded assumptions in decoupled layers may fail when dynamics are tightly coupled.

Recent continuous and uncertainty-aware EMS approaches maintain this separation. Multi-agent DDPG scheduling [70], multi-timescale SAC corrective control [71], and various continuous-action EMS formulations [41,72,73,74,75,76,77] all treat PV as forecasted or measured input while policies act on storage and grid exchange. Even when a PV utilization variable is introduced, it reflects dispatch or curtailment rather than converter operating-point control.

A smaller subset incorporates converter-adjacent variables while retaining MPPT as an external conventional loop. TD3 has been used to augment DC-bus-voltage regulation on top of classical PI control, with MPPT implemented via fixed P&O [78]. Similarly, RL-based supervisory controllers for PV feed-in and voltage quality retain conventional voltage/reactive-power control and treat PV generation as exogenous [79]. Even near the converter layer, MPPT and storage dispatch are usually not co-optimized, although a small number of recent centralized RL controllers do jointly regulate PV MPPT, bidirectional battery charging, and load voltage in compact DER systems [48]. The system presented therein illustrates a centralized, integrated converter-level RL architecture in which a single agent observes coupled electrical and energy-state variables, including changes in PV power and voltage, BESS power, state of charge, load power, and output voltage, and issues PWM-related duty-cycle commands to both the PV-side and battery-side converters. This differs fundamentally from mainstream PV–battery EMS formulations, where PV generation is treated as an exogenous disturbance and control authority is limited to supervisory dispatch. By placing MPPT-related PV control and battery charging/discharging within a single learned control loop, the architecture represents a rare counterexample to the dominant separation between energy harvesting and energy management in the reviewed literature.

These studies represent rare counterexamples to the dominant separation between energy harvesting and energy management, because PV-side control actions and battery-side actions are placed within the same learned control loop. Abstracting from these emerging integrated formulations, Figure 15 summarizes the target co-control structure implied by the architectural gap identified in this review. The controller observes PV electrical variables, battery state information, and load/grid conditions, and jointly produces PV-side, battery-side, and grid/load-side control actions. The key distinction from mainstream EMS-oriented RL is that the PV operating point is no longer treated only as an exogenous input or forecasted resource but becomes an explicit decision variable through commands such as a PV duty cycle or PV voltage reference.

Figure 15. Conceptual synthesis of a unified PV–battery RL co-control architecture. The RL controller receives PV measurements, battery state information, and load/grid variables and jointly determines PV-side, battery-side, and grid/load-side control actions. Unlike purely supervisory EMS formulations, the PV operating point is treated as an explicit decision variable through commands such as the PV duty cycle or PV voltage reference, enabling tighter coordination of MPPT, battery management, and system-level objectives.

With that, decoupling simplifies modeling and training but prevents unified reasoning over PV extraction, DC-bus dynamics, inverter stress, and battery constraints. MPPT trade-offs therefore remain outside the learning loop in the vast majority of reviewed studies, underscoring the structural nature of the architectural gap addressed in this review, while also indicating that recent proof-of-concept integrated controllers have begun to narrow that gap at the small scale [43,48,51].

4.5.2. PRQ2: Control Variables and Actuation Levels in Learning-Based PV–Battery Approaches

Across the reviewed literature, control variables are overwhelmingly defined at the EMS supervisory layer, even though a parallel converter-control literature shows that DRL can also act directly on PWM duty cycle, modulation variables, phase shift, and MPPT objectives [43,50,51]. Some recent works formulate RL control directly at the MPPT layer, using a continuous-action SAC agent to adjust the DC–DC converter duty cycle for global maximum power point tracking under irradiance variation, temperature changes, and partial shading, showing a clearly converter-level rather than supervisory EMS-level control formulation [33]. Others have also modeled PV MPPT under partial shading as an MDP and compare SAC and DDPG using converter duty-cycle perturbation as the control action, showing a clearly converter-level continuous-control formulation [80]. Typical actions include (i) battery charge/discharge and grid exchange power [29,42,55,57,81]; (ii) allocation ratios among PV, load, storage, and grid [56,82]; and (iii) EV charging and tariff-driven decisions in EV-integrated systems [52,53,54,81,83]. In all cases, actuation occurs at the energy scheduling level. This distinction matters because “RL for PV–battery systems” now spans at least two different actuation regimes: dominant EMS-level scheduling formulations that regulate battery, grid, and flexible-load power and a separate converter-level strand in which DRL manipulates duty cycle, switching, or MPPT-related variables. A representative example is the HEMS formulation by Si et al. [84], which explicitly defines different action sets for different device classes: binary on/off actions for interruptible and non-interruptible constant-power appliances, continuous power actions for thermostatically controlled loads, and bidirectional continuous charge/discharge actions for energy storage devices. Their state design likewise combines supervisory variables such as electricity price, PV output, and non-schedulable load with device-specific states such as task progress, indoor temperature, and battery energy state. This is useful because it shows that recent DRL-based PV–battery/home-energy studies can employ rich mixed discrete–continuous-action designs while still locating control authority at the EMS layer; PV appears as forecasted available generation rather than as a directly controlled MPPT duty-cycle or operating-voltage decision.

Discrete formulations typically use quantized battery power levels or enumerated operating modes [61,62,63], sometimes restricting actions to precomputed feasible power-balance combinations [64,66]. Deep RL approaches replace tabular representations with neural approximators while retaining EMS-level control [65]. Continuous actor–critic methods directly parameterize battery or grid power as real-valued variables [67,68] and are now widely adopted using DDPG, TD3, or SAC across residential, community, and grid-scale applications [41,70,71,72,73,74,75,76,77]. Multi-agent decompositions distribute scheduling across time-indexed agents or interacting prosumers [70,74]. Some works treat residential PV–battery energy management as a discrete–continuous hybrid control problem rather than a purely discrete or purely continuous one. Huang et al. [85], for instance, couple discrete appliance scheduling with continuous HVAC and battery actions, highlighting that many practical RL-based formulations remain hybrid supervisory EMS problems rather than converter-level MPPT control problems.

Parallel to EMS-level developments, a growing body of literature investigates RL-based control at the power-electronics layer, where actions correspond to converter duty cycles, voltage references, or switching behavior rather than energy dispatch variables [50].

Despite this progression from discrete to continuous neural policies, the dominant actuation target in integrated PV–battery RL remains supervisory dispatch rather than converter-level PV control. In most studies, “continuous control” refers to battery, grid, or load power references, whereas direct optimization of converter duty cycle or PV operating point is treated separately or assumed implicitly through external MPPT loops.

MPPT is therefore rarely modeled as a decision variable in EMS-focused PV–battery studies, with only limited recent exceptions in which PPO-based controllers explicitly command PV MPPT duty cycle alongside BESSs and load-control actions [48,49]. PV generation is typically treated as irradiance-derived or forecasted available power, or as an upper-bounded source with implicit MPPT. Even when PV flexibility is included, it is expressed as curtailment or allocation of already-produced energy rather than DC–DC operating-point control [52,56,58]. Recent continuous-action EMS formulations introduce PV utilization or feed-in variables [79,86], but these remain dispatch-level quantities.

Thus, learning-based PV–battery approaches show increasing sophistication in EMS-level scheduling, yet control authority consistently remains above the converter layer in almost all studies, leaving PV operating-point optimization outside the RL loop except for a small number of recent converter-centric integrated controllers [43,48,51].

A limited number of studies extend RL action spaces to include both converter-level and supervisory variables. For example, centralized DRL controllers have been demonstrated for simultaneous MPPT, battery charging, and voltage regulation using direct duty-cycle control [48]. However, such approaches are typically confined to single-system prototypes, without incorporation of multi-timescale scheduling, market interaction, or distributed coordination. Consequently, despite isolated examples of low-level actuation, the dominant paradigm in the literature continues to define control variables at the EMS layer, leaving converter-level decisions structurally separated from system-level optimization.

4.5.3. PRQ3: RL Formulation Design for Joint PV–Battery Optimization and Evidence on Discrete vs. Continuous Control

MDP/POMDP Construction

Most works adopt a standard MDP in which the state combines battery SoC with exogenous drivers such as PV generation, load, prices, and time features. This pattern is consistent across residential HEMSs [53,54,87], community dispatch [42,57], and microgrid scheduling [56]. States reflect supervisory energy variables rather than electrical-layer dynamics. Härtel and Bocklisch [29] used PPO with an LSTM-based structure for PV–battery energy management, reporting performance comparable to MPC and superior to rule-based strategies. Xu et al. [30] formulated a residential PV–battery system as an MDP and applied model-based RL to improve training efficiency while increasing self-consumption and reducing operating cost.

Extensions beyond fully observable MDPs are limited. Härtel and Bocklisch [29] formulated a POMDP and employed an LSTM encoder within PPO to infer latent states from short histories. Multi-agent formulations address decentralized storage coordination with continuous actions and explicit coupling [88]. Despite structural variations, PV generation remains modeled as an exogenous input at the EMS layer. By contrast, converter-level DRL formulations use electrical measurements such as voltage, current, temperature, and duty cycle within the state, and define actions directly over the PWM duty cycle, switching signals, or other modulation variables. This confirms that the omission of PV operating-point control in most PV–battery EMS studies is not due to an intrinsic limitation of RL formulations but to the control layer at which the problem is posed [50].

Offline-to-Online Policy Approximation

Several works derive near-optimal dispatch policies offline and approximate them for real-time execution. Keerthisinghe et al. [89] generated optimal state–action trajectories using dynamic programming and trained a policy function approximator, enabling near-optimal real-time decisions without repeated online optimization. Mbuwir et al. [90] similarly combined fitted Q-iteration with clustering-based transfer learning, approaching MILP performance with reduced convergence time. These methods often treat PV output as an external input rather than a controllable variable, limiting MPPT–storage co-optimization.

Action Spaces and Algorithm Choices

A central design axis concerns discrete- versus continuous-action spaces. Discrete formulations, often paired with value-based methods, remain common in residential HEMSs and small microgrids, where battery power is quantized or operating modes enumerated [53,54,69,83,87]. Tabular Q-learning and Watkins Q($\lambda$) are still used in degradation-aware scheduling [61,62], while SARSA has been applied to wind–PV–storage coordination [60]. Hybrid variants embed deterministic solvers within discrete RL loops (e.g., MINLP-guided Q-learning) [66]. Deep value-based extensions such as DDQN expand state dimensionality but retain discretized actions [65,91]. Discrete-action formulations can improve energy capture but restrict fine-grained modulation [83]. Continuous actor–critic methods address this limitation by parameterizing actions as real-valued variables. For example, Zhou et al. [41] used DDPG for continuous coordination of PV–battery dispatch and EV demand, improving supply–demand balance and reducing operating cost relative to rule-based control. However, the term “continuous action” is used across different physical actuation layers in the literature. As summarized conceptually in Figure 16, continuous actions may correspond to supervisory dispatch variables, intermediate setpoint references, or direct converter-level commands. Figure 16 also distinguishes control actions according to their abstraction level, physical actuation variables, and typical operating time scale. This classification does not imply a hierarchy of technical maturity or importance. Supervisory EMS controllers address long-horizon and system-wide objectives, including power balance, battery utilization, economic operation, and grid interaction, whereas converter-level controllers provide fast and accurate electrical regulation. The intermediate coordination layer communicates system-level objectives and constraints to the converter controllers through appropriate references. Consequently, effective PV–battery control generally depends on coordinated operation across these layers, often through bidirectional coordination between them, rather than replacement of one layer by another.

Figure 16. Complementary control layers and associated action variables in RL-based PV–battery systems. Supervisory EMS actions address system-level objectives such as battery power scheduling, grid exchange, load management, and energy routing. The coordination layer translates these objectives and operational constraints into electrical references, including PV-voltage, battery-current, DC-bus-voltage, and converter-power references. Converter-level controllers implement these references through duty-cycle, modulation, switching, or phase-shift actions. The layers operate at different time scales and abstraction levels but are complementary rather than hierarchical in technical value; practical high-performance systems generally rely on their coordinated operation. Converter-level actions directly manipulate power-electronic variables and are therefore specifically relevant when MPPT is formulated as a fast electrical-control problem.

Continuous-action actor–critic methods directly parameterize battery or grid power as real-valued variables, avoiding discretization at the EMS layer. In the converter-control literature, continuous actions are instead naturally associated with duty-cycle adjustment, phase-shift control, and MPPT-related modulation, again underscoring that “continuous control” in the broader RL literature spans materially different actuation layers [50]. Examples include PPO-based scheduling [82], SAC under uncertainty [42], TD3 for microgrid and charging dispatch [55,81,88], and integral Q-learning or neuro-evolution-enhanced actor–critic schemes [67,68]. Continuous control is now widely adopted for multi-device coordination across storage, EVs, flexible loads, and multi-energy systems [41,71,72,73,75,76,77]. Multi-agent actor–critic architectures further scale coordination temporally or across prosumers [70,74].

Recent trends include safety-aware continuous RL with differentiable projection layers for hard constraint enforcement [86] and the structured benchmarking of PPO, SAC, TD3, and A2C against MILP or rule-based baselines [29,30,39,92,93].

Discrete vs. Continuous Evidence

Empirical evidence suggests advantages of continuous control when decisions correspond to power setpoints rather than mode switches. PPO has been shown to outperform DDQN/DDPG variants in convergence and profitability [82]. SAC demonstrates improved cost performance and constraint behavior under uncertainty relative to PPO and DDPG [42]. Projection-based continuous PPO reduces optimality gaps versus discrete DQNs while guaranteeing feasibility [56].

Conversely, discrete HEMS studies reveal quantization limitations. MILP outperforms DQNs under deterministic conditions [53], and MPC surpasses DQNs under accurate forecasts, with DQN performance being sensitive to reward shaping and action design [54]. Tabular methods remain competitive in small-scale problems but face scalability and resolution limits [60,62], while deep value-based discrete methods mitigate state complexity but retain action quantization [65]. Continuous actor–critic approaches typically yield smoother trajectories and improved convergence, particularly when degradation or stability-aware objectives are included [67,68].

However, in the majority of the reviewed RL studies, “continuous control” refers to supervisory dispatch decisions operating at hourly or multi-minute resolution rather than converter-level electrical control. Only a limited number of works extend continuous decision variables to converter duty-cycle control or direct PV operating-voltage optimization within integrated PV–battery formulations, although converter-level DRL studies and a small number of recent PPO-based integrated DER controllers demonstrate that such actions are technically feasible [48,50]. Consequently, the discrete-versus-continuous distinction in much of the reviewed literature primarily concerns scheduling and dispatch granularity rather than true cross-layer PV–battery control integration, with MPPT still being treated as external to the learning loop in most EMS-oriented formulations.

4.5.4. PRQ4: Practical Implementation Considerations (Training Environments, Constraints, Observability, Robustness, and Validation)

Training Environments and Data

Training and evaluation are predominantly simulation-based, typically using real-world datasets for PV generation, load demand, and electricity prices (e.g., Pecan Street or market datasets) [42,53,54,56,57]. Some works employ richer pipelines, such as SAM-generated PV data or simulator-based forecasts [52,58]. Forecast integration is increasingly common, including RNN-based 24-h PV/load prediction [65] and tariff forecasting in EV-integrated EMSs [63]. Pre-training on historical data followed by limited online adaptation has also been explored [60].

Uncertainty handling includes scenario-based learning [70], multi-timescale formulations under forecast errors [71], and offline-to-online approaches combining imitation learning from MIQP experts with SAC refinement [75]. However, robustness to model mismatch, sensor noise, and forecast bias is rarely quantified explicitly. Some works illustrate that recent EMS-focused RL studies increasingly address practical issues such as network-constraint enforcement, uncertainty adaptation, and scalability but still do so at the supervisory dispatch layer rather than through integrated MPPT/converter-level control [94]. Validation remains largely simulation-bound. Early hardware-in-the-loop (HIL) demonstrations exist for continuous PV–battery EMS architectures [95], but PV remains exogenous and converter-level MPPT or duty-cycle control is not integrated. More broadly, the recent review literature also emphasizes that hardware validation remains limited and benchmarking practices remain non-standardized in standalone and converter-oriented microgrid control research. Esfahani et al. [96] employed GAN-based scenario generation to expose controllers to realistic variability patterns, improving resilience under high-variance operating conditions. Although isolated integrated RL demonstrations now exist for small DER systems with MPPT, bidirectional charging, and load-voltage control, the literature still lacks a mature validation pipeline for unified PV operating-point and battery co-optimization under realistic converter dynamics and deployment constraints [48,50,97].

Constraint Enforcement Mechanisms

Practical deployment requires explicit consideration of physical constraints such as SoC bounds, charge/discharge limits, voltage stability, and converter operating regions [57,94]. Constraint handling ranges from heuristic penalty shaping to optimization-based feasibility guarantees. Most approaches rely on reward penalties for SoC and power violations [53,54,55,87] or action clipping [52], which are simple but lack guarantees. More structured methods include pre-filtered feasible action spaces and hybrid RL–optimization schemes [64,66], Lagrangian relaxation [57], Lyapunov-based updates [67], robust $H_{\infty }$-inspired reward design [68], and externally coordinated multi-agent balancing [88]. More structured approaches explicitly enforce admissibility during policy execution. Huang and Wang [82] introduced a safety layer to project actions onto feasible regions for continuous battery dispatch via PPO. Fan and Wang [42] applied SAC with entropy regularization for robust community battery scheduling under uncertainty. Eyimaya [43] incorporated a safe action projection layer for an off-grid PV–battery nanogrid, ensuring physical admissibility during learning and execution. The general constraint-aware control pattern emerging from these studies is summarized conceptually in Figure 17.

Figure 17. Constraint-aware reinforcement learning loop for integrated PV–battery control. The RL policy generates a raw continuous-action vector, including PV-side and battery-side control commands. A safety or projection layer then enforces operational limits related to battery SoC, battery power, DC-bus voltage, and converter duty cycle before applying the feasible action to the PV–battery plant. Feedback from the plant updates the measured state and provides reward and constraint information for policy learning.

Projection-based safety layers provide stronger feasibility guarantees. This trend is reinforced by recent SAC-based nanogrid work, where safe action projection is used to enforce SoC and charge/discharge limits before actions reach the plant. However, these guarantees remain focused on supervisory energy variables rather than converter switching constraints or PV operating-point feasibility [43]. Early feasibility checks [82] have evolved into differentiable optimization-based projections enabling gradient backpropagation through convex solvers [56,86]. Despite these advances, constraint handling remains focused on supervisory scheduling variables [41,70,73,76,77]. Converter-level protection, DC-bus dynamics, and subsecond constraints are generally not addressed in EMS-oriented studies, even though recent perspective and review papers explicitly identify the continued separation between RL-based EMS decisions and certified low-level protection/control layers as a major deployment constraint [50,98].

Observability and SoC Estimation

Most studies assume direct, noise-free SoC availability [55,58], with little discussion of estimation uncertainty. Reliable state estimation is equally critical. Talha et al. [99] proposed neural network-based SoC estimation using voltage and current measurements, highlighting the importance of accurate observability for learning-based control. Partial observability is rarely modeled explicitly. Notable exceptions formulate the problem as a POMDP and use LSTM-based encoders within PPO to reconstruct latent states [29]. However, systematic analysis of estimation errors, sensor noise, and their impact on closed-loop stability remains limited. Recent perspective work likewise notes that standardized state representations and uncertainty-modeling frameworks remain underdeveloped, which further limits reproducibility and robustness assessment in RL-driven DC microgrid applications [98].

Validation and Physical Realism

Validation is predominantly simulation-based. Explainability tools have been introduced to interpret policy dependence on price and demand signals [87]. Some works verify RL schedules against physical microgrid models to ensure PCC voltage compliance [100], though this remains offline validation.

Physical degradation modeling is increasingly incorporated. Inverter (IGBT) wear has been embedded via mission-profile-based modeling [101] and battery aging via semi-empirical degradation laws integrated through structured reward shaping [102]. While these improve physical relevance, validation remains at supervisory time scales. Converter-control reviews similarly identify sim-to-real transfer, safe exploration, hardware deployment constraints, and benchmarking gaps as unresolved issues, indicating that the challenge is not merely the missing algorithms but the absence of validated cross-layer integration procedures [50].

Synthesis and Implication

The literature demonstrates methodological maturity of RL at the EMS layer, including actor–critic methods, hybrid RL–optimization architectures, multi-agent coordination, degradation-aware objectives, and projection-based safety mechanisms. At the same time, recent converter-control reviews show that DRL is also capable of MPPT, duty-cycle, and inverter-level control when the problem is formulated at the power-electronics layer. Yet these two strands remain largely disconnected. In the dominant PV–battery EMS literature, PV generation is still treated as exogenous or implicitly MPPT-operated, and converter-level operating-point control is almost never included as a learning variable. A few recent proof-of-concept studies have begun to unify PV MPPT, bidirectional battery charging, and load-voltage regulation within a single RL controller, but they remain limited in scale and do not yet establish a general framework for safety-aware, storage-aware, and converter-feasible joint optimization. This persistent disconnect motivates unified continuous-control formulations that jointly optimize PV operating-point control and battery dispatch while enforcing converter-level feasibility, protection constraints, and dynamic coupling.

5. Discussion and Research Gaps

This article reviewed the progression of MPPT techniques from conventional perturbation-based algorithms to intelligent approaches based on ANNs, deep learning, and RL. The trajectory is clear: increasing PV system complexity and operating non-stationarity have driven methods toward higher adaptability and richer representations, particularly under fast irradiance variations and partial shading.

Conventional MPPT methods such as P&O and IncCond remain attractive due to low complexity, yet comparative studies consistently report the same structural weaknesses: steady-state oscillations, sensitivity to irradiance transients, and vulnerability to local maxima under partial shading. Improved variants and optimization-assisted approaches mitigate specific symptoms but remain rooted in instantaneous, local-search logic.

Supervised learning methods (ANN and deep learning) reduce oscillations by replacing perturbation with nonlinear regression from measurements to an estimated optimum. While these approaches often achieve high tracking efficiency under trained conditions, their performance remains tightly coupled to offline dataset coverage, and they generally lack reliable online adaptation when conditions deviate from the training distribution.

Reinforcement learning introduces a different paradigm by optimizing policies through interaction and cumulative reward. Discrete-action RL validates the sequential decision formulation but remains constrained by action quantization. Continuous-action actor–critic methods better match power-electronic control variables and frequently report smoother duty-cycle trajectories and improved transient performance. However, much of the RL MPPT literature still optimizes instantaneous PV power and relies predominantly on simulation-based validation.

A central finding of this review is that most RL-based PV–battery studies are formulated at the EMS or supervisory layer, where battery dispatch, grid exchange, or load coordination are optimized, while PV operating-point selection is assumed, abstracted, or delegated to a separate MPPT loop. Thus, the main limitation in the literature is not simply that RL has been used too little but also that it has often remained concentrated primarily at supervisory control layers rather than extending to converter-level PV operating-point optimization and unified MPPT–battery coordination.

A consistent insight across PV–battery studies is that strict instantaneous MPPT is not necessarily system-optimal once storage is present. PV operating-point selection influences battery charging current, SoC trajectory, conversion losses, and potentially degradation. Consequently, treating MPPT and battery energy management as separate layers can yield conflicting actions, avoidable curtailment, and inefficient cycling, particularly under variable loads and constrained SoC headroom.

These observations support reframing MPPT in PV–battery systems as part of a coupled, constrained, long-horizon optimization problem rather than an isolated instantaneous maximization task.

More specifically, the literature reveals a persistent separation between converter-level PV control and system-level energy management. This layer mismatch helps explain why unified formulations that jointly optimize duty-cycle-level PV actuation and battery-aware long-horizon objectives remain relatively uncommon.

The literature on PV–battery systems reveals several consistent insights:

• Strict instantaneous MPPT can become sub-optimal when storage, converter, or grid constraints are present.
• Decoupled MPPT and EMS architectures may insufficiently represent temporal interdependence.
• Reinforcement learning provides one possible framework for unified, long-horizon optimization.
• Continuous-action formulations are particularly well aligned with converter-level control.

Nevertheless, fully integrated continuous-action frameworks that simultaneously optimize PV operating-point selection and battery dispatch remain comparatively underexplored. Many existing works either assume fixed MPPT operation or treat PV generation as exogenous.

This gap motivates the investigation of unified formulations in which duty-cycle control and storage scheduling are optimized jointly within a single sequential decision-making framework.

5.1. Identified Research Gaps

Based on the reviewed literature, several gaps remain:

• Objective formulation gap: Many approaches, including RL-based MPPT, continue to optimize instantaneous PV power even when long-term metrics (battery stress, degradation-aware operation, and efficiency over time) are dominant.
• Coordination gap: MPPT and battery management are often treated as separable problems, leaving the PV–converter–battery coupling underexploited.
• Control-layer gap: Much of the RL literature is formulated at the supervisory or EMS layer, where PV generation is treated as exogenous or implicitly MPPT-operated, rather than at the converter layer where PV operating-point selection can be directly optimized.
• State observability gap: Many integrated control formulations assume reliable access to battery internal states such as SoC and, sometimes, degradation-related indicators, even though these variables must be estimated online and may be uncertain, noisy, or model-dependent in practice.
• Adaptation gap: Supervised MPPT methods remain largely offline-trained and can degrade under operating regimes not represented during training (e.g., evolving parameters, atypical shading patterns, and aging).
• Control fidelity and constraint gap: Discrete abstractions, reward-only constraint penalties, and action clipping can be poorly aligned with continuous converter control and strict safety constraints (SoC, current, and voltage), motivating more principled constrained/safe RL formulations.
• Training-efficiency gap: Coupled PV–battery environments are high-dimensional and temporally dependent, which increases sample requirements and computational burden for RL training and limits reproducibility and practical online adaptation.
• Validation gap: Integrated PV–battery learning-based controllers are still predominantly evaluated in simulation; systematic hardware validation and sim-to-real methodologies remain limited.
• Benchmarking gap: Cross-study comparison remains difficult because the literature uses heterogeneous PV–battery architectures, irradiance and load profiles, reward definitions, constraint treatments, and evaluation metrics, making it hard to isolate the true benefit of integrated continuous-action control.

5.2. Time-Scale Separation and Cross-Layer Coordination

Classical time-scale separation remains justified in PV–battery systems because fast current, voltage, and DC-link regulation operate at different bandwidths and impose distinct stability, protection, communication, and computational requirements from supervisory energy management [103,104]. The principal limitation is therefore not the layered architecture itself but insufficient coordination among the objectives assigned to its layers. For example, an MPPT controller may maximize instantaneous PV power without considering battery-SoC headroom, charging limits, degradation, or future demand. Cross-layer coordination can address these conflicts by sharing states, constraints, and references while retaining established fast inner loops. Thus, the decision context may be integrated without combining every physical control time scale.

The reviewed evidence illustrates both the potential and the limitations of greater integration:

• Multi-time-scale and hierarchical coordination: Watari et al. [105] reported an electricity-cost reduction of up to $47.5\%$ using a coordinated multi-time-scale formulation relative to the two evaluated single-time-scale formulations. Hu et al. [71] combined hourly optimization with real-time DRL corrective control, noting that complete reliance on a single learning agent may reduce reliability and interpretability. Liu et al. [83] directly compared a hierarchical formulation with a packaged agent that jointly selected power-purchasing and charging decisions. The hierarchical structure achieved a higher reward and less oscillatory convergence, indicating that an enlarged joint action space can impair learning.
• Centralized converter control: Wiencek and Ghosh [48] used one PPO agent to control PV MPPT, battery charging/discharging, and inverter voltage regulation. Voltage THD decreased from approximately $1.70\%$ to $0.23\%$ relative to the evaluated conventional controllers. However, the agent occasionally extracted less PV power than P&O, and validation was limited to a simulated 200 W system without hardware testing or a formal closed-loop stability proof.
• Computational trade-offs: Härtel and Bocklisch [29] trained 216 PPO–LSTM policy configurations, with full-run performance assessed after 48 h, but obtained energy costs only similar to or slightly better than MPC. Conversely, Cai et al. [93] reported a $37.7\%$ improvement in their evaluated performance measure relative to TD3 using MPC–RL. Although offline training was longer, each online MPC calculation required only $0.08$ s within a 15 min supervisory interval.

These results do not prove that physical time-scale separation should be abandoned. Rather, they support selective integration in which shared supervisory objectives coordinate PV and storage decisions while conventional inner control, protection, and fallback functions remain active. Fully joint control should therefore be regarded as a testable architectural alternative, not as an inherently superior replacement for hierarchical control.

A definitive assessment requires matched comparisons under identical models, operating profiles, constraints, control intervals, and computing platforms. Such comparisons should report energy and life-cycle performance together with training effort, online computation, constraint violations, closed-loop robustness, and hardware performance.

5.3. Future Directions

The literature suggests potential for unified continuous-action control frameworks that jointly optimize PV operating-point selection and battery dispatch under realistic constraints. Continuous-action RL may be a potential candidate due to its ability to optimize long-horizon objectives and operate directly over real-valued control actions. Progress toward deployable solutions will likely require: (i) constraint-aware policy optimization (e.g., safe layers or CMDP formulations); (ii) reward structures that reflect battery health, curtailment trade-offs, and system-level efficiency over time; (iii) improved state estimation and observability for variables such as SoC and degradation-related indicators; (iv) robustness to model mismatch and non-stationarity via uncertainty-aware or domain-randomized training; (v) improved sample efficiency through hybrid or model-informed learning architectures; and (vi) broader experimental validation and benchmarking beyond simulation.

This work motivates these directions by positioning PV–battery MPPT as a coupled sequential decision-making problem where control fidelity, safety, and long-term optimality must be addressed simultaneously.

In this sense, the key open challenge is not only to improve MPPT accuracy but also to bridge converter-level PV actuation and battery-aware energy management within a single practical learning framework.

6. Conclusions

This article presented a structured literature review on RL for integrated MPPT and battery management in PV systems. This review showed that the development from conventional perturbation-based MPPT methods toward intelligent and learning-based control has been driven by the increasing complexity, variability, and coupling of modern PV–battery energy systems. While classical MPPT methods remain attractive because of their simplicity, their limitations under fast irradiance changes, partial shading, and storage-coupled operation motivate the investigation of more adaptive control strategies.

The reviewed literature indicates that RL provides a useful framework for formulating control as a sequential decision-making problem and for considering long-term objectives beyond instantaneous power extraction. Continuous-action actor–critic methods are relevant to converter-based energy systems because they can operate directly over real-valued control variables and may provide smoother control actions than discretized approaches. However, their performance depends strongly on assumptions regarding state observability, reward formulation, model fidelity, constraint enforcement, and the training environment. Moreover, most existing RL-based studies remain concentrated at the supervisory or energy management layer, where PV generation is often treated as exogenous and MPPT is implemented outside the learning loop.

A key conclusion of this review is therefore that the main research gap is not simply the limited use of RL but also the limited coordination between converter-level PV operating-point control and battery-aware long-horizon optimization. In PV–battery systems, strict instantaneous MPPT is not always system-optimal, since PV operating-point decisions affect battery state of charge, charging stress, efficiency, curtailment, and long-term performance. This motivates explicit coordination between MPPT and battery management, whether implemented through a hierarchical layered architecture or a unified decision framework.

Finally, under specific assumptions and constraints, continuous-action RL may provide new ideas for integrated control, but its challenges in constraint handling, stability, and hardware deployment still need to be overcome. Future work should therefore investigate safe and constraint-aware learning formulations, stability assessment, improved state estimation, hybrid model-informed approaches, and systematic sim-to-real and experimental validation. Addressing these issues is necessary before such methods can be considered practical and deployable for PV–battery energy systems.