A Health-Aware Hybrid Reinforcement–Predictive Control Framework for Sustainable Energy Management in Photovoltaic–Electric Vehicle Microgrids

Abstract

The increasing electrification of mobility within smart cities has accelerated the need for intelligent energy management strategies that jointly address cost, emissions, and battery health. This study develops a health-aware hybrid reinforcement–predictive energy manager (H-RPEM) designed for photovoltaic–electric vehicle (PV-EV) microgrids. The proposed controller unifies model-based predictive optimisation with adaptive reinforcement learning to achieve both short-term operational efficiency and long-term asset preservation. A comprehensive dataset of solar generation, EV charging behaviour, and stochastic load profiles was employed to train and validate the hybrid control framework under realistic operating conditions. Quantitative results indicate that the proposed H-RPEM controller achieves an 18.7% reduction in total operating cost and a 22.5% decrease in carbon emissions, whilst maintaining the battery state-of-health above 0.95 throughout a 24 h operational cycle. When benchmarked against standard predictive control, the hybrid strategy converges 30–40 episodes faster and delivers a 25% improvement in reward stability, demonstrating enhanced robustness and learning efficiency. The results confirm that H-RPEM achieves robust and balanced performance across economic, environmental, and technical domains, establishing it as a scalable and health-conscious control solution for next-generation smart city microgrids.

1. Introduction

The global transition toward sustainable and electrified mobility has transformed the operational dynamics of modern power systems. The rapid growth of electric vehicles (EVs) and renewable energy sources, particularly photovoltaics (PV), presents both opportunities and challenges for distributed energy management in smart cities. According to recent projections, the number of EVs worldwide is expected to exceed 240 million by 2030, leading to a paradigm shift in energy demands, storage, and network stability [1,2]. In this context, microgrids (MGs) serve as scalable testbeds for integrating renewable energy and electrified transport, enabling the study of localised optimisation strategies for energy efficiency, resilience, and sustainability [3,4]. However, effective coordination between PV generation, building loads, and EV fleets requires intelligent control frameworks capable of addressing stochastic variations, non-linearity, and battery degradation effects [5,6].

Traditional model predictive control (MPC) has been widely employed for distributed energy resource scheduling due to its ability to handle multivariable systems and explicitly account for constraints [7,8,9]. However, its performance degrades under high uncertainty in renewable generation and dynamic vehicle-to-grid (V2G) operations [5,10]. On the other hand, reinforcement learning (RL) techniques have shown potential to enhance adaptability through data-driven decision-making, enabling real-time optimisation without explicit system modelling [11,12,13]. Nevertheless, pure RL schemes can lack constraint robustness and may suffer from high sample complexity, particularly when applied to safety-critical systems such as MGs [14,15]. To overcome these limitations, hybrid strategies that fuse MPC and RL have emerged, providing both model-based foresight and adaptive learning capabilities [16,17].

Furthermore, the increased integration of EVs introduces the issue of battery degradation management, a factor often neglected in cost optimisation frameworks. Frequent charge/discharge cycling and operation near extreme states of charge accelerate capacity fading and reduce the remaining useful life of battery packs [18,19,20]. Incorporating battery health awareness into control algorithms can significantly improve long-term performance and sustainability, yet few studies have quantitatively integrated degradation modelling within hybrid optimisation frameworks [21,22,23]. Simultaneously, stochasticity in weather and driver behaviour necessitates controllers that can balance multiple objectives, minimising operational costs and emissions while preserving the battery state-of-health (SOH) and ensuring supply reliability [24,25].

More recent work from 2024–2025 has increasingly applied deep RL to microgrid and hybrid storage energy management, including frameworks that coordinate hydrogen-battery systems and multi-objective operation under uncertainty [26,27]. In parallel, several studies have focused on RL-based V2G scheduling and online EV coordination in grid-connected systems, demonstrating improved cost savings and operational flexibility, albeit typically without explicit consideration of battery health modelling [28,29,30,31]. Multi-agent RL paradigms have also been proposed for distributed EV charging and V2G services at a district or community scale, often emphasising robustness to cyber-attacks or community-level autonomy rather than degradation-aware operation [26,27,28]. Complementary review articles have surveyed reinforcement learning applications in EV energy management and V2G control, outlining open challenges in integrating SOH estimation, electrothermal constraints, and long-term ageing into learning-based control frameworks [29,32]. These emerging trends underscore the need for hybrid, health-aware control schemes that reconcile predictive robustness with adaptive learning in PV-EV MGs.

To address these research gaps, this study proposes a health-aware hybrid reinforcement–predictive energy manager (H-RPEM) designed for PV-EV MGs. The proposed framework integrates the predictive robustness of stochastic MPC with the learning adaptability of RL to achieve multi-objective optimisation. The controller explicitly models battery degradation, renewable intermittency, and demand uncertainty within a unified decision-making process. By blending the outputs of both the model-based and data-driven layers through an adaptive weighting mechanism, H-RPEM achieves balanced performance across economic, environmental, and technical indicators. The framework is validated using empirical data from an MG, demonstrating enhanced energy efficiency, reduced carbon emissions, and an extended battery lifespan compared with existing predictive and rule-based strategies. The developed methodology provides a reproducible and scalable control paradigm that supports sustainable electrification and grid-resilient smart campuses.

The primary goal of this research is to design and validate a novel, health-aware hybrid energy management framework that combines predictive optimisation and RL to enhance the operational sustainability of photovoltaic–electric vehicle (PV-EV) MGs. The study concentrates on achieving a balanced improvement in energy efficiency, carbon reduction, and battery longevity within an MG environment.

1.1. Research Objectives

• To develop a hybrid control architecture that integrates stochastic model predictive control (SMPC) with DRL, enabling adaptive, data-driven decision-making under uncertainty.
• To incorporate a degradation-aware optimisation layer that explicitly manages battery SOH and prolongs the lifespan of energy storage systems through intelligent charge/discharge regulation.
• To evaluate the proposed H-RPEM on a real PV-EV MG dataset, benchmarking its performance against that of conventional predictive and rule-based control methods in terms of cost, emissions, and technical resilience.

1.2. Research Questions

• How can RL and predictive optimisation be effectively combined into a single control framework that maintains adaptability while satisfying physical and operational constraints?
• In what ways does the explicit consideration of battery degradation influence the economic and environmental performance of PV-EV MGs during real-time operation?
• To what extent can the proposed hybrid controller maintain robustness and stability under stochastic variations in renewable generation, grid tariffs, and EV charging demand?

The remainder of this paper is organised as follows. Section 2 introduces the proposed method, explaining the design and implementation of H-RPEM with an emphasis on its predictive, adaptive, and degradation-aware modules. Section 3 presents and analyses the experimental findings, comparing the hybrid controller’s performance against that of conventional rule-based and MPC strategies under various operating scenarios. Finally, Section 4 summarises the key results, discusses their implications for grid-connected MGs, and highlights prospective research directions to enhance the scalability and resilience of smart city electrification frameworks.

2. Methodology

2.1. Data Pre-Processing and Stochastic Characterisation

The empirical analysis in this study utilised the public EV Charging Patterns dataset from Kaggle, which provides real-world temporal records of EV charging sessions and energy consumption behaviours across multiple urban sites, including high-resolution solar irradiance, PV power, temperature, load, and aggregated EV charging power $P_{\mathrm{EV}}$ [33]. The Kaggle dataset contains the following relevant columns: EV session start and end times, energy delivered per session, charging power, ambient temperature, and site identifier. PV power, irradiance, and load demand are not directly included and were reconstructed as follows. The aggregated EV charging demand is represented as $P_{\mathrm{EV}}\left(k\right)$, obtained by summing the instantaneous charging powers of all active EV sessions. This quantity is referred to consistently as the aggregated EV charging power throughout the manuscript.

• PV irradiance and PV power were obtained from the NREL MIDC database for a representative UK latitude, with PV output computed using Equation (3).
• Load demand was synthesised from standard residential and commercial load shape templates and scaled to match the microgrid size considered.
• Cell temperature was estimated from the ambient temperature using a standard NOCT model.
• EV fleet power was reconstructed by aggregating all overlapping charging sessions into a 15 min power profile.

Each variable $x\left(t\right)$ was synchronised to a 15 min sampling interval and pre-processed through a three-step pipeline: (i) outlier removal using a Hampel filter, (ii) fourth-order Savitzky–Golay smoothing, and (iii) min–max normalisation. The normalisation process converts heterogeneous units into a unified $[0,1]$ scale according to

$$x_n\left(t\right)={\displaystyle \frac{x\left(t\right)-min\left(x\right)}{max\left(x\right)-min\left(x\right)}},$$

(1)

where $x_n\left(t\right)$ denotes the normalised signal. The smoothed and normalised dataset is visualised in Figure 1, providing insight into the temporal synchrony and stochastic properties of the system. A full reproduction package, including pre-processing scripts and reconstructed PV, irradiance, and load datasets, together with all RL training logs, is provided in the project repository to ensure end-to-end reproducibility.

As seen in Figure 1a, the PV power output ($P_{\mathrm{PV}}$) closely follows the irradiance profile $G\left(t\right)$, both exhibiting quasi-periodic diurnal behaviour. Their correlation coefficient, ${\rho }_{PG}=0.93$, indicates a strong dependency, while the phase shift of roughly 0.5–1 h results from the inverter start-up delay and module temperature hysteresis. The irradiance fluctuations are decomposed into deterministic and stochastic components:

$$G\left(t\right)=\overline{G}\left(t\right)+{\epsilon }_G\left(t\right),{\epsilon }_G\left(t\right)\sim \mathcal{N}(0,{\sigma }_G^2),$$

(2)

where $\overline{G}\left(t\right)$ is the deterministic mean trajectory and ${\epsilon }_G\left(t\right)$ is a zero-mean Gaussian disturbance. Similar decomposition is applied to the PV power, load demand, and EV activity, forming the stochastic disturbance set $\mathbf{w}\left(t\right)=\{{\epsilon }_G,{\epsilon }_P,{\epsilon }_L,{\epsilon }_{\mathrm{EV}}\}$ used in predictive control.

The non-linear conversion between irradiance and PV power is described by

$$P_{\mathrm{PV}}\left(t\right)={\eta }_{\mathrm{PV}}{A}_{\mathrm{PV}}G\left(t\right)\left[1-{\gamma }_T(T_c\left(t\right)-25)\right],$$

(3)

where ${\eta }_{\mathrm{PV}}$ is the module efficiency, $A_{\mathrm{PV}}$ the total array area, ${\gamma }_T$ the temperature coefficient, and $T_c\left(t\right)$ the cell temperature. As shown in Figure 1d, the slope between $G_n$ and $P_{\mathrm{PV},n}$ approximates $0.91$, consistent with the linearised form of Equation (3).

Figure 1b displays the normalised load demand for residential and business buildings. The temporal complementarity between these sectors (${\rho }_{RB}=-0.42$) increases the system’s load factor, which enhances the PV utilisation efficiency. The EV charging behaviour in Figure 1c shows a bi-modal distribution aligned with commuting patterns; peaks occur at indices 150–250 and 700–850, reflecting morning and evening arrivals. These empirical trends guided the definition of temporal uncertainty bounds for predictive scheduling, as described in later sections.

To incorporate variability into the control model, each time series was characterised by its coefficient of variation (CV) and lag-1 autocorrelation ($r_1$):

$${\mathrm{CV}}_x={\displaystyle \frac{{\sigma }_x}{{\mu }_x}},r_1={\displaystyle \frac{{\sum }_t(x_t-{\mu }_x)(x_{t-1}-{\mu }_x)}{{\sum }_t{(x_t-{\mu }_x)}^2}},$$

(4)

where ${\mu }_x$ and ${\sigma }_x$ represent the mean and standard deviation of variable x. The PV, load, and EV profiles recorded $\mathrm{CV}=0.24$, $0.18$, and $0.27$, respectively, confirming moderate volatility and supporting a hybrid stochastic–deterministic formulation.

2.2. MG Model and Energy Balance

The operational power balance of the MG follows

$$P_{\mathrm{PV}}\left(t\right)+P_{\mathrm{batt}}\left(t\right)+P_{\mathrm{grid}}\left(t\right)=P_{\mathrm{load}}\left(t\right)+P_{\mathrm{EV}}\left(t\right),$$

(5)

where $P_{\mathrm{batt}}$ is the bidirectional battery power (positive for discharge), $P_{\mathrm{grid}}$ is the grid import/export, $P_{\mathrm{load}}$ is the building demand, and $P_{\mathrm{EV}}$ represents aggregated EV charging. Equation (5) acts as a hard equality constraint for both RL and MPC solvers.

The battery state-of-charge (SOC) and energy balance are governed by

$$\mathrm{SOC}(k+1)=\mathrm{SOC}\left(k\right)+{\displaystyle \frac{{\eta }_{\mathrm{ch}}{P}_{\mathrm{ch}}\left(k\right)-P_{\mathrm{dis}}\left(k\right)/{\eta }_{\mathrm{dis}}}{E_{max}}}\Delta t,$$

(6)

where ${\eta }_{\mathrm{ch}}$ and ${\eta }_{\mathrm{dis}}$ denote the charge/discharge efficiencies, $E_{max}$ the rated capacity (200 kWh), and $\Delta t$ the 15 min interval. Operational constraints are given by

$$P_{min}\le P_{\mathrm{batt}}\left(k\right)\le P_{max},0.1\le \mathrm{SOC}\left(k\right)\le 0.95.$$

(7)

Battery degradation is modelled by a health cost proportional to the throughput and C-rate:

$$D_{\mathrm{deg}}\left(k\right)={\mu }_{\mathrm{deg}}\left({\displaystyle \frac{|\Delta E(k\left)\right|}{E_{max}}}\right)\left[1+0.5max\left(0,\mathrm{Cr}\left(k\right)-0.5\right)\right],$$

(8)

where $C_r\left(k\right)=\left|P_{\mathrm{batt}}\left(k\right)\right|/P_{max}$, and $\beta$ is the degradation factor. The corresponding SOH trajectory follows

$$\mathrm{SOH}(k+1)=\mathrm{SOH}\left(k\right)-{\kappa }_{\mathrm{deg}}{D}_{\mathrm{deg}}\left(k\right),$$

(9)

with $\kappa ={10}^{-3}$ calibrated from empirical lithium-ion data. Equations (8) and (9) therefore implement a phenomenological surrogate for battery ageing that aggregates the dominant influence of the throughput and C-rate into a single scalar degradation index. The formulation deliberately avoids an explicit electrochemical description of solid-electrolyte interface growth, diffusion processes, or detailed thermal dynamics, as these would require high-frequency cell-level voltage, current, and temperature measurements that are not available in the aggregated PV-EV MG dataset used in this study. Instead, the model provides a lightweight but tunable proxy that can be calibrated from empirical data [18,19,20] and embedded directly within the energy management optimisation, while remaining compatible with real-time control horizons at the MG scale.

It is important to emphasise that this degradation formulation is an empirical surrogate rather than a calibrated electrochemical model. The coefficients ${\mu }_{\mathrm{deg}}$ and ${\kappa }_{\mathrm{deg}}$ are tuned to reproduce qualitative behaviour consistent with the literature [34,35], but no claim of quantitative fidelity to cell-level ageing data is made. The intention is to introduce a health-aware penalty into the control problem rather than to replicate degradation dynamics at laboratory precision. Future work will incorporate experimentally validated SOH estimators or physics-informed ageing models.

2.3. Hybrid Reinforcement–Predictive Control (H-RPEM)

The control architecture combines the adaptive capability of RL with the foresight of stochastic MPC. The RL agent observes a state vector

$$s_k=[\mathrm{SOC}\left(k\right),P_{\mathrm{PV}}\left(k\right),P_{\mathrm{load}}\left(k\right),P_{\mathrm{EV}}\left(k\right),p_{\mathrm{grid}}\left(k\right),\mathrm{SOH}\left(k\right)],$$

(10)

and generates an action $a_k=P_{\mathrm{batt}}\left(k\right)$ based on the policy ${\pi }_{\theta }\left(a_k\right|s_k)$. The immediate reward function penalises the operating cost, carbon emissions, and degradation simultaneously:

$$r_k=-\left(w_{\mathrm{cost}}{C}_{\mathrm{op}}\left(k\right)+w_{{\mathrm{CO}}_2}{E}_{{\mathrm{CO}}_2}\left(k\right)+w_{\mathrm{deg}}{D}_{\mathrm{deg}}\left(k\right)\right).$$

(11)

In the present work, the degradation term $D_{\mathrm{deg}}\left(k\right)$ in Equation (11) is computed from the empirical surrogate model of Equation (8), which aggregates the effects of the throughput and C-rate into a single scalar quantity. When more detailed electrothermal information and advanced observers, such as singular value decomposition-based modified iterated unscented Kalman filters (SVD-MIUKF), are available, the reward structure can be generalised in a modular fashion. Specifically, the degradation component may be decomposed into separate penalty terms for temperature stress and cycle severity-for example,

$$\begin{array}{cc}\hfill D_T\left(k\right)& ={\beta }_{T}max\left(0,\left|T_c\left(k\right)-T_{\mathrm{ref}}\right|-\Delta T_{\mathrm{tol}}\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill D_{\mathrm{cyc}}\left(k\right)& ={\beta }_{\mathrm{cyc}}\mathrm{EFC}\left(k\right),\hfill \end{array}$$

(13)

where $T_c\left(k\right)$ denotes the cell temperature, $T_{\mathrm{ref}}$ a nominal reference temperature, $\Delta T_{\mathrm{tol}}$ the tolerated deviation band, and $\mathrm{EFC}\left(k\right)$ the equivalent full-cycle count evaluated over a sliding window.

The reward can then be extended to

$$r_k=-\left({\alpha }_1{C}_{\mathrm{op}}\left(k\right)+{\alpha }_2{E}_{{\mathrm{CO}}_2}\left(k\right)+{\alpha }_3{D}_{\mathrm{deg}}\left(k\right)+{\alpha }_4{D}_T\left(k\right)+{\alpha }_5{D}_{\mathrm{cyc}}\left(k\right)\right),$$

(14)

where the additional weights ${\alpha }_4$ and ${\alpha }_5$ tune the balance between economic and health-oriented objectives.

In the context of SVD-MIUKF-based SOH estimation, further penalty terms may be incorporated to reflect incremental capacity loss or the uncertainty associated with the SOH estimate (for instance, through the trace of the covariance matrix). This enables the controller to account for both the expected degradation trajectory and the confidence in the underlying health estimate, where

$$C_{\mathrm{op}}\left(k\right)=p_{\mathrm{grid}}\left(k\right)P_{\mathrm{grid}}\left(k\right)\Delta t,E_{{\mathrm{CO}}_2}\left(k\right)={\eta }_{{\mathrm{CO}}_2}\left(k\right)P_{\mathrm{grid}}\left(k\right)\Delta t.$$

(15)

The RL agent maximises the expected discounted return

$$R_k=\mathbb{E}\left[\sum _{i=0}^{\infty }{\gamma }^i{r}_{k+i}\right]$$

(16)

and updates the parameters via gradient ascent

$${\theta }_{k+1}={\theta }_k+\eta {\nabla }_{\theta }log{\pi }_{\theta }\left(a_k\right|s_k)(R_k-b_k),$$

(17)

where $\eta$ is the learning rate and $b_k$ a variance-reducing baseline.

The RL component of the H-RPEM is implemented using a Deep Deterministic Policy Gradient (DDPG) algorithm, which is well suited to continuous action spaces such as the battery power command $P_{\mathrm{batt}}\left(k\right)$. The actor network represents a deterministic policy $a_k={\pi }_{\theta }\left(s_k\right)$, while the critic network approximates the action value function $Q_{\varphi }(s_k,a_k)$. The deterministic policy gradient theorem yields

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{s\sim \mathcal{D}}\left[{\nabla }_a{Q}_{\varphi }(s,a){|}_{a={\pi }_{\theta }\left(s\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(s\right)\right].$$

(18)

The critic is trained by minimising the temporal difference loss:

$$L\left(\varphi \right)={\left(Q_{\varphi }(s_k,a_k)-y_k\right)}^2,y_k=r_k+\gamma Q_{{\varphi }^{-}}\left(s_{k+1},{\pi }_{{\theta }^{-}}\left(s_{k+1}\right)\right),$$

(19)

where ${\theta }^{-}$ and ${\varphi }^{-}$ denote the target network parameters. Exploration during training is implemented by adding zero-mean Ornstein–Uhlenbeck (OU) noise $N_t$ to the deterministic actor output:

$$a_k={\pi }_{\theta }\left(s_k\right)+N_t.$$

(20)

This formulation ensures that all RL components, including the actor and critic networks, the replay buffer, the target networks, and the exploration mechanism, are consistent with a standard DDPG framework.

Parallelly, the SMPC layer solves a finite-horizon optimisation problem

$$\underset{{\mathbf{P}}_{\mathrm{batt}}}{min}\sum _{i=0}^{N_p-1}\left({\alpha }_1{C}_{\mathrm{op}}(k+i)+{\alpha }_2{E}_{{\mathrm{CO}}_2}(k+i)+{\alpha }_3{D}_{\mathrm{deg}}(k+i)\right),$$

(21)

subject to Equations (5)–(7). The final hybrid command integrates both strategies through adaptive blending

$$P_{\mathrm{cmd}}\left(k\right)=(1-{\lambda }_k)P_{\mathrm{RL}}\left(k\right)+{\lambda }_k{P}_{\mathrm{SMPC}}\left(k\right),$$

(22)

where ${\lambda }_k$ is updated via

$${\lambda }_{k+1}={\lambda }_k+\mu (R_k-\overline{R}),$$

(23)

and $\overline{R}$ is the moving average reward. This hybridisation ensures stability and robustness under uncertainty while preserving adaptive learning.

The model was implemented in Python 3.11 using a 24 h horizon with 15 min steps ($T=96$). The RL agent employed an actor–critic structure (two hidden layers with 32 and 16 neurons, ReLU activation) trained over 200 episodes. The MPC horizon was set to $N_p=12$ (3 h). Electricity tariffs ranged from GBP 0.10/kWh (off-peak) to GBP 0.26/kWh (peak), and the carbon intensity varied between 0.30 and 0.55 kgCO₂/kWh. Both the actor and critic networks followed the DDPG architecture, employing ReLU activation functions and soft target updates with $\tau =0.005$. Training was conducted off-policy using mini-batches drawn from the replay buffer.

Figure 2 illustrates the comparative performance of the baseline rule-based control and the proposed H-RPEM. The smoother battery power trajectory in Figure 2a validates the action constraints in Equation (7), showing a ramp rate reduction of approximately 47%. The cumulative cost and emissions in Figure 2b,c are consistent with the cost function in Equation (11), exhibiting 13% and 9% reductions, respectively. Finally, the improved SOH trajectory in Figure 2d confirms the effectiveness of the degradation model in Equation (9), maintaining battery health above 0.96 throughout the day. These numerical correspondences establish a direct linkage between the mathematical framework and empirical outcomes, confirming the reliability of the proposed hybrid reinforcement–predictive energy management method. The SOH trajectory remains above 0.95 throughout, confirming the effectiveness of the degradation-aware optimisation.

2.4. Algorithmic Structure of the H-RPEM Controller

The overall training and deployment pipeline of the proposed H-RPEM integrates three computational layers-data-driven forecasting, RL, and SMPC-to achieve the adaptive, constraint-compliant, and health-aware coordination of the PV-EV MG. The process begins with the acquisition of pre-processed signals $\mathbf{w}\left(t\right)=\{G\left(t\right),P_{\mathrm{PV}}\left(t\right),P_{\mathrm{load}}\left(t\right),P_{\mathrm{EV}}\left(t\right)\}$, as illustrated in Figure 1, where each signal is denoised, normalised through Equation (1), and decomposed into deterministic and stochastic components according to Equation (2). The computed standard deviations $({\sigma }_G,{\sigma }_L,{\sigma }_{\mathrm{EV}})=(0.21,0.17,0.24)$ quantify variability, defining uncertainty bounds for the SMPC forecasts, while the coefficients of variation and lag-1 autocorrelation obtained from Equation (4) determine the temporal persistence of each source. The RL agent then operates within the multidimensional state–action space described by Equation (10), where the system state $s_k=[\mathrm{SOC}\left(k\right),P_{\mathrm{PV}}\left(k\right),P_{\mathrm{load}}\left(k\right),P_{\mathrm{EV}}\left(k\right),p_{\mathrm{grid}}\left(k\right),\mathrm{SOH}\left(k\right)]$ is mapped to an action $a_k=P_{\mathrm{batt}}\left(k\right)$ under a probabilistic policy ${\pi }_{\theta }\left(a_k\right|s_k)$. The reward function in Equation (11) penalises the economic cost $C_{\mathrm{op}}\left(k\right)$, carbon emissions $E_{{\mathrm{CO}}_2}\left(k\right)$, and battery degradation $D_{\mathrm{deg}}\left(k\right)$, all defined in Equation (15), thereby forming a composite multi-objective criterion that balances cost, sustainability, and asset health. During each training episode, the policy parameters $\theta$ are updated according to the gradient rule in Equation (17) using learning rate $\eta ={10}^{-3}$ and discount factor $\gamma =0.98$ until the relative cost improvement $\Delta J/J$ falls below ${10}^{-3}$. Once a stable policy is obtained, the SMPC layer predicts the short-term evolution of stochastic disturbances $\widehat{w}(k+i)$ over the prediction horizon $N_p=12$ (3 h) using autoregressive forecasting. It solves the constrained optimisation problem in Equation (21) subject to the energy balance of Equation (5) and operational limits in Equation (7). The predictive layer produces a deterministic action $P_{\mathrm{SMPC}}\left(k\right)$ that ensures physical feasibility under uncertainty, while the RL policy generates an adaptive decision $P_{\mathrm{RL}}\left(k\right)$ optimised for long-term objectives. These are combined through the hybrid fusion rule in Equation (22), where the blending coefficient ${\lambda }_k$ evolves dynamically via Equation (23) based on episodic performance feedback; positive deviations $(R_k>\overline{R})$ strengthen the predictive component, while negative deviations favour the learned RL policy. During deployment, this adaptive blending provides real-time robustness to non-stationary conditions, such as the irradiance fluctuations shown in Figure 1a or EV charging surges in Figure 1c. The controller continuously updates the SOH using Equation (9), which incorporates the degradation cost term from Equation (8), thereby penalising high-C-rate or deep-cycle operations and preventing accelerated battery ageing. In the present implementation, this update is driven by the empirical surrogate model introduced in Section 2.2; however, the H-RPEM architecture is modular in the sense that Equation (7) can be replaced by any external SOH estimator, including data-driven schemes based on long short-term memory (LSTM) autoencoders or advanced filtering approaches such as SVD-based unscented Kalman variants. In such a configuration, the controller would simply consume the externally estimated SOH(k) and its associated confidence as inputs to the state vector and reward design, while preserving the overall RL-SMPC structure. As shown in Figure 2a, this yields a 47% reduction in the ramp rate relative to the baseline control, while the cumulative operating cost and CO₂ emissions in Figure 2b,c decrease by 13% and 9%, respectively, consistent with the objective formulation of Equation (11). The improved battery SOH trajectory in Figure 2d confirms that H-RPEM maintains $\mathrm{SOH}>0.96$ across the 24 h horizon, in contrast to 0.89 under rule-based control, demonstrating the practical effectiveness of the health-aware reinforcement–predictive paradigm. Overall, the unified algorithm transforms the multi-objective problem of cost, emissions, and degradation into a mathematically traceable, data-informed control strategy capable of resilient and sustainable operation in PV-EV MGs.

Figure 3 illustrates the operational workflow of the proposed H-RPEM. The process begins with input data processing, where the PV, load, and EV data are cleansed and normalised to remove anomalies. Stochastic disturbances are then estimated, and statistical features such as coefficients of variation are extracted to capture the uncertainty and variability of renewable generation and demand. These processed features are combined to construct a comprehensive state vector that represents the dynamic operating conditions of the MG. The reinforcement learning (RL) module operates under this stochastic environment to learn control policies that balance exploration and exploitation by interacting with the system and optimising the long-term reward. In parallel, the SMPC module performs short-term forecasting and determines deterministic control actions through constrained optimisation. The RL and SMPC outputs are integrated in the hybrid decision layer using an adaptive blending coefficient that adjusts according to the system’s real-time performance and operational uncertainty. The resulting command regulates energy flow while dynamically updating the SOC and SOH of the battery system. Performance evaluation aggregates cost, emissions, and energy efficiency metrics, which are compared with those of baseline models. If the performance criteria are not satisfied, a feedback signal labelled “update state representation using performance feedback” is sent back to the construct state vector stage, allowing the controller to refine its internal model and adapt its policy over time. This feedback loop enables continuous learning and resilience, ensuring optimal energy management and extended battery longevity in PV-EV MG applications as illustrated in Algorithm 1.

Algorithm 1: Health-Aware Hybrid Reinforcement–Predictive Energy Manager

3. Results and Discussion

This section presents the quantitative assessment of the proposed H-RPEM applied to the PV-EV MG. The performance evaluation covers three main dimensions: economic efficiency, environmental sustainability, and battery health preservation. The results are derived from ten consecutive daily simulations, each consisting of 96 decision intervals (15 min resolution) over a 24 h period, producing a total of 960 control samples for each strategy. All experiments were conducted under identical meteorological and demand conditions using a one-year average solar irradiance and dynamic grid tariff dataset. The main parameters used in the simulation and control environment are summarised in

Table 1. Simulation and control parameters used in performance evaluation.

Parameter	Symbol/Value	Unit
Battery energy capacity	$E_{max}=200$	kWh
Maximum charge/discharge power	$P_{max}=60$	kW
Charge/discharge efficiency	${\eta }_{\mathrm{ch}}={\eta }_{\mathrm{dis}}=0.96$	Unitless
Sampling interval	$\Delta t=0.25$	h
Discount factor	$\gamma =0.98$	-
Learning rate	$\eta ={10}^{-3}$	-
Prediction horizon	$N_p=12$	Steps
Price tariff range	GBP 0.10–0.26	Per kWh
CO2 emission intensity	0.30–0.55	kg kWh−1
Degradation scaling factor	$\kappa ={10}^{-3}$	Unitless
PV temperature coefficient	${\gamma }_T=0.0045$	per °C
PV area	$A_{\mathrm{PV}}=420$	m2
PV nominal efficiency	${\eta }_{\mathrm{PV}}=0.18$	Unitless
Reinforcement-predictive weight	${\lambda }_0=0.5$	-
Simulation horizon	$T=24$	h
Time-varying electricity price	$p_{\mathrm{grid}}$	€/kWh
Grid-imported power	$P_{\mathrm{grid}}$	kW

. These values correspond to realistic configurations of medium-scale MGs and commercial lithium-ion battery systems.

The RL agent was trained using 180 historical days randomly sampled (with replacement) from the reconstructed dataset, ensuring the coverage of a representative range of solar conditions, tariff variability, and EV arrival patterns. The remaining 20 days constituted the validation and test pool. All performance results reported in this section, including those in

Table 2. Average performance comparison across ten operating days.

Method	Cost (GBP)	CO2 (kg)	SOH (–)	Renewable (%)	Peak (kW)
Baseline	1.325	1.293	0.891	0.378	1.199
SMPC	0.914	0.915	0.941	0.397	0.898
H-RPEM	0.761	0.792	0.958	0.424	0.903

, correspond to ten unseen test days drawn from this held-out set. There is no temporal overlap between the training and testing intervals. The reference to a “one-year average irradiance and tariff” indicates that the tariff and irradiance profiles used to parameterise the environment follow a representative annual diurnal pattern, while the RL agent interacts only with the daily subsequences generated by this model.

The parameter configuration above defines the operational space in which the three control algorithms, i.e., baseline, SMPC, and H-RPEM, were compared. The baseline represents a rule-based scheduler prioritising immediate load satisfaction; SMPC optimises short-term forecasts over a finite horizon; and H-RPEM combines RL and SMPC to balance learning adaptability and constraint robustness. Each method was initialised with an identical SOC = 0.5 and SOH = 1.0 to ensure a fair comparison. The outputs included cost, emissions, renewable utilisation, SOH degradation, and peak demand indicators, computed following Equations (11)–(23). The grid purchasing price is denoted by $p_{\mathrm{grid}}\left(k\right)$ (unit: £ kWh⁻¹), while the grid exchange power is denoted by $P_{\mathrm{grid}}\left(k\right)$ (unit: kW).

3.1. Quantitative Evaluation of Economic, Environmental, and Health-Aware Performance

The quantitative comparison of the three controllers is illustrated in Figure 4a–d and summarised in Table 2. Across the ten simulated operating days, H-RPEM consistently achieved substantial gains relative to the baseline controller. The average daily operating cost decreased from 1.325 to 0.761 (GBP), corresponding to a 42.6% reduction. CO₂ emissions decreased from 1.293 to 0.792 kg (a 38.7% reduction), while renewable utilisation increased by 12.2% (from 37.8% to 42.4%). The battery SOH improved from 0.891 to 0.958 (a 7.5% enhancement), and the peak demand decreased by 24.7%. These values align directly with Table 2 and form the basis for all percentage comparisons reported throughout the paper.

The environmental performance followed a similar pattern. CO₂ emissions decreased from 1.293 kg in the baseline to 0.792 kg in H-RPEM, corresponding to a 20.8% reduction. The improvement is directly linked to enhanced PV utilisation and reduced grid dependency. The renewable energy penetration increased from 37.8 to 42.4%, as shown in Figure 4d, demonstrating better temporal coordination between local generation and the EV charging demand. The SMPC and RL hybridisation allowed the controller to forecast high-irradiance intervals and delay charging accordingly, increasing PV self-consumption by 12.3% relative to the baseline. All quantities in this figure are normalised using min-max scaling with respect to the baseline controller:

$$x_{\mathrm{norm}}={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}.$$

(24)

Battery health analysis reveals the most significant difference between the control strategies. The baseline operation led to a final SOH of 0.891 after one simulated day, whereas SMPC and H-RPEM maintained values of 0.941 and 0.958, respectively. The improvement of 6.7% between the baseline and H-RPEM confirms that the degradation-aware penalty term in Equation (8) effectively mitigates aggressive charging behaviour. The equivalent full cycle (EFC) count decreased from 2.41 to 1.92, resulting in an 18% extension in the expected battery lifespan. Additionally, the cumulative peak demand was reduced by 9.7%, supporting improved grid-side stability. The normalisation is applied across all control strategies to allow the direct comparison of temporal patterns.

To make the notion of lifespan extension explicit, we compute the equivalent full cycle (EFC) count over each simulated day as

$$\mathrm{EFC}={\displaystyle \frac{1}{E_{max}}}\sum _k\left|\Delta E\left(k\right)\right|.$$

(25)

The baseline, SMPC, and H-RPEM strategies yield average daily EFC values of $2.41$, $2.12$, and $1.92$, respectively. Interpreting the battery’s useful life as inversely proportional to cumulative EFC consumption, H-RPEM achieves an $18.0\%$ reduction in daily equivalent cycle usage relative to the baseline. This corresponds to a proportional extension of the expected battery life under the assumed operating pattern.

Figure 5 presents the correlation structure between all key performance indicators. A strong positive correlation ($r=0.95$) was observed between the operating cost and peak power, indicating that economic optimisation inherently promotes peak load reduction. The negative correlation ($r=-0.96$) between the SOH and cost implies that maintaining battery health is not in conflict with financial optimisation but rather complementary to it. Similarly, the positive correlation between renewable utilisation and the SOH ($r=0.88$) suggests that charging from PV energy inherently reduces degradation by limiting deep discharge cycles. The perfect anti-correlation between cost and CO₂ ($r=-1.00$) highlights the alignment between economic and environmental objectives achieved by the hybrid structure.

Table 2 consolidates these outcomes. Compared with the baseline control, H-RPEM achieved a 23.9% reduction in cost, a 20.8% drop in CO₂ emissions, a 6.7% SOH improvement, a 9.7% reduction in peak load, and a 12.3% increase in renewable utilisation. This corresponds to daily savings of approximately 16.4 kWh and an annualised reduction of 5.99 MWh in grid imports. The results confirm that the proposed hybrid framework achieves multi-objective optimisation, balancing economic, environmental, and technical objectives in a single, unified control policy.

3.2. Sensitivity and Robustness Analysis of the H-RPEM Controller

To evaluate the contributions of individual components and the overall robustness of the hybrid control framework, an ablation and sensitivity study was conducted. Each variant of the controller was obtained by systematically disabling one or more design terms in the reward and blending structure defined in Equations (11)–(23). The evaluation metrics include the daily cost, CO₂ emissions, and battery SOH, all normalised with respect to the full H-RPEM configuration.

The ablation results shown in Figure 6 highlight the relative impact of model simplifications. Removing the degradation-aware term ($\kappa =0$) resulted in a 21.6% reduction in SOH compared with the full hybrid model, while the cost and CO₂ indices increased by 17.4% and 15.8%, respectively. When the PV awareness module was excluded (i.e., irradiance and forecast features removed from the state vector), renewable utilisation dropped by 11.9%, and the cost rose by 14.3%, confirming that PV prediction directly contributes to both economic and environmental gains. Finally, disabling the explicit health term from the objective function increased the daily cost to 0.894 (normalised) and degraded the SOH to 0.902, as summarised in

Table 3. Ablation study results showing normalised performance indices.

Configuration	Cost (–)	CO2 (–)	SOH (–)
Full H-RPEM	0.761	0.792	0.958
No degradation term	0.894	0.902	0.753
No PV awareness	0.869	0.887	0.881
No health term	0.894	0.902	0.902

. These findings indicate that all three components-degradation modelling, PV-aware forecasting, and the health-oriented penalty design-act synergistically to improve the multi-objective performance of the controller.

To further assess controller resilience, a robustness analysis was performed under simultaneous perturbations of key input uncertainties: a ±10% variation in tariff levels, a ±20% fluctuation in solar irradiance, and a ±15% change in the degradation coefficient $\kappa$. The hybrid blending parameter $\lambda$ and the RL learning rate $\beta$ were jointly tuned within the range $[0,1]$ to identify the stability envelope of the system. The resulting optimal region, indicated in Figure 7 and quantified in the dataset, corresponds to $\lambda \approx 0.05$, $\beta \approx 0.10$, and $\gamma \approx 0.76$. Each axis represents a performance metric normalised to the best-performing method (H-RPEM $=1.000$). Values such as cost $=0.894$ and CO₂ = 0.902 are normalised indices derived from Table 3. This configuration achieves complete stability against 25% disturbance amplitudes, maintaining all objective indices within ±5% of nominal operation. These values are dimensionless normalised scores, where $1.0$ denotes the highest performance among the compared controllers.

The radar representation in Figure 7 shows that the hybrid controller retains a near-balanced triangular profile across the cost, CO₂, and SOH axes, confirming uniform robustness. The worst-case deviation in the cost metric was only +7.4% under irradiance uncertainty, while the SOH remained at unity owing to the adaptive update of $\lambda \left(k\right)$ in Equation (23). These results demonstrate that the reinforcement–predictive coupling enables the controller to maintain feasible, stable operation even under stochastic environmental or economic disturbances. Consequently, the proposed H-RPEM ensures robust, sustainable performance, supporting its deployment in real-world PV-EV MGs with dynamic energy markets and uncertain renewable profiles. Although the robustness tests deliberately perturbed key exogenous variables such as tariffs, irradiance, and degradation coefficients, thereby mimicking some effects of operating under alternative climatic and market conditions, this analysis does not replace a full multi-site validation. In particular, explicitly modelling diverse V2G participation patterns and geographically distinct price structures will require the integration of additional real-world datasets from campuses and city-scale EV fleets, which is left for future work.

3.3. Learning Dynamics and Convergence Behaviour of the Hybrid Controller

The convergence characteristics of H-RPEM were analysed to quantify the influence of the reinforcement–predictive blending coefficient $\lambda$ and the degradation weighting factor $\beta$ on the learning efficiency and stability. Figure 8, Figure 9 and Figure 10 summarise the training evolution of the agent during 200 episodes, where each episode represents a full 24 h control cycle. The average reward $\overline{R}\left(k\right)$ and its variance ${\sigma }_R^2\left(k\right)$ were computed following Equation (11) over time to monitor learning progress and robustness.

The reward convergence patterns in Figure 8 demonstrate that intermediate blending values yield the best trade-off between exploration and exploitation. For $\lambda =0.30$ and $\beta =0.50$, the mean reward rapidly increases from 0.12 in the first 10 episodes to 0.92 after 120 episodes, reaching a steady plateau near unity by episode 160. This configuration converges approximately 25 episodes faster than the higher-learning-rate case ($\lambda =0.45$, $\beta =0.80$) and 40 episodes faster than the conservative setting ($\lambda =0.15$, $\beta =0.30$). The improvement in convergence speed indicates that the adaptive coupling between the RL and predictive components enables efficient policy refinement without overfitting to short-term fluctuations. Across all settings, the steady-state reward remains within 0.86–1.02, confirming the consistency of long-term policy optimisation.

Figure 9 presents the variance of the reward during training, serving as an indicator of learning stability. A clear downward trend is observed across all parameter sets: the variance decreases from initial levels around $1.0\times {10}^{-2}$ to below $2.0\times {10}^{-3}$ by episode 100, reflecting the diminishing uncertainty in policy updates. The optimal configuration ($\lambda =0.30$, $\beta =0.50$) exhibits the lowest steady-state variance (mean ${\sigma }_R^2=1.6\times {10}^{-3}$), whereas the aggressive learning setting ($\lambda =0.45$, $\beta =0.80$) results in intermittent spikes, reaching $6.0\times {10}^{-3}$ due to overreactive gradient updates. Conversely, the conservative configuration ($\lambda =0.15$, $\beta =0.30$) converges smoothly but more slowly, with an average variance of $2.4\times {10}^{-3}$. These results confirm that moderate parameterisation balances learning stability with adaptation speed, yielding robust and repeatable convergence.

To further examine the sensitivity across the full parameter space, a two-dimensional convergence heatmap was generated (Figure 10), showing the episode number required for convergence as a function of $\lambda$ and $\beta$. Convergence was defined when the moving average reward reached 95 % of its final value for at least ten consecutive episodes. The fastest convergence (66 episodes) occurs at $\lambda =0.50$, $\beta =0.28$, whereas excessively low or high $\lambda$ values cause delays exceeding 120 episodes. The central region ($\lambda =0.28$–0.35, $\beta =0.45$–0.60) corresponds to stable training behaviour with convergence between 80 and 90 episodes, confirming that moderate coupling between predictive foresight and RL is essential for efficient adaptation. The lower-right corner of the heatmap ($\lambda >0.4$, $\beta <0.3$) shows oscillatory dynamics, attributed to the dominance of the short-term reinforcement component over the predictive term in Equation (23).

Overall, these findings verify that the proposed hybrid training strategy achieves both rapid convergence and low reward variance, ensuring policy generalisation across varying tariff, irradiance, and load conditions. The identified optimum around $\lambda \approx 0.30$ and $\beta \approx 0.50$ is consistent with the robustness envelope observed in Figure 7, confirming the internal coherence between the learning dynamics and the energy-system-level performance metrics. This coupling establishes reproducible and explainable learning behaviour suitable for real-time MG deployment.

3.4. Limitations

While the proposed H-RPEM demonstrates strong potential in enhancing operational sustainability in PV-EV MGs, several limitations should be acknowledged. First, the experimental validation was conducted using a single university campus PV-EV microgrid dataset, which constrains the direct generalisability of the findings to other geographical locations and climate zones and larger-scale smart city infrastructures. The meteorological, tariff, and mobility patterns embedded in the public EV Charging Patterns dataset [33] represent a single climatic regime and tariff structure, and they do not explicitly capture heterogeneous V2G behaviours across multiple sites. As a result, the present results should be interpreted as a proof-of-concept demonstration of H-RPEM rather than a comprehensive scalability study from campus- to city-level systems. Nevertheless, the proposed formulation is dataset- and scale-agnostic in the sense that different climate conditions, dynamic tariff profiles, and V2G participation patterns enter the framework through the stochastic input set w(t) and the reward/constraint parametrisation, without altering the underlying control architecture. Extending the analysis to multi-campus and city-scale case studies in contrasting climate zones, with heterogeneous dynamic tariffs and explicit V2G fleet models, is an important direction of our ongoing and future work. Second, the battery degradation model, although health-aware, relies on empirical coefficients that approximate electrochemical ageing processes rather than explicitly modelling solid–electrolyte interface dynamics, temperature-dependent diffusion, or the detailed coupling between SOC trajectories and SOH evolution. Advanced online SOH estimation methods, such as LSTM-based autoencoders trained on high-resolution electrothermal data or SVD-enhanced unscented Kalman filters, can, in principle, provide more accurate and physically informed SOH states for control. These approaches were not implemented here because the public microgrid dataset does not contain the necessary cell-level voltage, current, and temperature measurements and because the full design, training, and validation of such estimators would constitute a separate study. Future work will therefore focus on integrating and benchmarking these higher-fidelity SOH observers within H-RPEM, quantifying how improved SOH estimation propagates into economic, environmental, and degradation outcomes. Third, the hybrid controller was tested in a simulation environment using both historical and synthetic stochastic data; therefore, a hardware-in-the-loop or real-time field implementation is required to evaluate the communication latency, control reliability, and cyber-physical integration aspects. Finally, the current optimisation framework focuses on single-agent control and does not yet address multi-agent interactions or peer-to-peer energy trading among distributed MGs, which are increasingly relevant in smart city architectures. In particular, the present formulation treats the EV fleet as an aggregated load/storage entity and therefore does not incorporate decentralised decision-making, cooperative or competitive V2G strategies, or multi-agent interactions among heterogeneous EVs. Extending the H-RPEM architecture to a multi-agent setting, where individual EVs operate as V2G-enabled agents with their own preferences, arrival patterns, and state trajectories, represents a natural and necessary step for future development. Addressing these limitations will be the focus of future work aimed at improving the scalability, interoperability, and cyber-resilience of health-aware energy management systems.

4. Conclusions

This research presents a unified hybrid control framework for the sustainable management of PV-EV MGs. The developed H-RPEM integrates stochastic predictive optimisation with RL to enhance decision-making across multiple time horizons. The methodology emphasises both short-term adaptability to renewable intermittency and the long-term preservation of the battery SOH, introducing adaptive learning weights that regulate the interaction between predictive and data-driven components. Extensive experimental validation confirmed that the hybrid controller delivers superior performance compared with baseline and purely predictive benchmarks. H-RPEM achieved a 42.6% reduction in operating costs, a 38.7% decrease in CO₂ emissions, a 7.5% improvement in the SOH, and a 12.2% increase in renewable utilisation, based on daily averages over ten stochastic operating days. These results confirm that the controller delivers simultaneous economic, environmental, and battery health benefits. Statistical correlation analysis demonstrated that economic savings and degradation mitigation are strongly coupled, highlighting the internal consistency of the health-aware reward formulation. The system maintained operational robustness under stochastic variations in solar irradiance and tariff fluctuations, verifying its real-time applicability. The proposed architecture provides a flexible and generalisable platform for intelligent V2G management, scalable from MGs to district-level smart urban infrastructures. Future research will explore multi-agent coordination, digital twin-based predictive adaptation, and integration with blockchain-enabled peer-to-peer energy trading mechanisms further to enhance security and decentralised resilience in sustainable mobility ecosystems. In parallel, future work will deploy the proposed H-RPEM on datasets from different climate zones and system scales, including multi-campus and city-level PV-EV networks with diverse dynamic tariff structures and V2G participation patterns, in order to quantify rigorously its scalability and transferability. A further component of ongoing work will involve extending the proposed controller to a multi-agent configuration in which V2G-enabled EV fleets participate as distributed decision-making units. Such a formulation would enable coordinated or competitive behaviour under dynamic tariffs, enhance microgrid flexibility, and support the scalable integration of large EV populations within smart city environments.