Real-Time Residential Energy Optimization in Smart Grids: A Deep Reinforcement Learning Framework for Demand-Side Management

Abstract

The integration of photovoltaic generation, battery storage, electric vehicles, smart appliances, and dynamic electricity pricing has made residential energy management a challenging real-time optimization problem. Conventional demand-side management methods often depend on fixed rules and are less effective under uncertain solar generation, changing tariffs, and variable user demand. To address this issue, this paper proposes a Proximal Policy Optimization-based deep reinforcement learning framework for smart home energy management. The proposed PPO controller learns adaptive scheduling decisions using real-time PV output, electricity price, battery state of charge, EV charging status, and appliance operating conditions. The controller coordinates shiftable, controllable, and non-shiftable loads while reducing electricity cost and maintaining user comfort. The proposed method is compared with DDPG and TRPO. Simulation results show that PPO reduces the average daily energy cost by 4.7% compared with TRPO and 8.3% compared with DDPG. The results confirm that PPO is an effective and stable approach for real-time residential demand-side management.

1. Introduction

The global transition toward clean and sustainable energy systems is being driven by rapid technological advancement, increasing electricity demand, and growing concerns regarding climate change. As a result, conventional power networks are gradually evolving into smart grids that integrate advanced sensing, communication, automation, and data-driven control technologies. Unlike traditional electricity networks, which mainly support one-way power delivery from generation to consumers, smart grids enable bidirectional power and information exchange among utilities, distributed energy resources, and end users. This two-way interaction improves system observability, enhances operational flexibility, and supports more efficient coordination between generation, distribution, storage, and consumption.

The complexity of energy management has increased due to the changes from traditional power systems to smart-grid architecture. The complexity is caused by demand-side flexibility, decentralized distributed renewable energy resources, and the coordination of variable renewable energy sources [1]. A real-time smart grid with optimization under uncertain operating conditions requires intelligent control strategies that may adapt to dynamic changes while satisfying operational constraints. For such sequential decision-making tasks, deep reinforcement learning (DRL) has shown promise. By offering adaptive control without the need for an explicit mathematical model of the system, its model-free learning capabilities help to overcome the drawbacks of traditional optimization techniques. In general, the transition from conventional power grids to smart grids has been enhanced by the shift toward sustainable energy systems, which is motivated by climate change and advancements in technology. Advanced information, communication, and control technologies are incorporated into these enhanced networks to facilitate two-way data and electrical flows. This enables them to manage the growing number of distributed energy resources, including decentralized storage, electric vehicles, and renewable energy as discussed in [2]. While real-time operation and management enhance efficiency and environmental sustainability, they face significant challenges and serious practical problems.

Sudden variations in the output of renewable energy sources such as solar and wind, together with changing consumer load patterns, create a highly stochastic and dynamic operating environment, as discussed in [3]. In conventional power systems, electrical energy management was relatively less complex because centralized generation and predictable load profiles enabled stable operation while maintaining user comfort, satisfaction, and convenience. In this case, the flow of power from huge centralized generation plants to end users was usually unidirectional. The author in [4] proposed a multi-objective home appliance scheduling method for smart homes. The method minimizes electricity cost while considering user comfort and convenience. It learns user preferences from historical appliance usage instead of asking users to enter them manually. It also provides Pareto-optimal schedules, allowing users to select a suitable trade-off between cost saving and comfort. However, the complexity of managing electricity has grown greatly with the shift to smart-grid topologies.

Demand-side flexibility, the increasing use of distributed generating energy sources from renewables such as PV systems, small-scale wind production, and battery storage, as well as the intermittent nature of renewable energy sources, are some of the causes of this complexity. Grid operators can no longer rely entirely on set fixed operating schedules under such unpredictable, dynamic conditions. Rather, intelligent and adaptive control systems that can react to respond to changes in load demand, renewable energy output, and system constraints are necessary for real-time grid optimization, as discussed in [5]. These tactics must guarantee that equipment runs within allowable bounds, preserve grid stability, and minimize energy waste. Because they typically rely on precise mathematical models and predetermined system assumptions, traditional optimization techniques frequently have drawbacks in this situation. As a result, deep reinforcement learning has become an effective approach for contemporary energy management.

Due to these uncertainties, traditional fixed-rule and deterministic control strategies are often insufficient for reliable and cost-effective operation of modern smart grids. Such methods usually depend on predefined operating rules or accurate system models, which may not remain valid under rapidly changing real-time conditions. Therefore, intelligent and adaptive control strategies are required to monitor system states, learn from operational data, respond dynamically to variations in demand and generation, and satisfy technical constraints related to safety, reliability, cost, peak demand, and user comfort. In residential smart homes, these issues become even more significant, as the energy management system must coordinate with both adjustable and non-adjustable energy devices at the same time. Real-time decisions need to account for household consumption, solar rooftop solar power, battery charge and discharge cycles, electric vehicle charging, home appliance scheduling climate preferences, and varying electricity tariffs. As a result, an effective HEMS must dynamically adapt to uncertain and time-varying conditions. It should also account for diverse user preferences while balancing the trade-off between cost minimization and comfort feasibility. Traditional optimization approaches, such as model predictive control (MPC) and mixed-integer linear programming (MILP), often rely on accurate system models and forecasts, which are difficult to obtain in practice due to the stochastic nature of residential energy systems. Rule-based control strategies, while simple, lack the adaptability required for dynamic environments. However, realizing the full potential of residential DR is fraught with difficulties. The inherent uncertainties in electricity prices, renewable generation, and household consumption patterns complicate decision-making.

Deep Reinforcement Learning offers a promising solution for real-time demand-side management in residences, as it enables the system to derive optimal control and scheduling strategies through direct interaction with the smart-home environment. This approach provides an effective model-free control strategy for intelligent energy management in modern smart grids. It helps balance multiple objectives, including reducing electricity costs, minimizing peak demand, maximizing renewable energy utilization, managing battery storage efficiently, and maintaining user comfort. Unlike conventional optimization and rule-based control methods, DRL does not require an exact mathematical model of the power system. Instead, it learns optimal decision-making policies through continuous interaction with the operating environment. This feature makes DRL highly suitable for complex energy systems where uncertainty, nonlinearity, and time-varying conditions are difficult to model accurately. In particular, DRL is well suited for sequential decision-making problems in smart grids, where control actions taken at one time instant can influence future energy cost, battery state of charge, user comfort, peak demand, and renewable energy utilization. Rather, by monitoring system conditions, choosing control actions, and getting feedback through reward signals, it gradually enhances its decision-making ability. The demand for efficient household energy management has increased due to the growing use of solar PV systems and behind-the-meter battery storage. These technologies facilitate cost reductions, the use of renewable energy, the reduction of carbon emissions, while increasing complexity [6]. While solar power output is intermittent and entirely dependent on the weather, battery storage systems necessitate careful charging and discharging choices in order to optimize both technical and economic advantages. Furthermore, the operational needs and usage patterns of household equipment vary [7]. As a result, a successful home energy management system needs to minimize user intervention while coordinating local generation, storage, grid interaction, and household consumption in real time.

Three primary categories—shiftable, non-shiftable, and controllable loads—are used to further classify residential loads. Washing machines and dishwashers are examples of shiftable loads that can be postponed within user-specified timeframes as long as their working cycles are finished before a given deadline. Lighting and refrigeration are typical instances of non-shiftable loads since they necessitate an instant power supply and hence entail significant operating constraints. According to the author in [8], variable loads heating, ventilation, and air-conditioning systems in particular allow for flexible power consumption modification but have a direct impact on user comfort. In addition to taking into account photovoltaic unpredictability, time-varying prices, the need for electric vehicle charging, and unexpected user behavior, a better energy management approach must coordinate with these load types.

In general, operating a real-time smart grid might be an uncertain high-dimensional optimization challenge. Under predetermined assumptions, precise forecasts, and fixed system characteristics, conventional rule-based tactics and traditional optimization techniques might function well. However, their usefulness is constrained in real-world settings where user behavior, electricity costs, and renewable generation fluctuate constantly. Adaptive-learning-based control strategies are needed in these situations. DRL can learn from experience without the need for tagged training data; it is especially well suited for energy management. Every control decision affects future system states and expenses, including when to schedule an appliance, charge a battery, release stored energy, or use grid power. Problems with storage management and household energy scheduling are ideally suited to this sequential DRL. Such issues have been addressed by a variety of DRL methods, such as Deep Q-Network, Double DQN, Dueling DQN, Deep Deterministic Policy Gradient, Trust Region Policy Optimization, and Proximal Policy Optimization. Because of its stable training behavior, effective use of samples, and capacity to handle continuous action spaces, PPO has attracted significant attention in power and energy management applications due to its stable learning performance, efficient policy updates, and suitability for handling complex control problems.

Results are validated against a real-world microgrid testbed in Cyprus, achieving more than 95% similarity with empirical data in [9]. These features, such as regulating the energy exchange between the grid and local storage, modifying HVAC power use, or regulating a battery’s charge rate are crucial for home energy management because they are based on continuous rather than binary data. The controller of a typical PPO-based home energy management system gets real-time data from smart meters, load profiles, battery condition, solar generation, weather forecasts, electricity price signals, and user preferences. In order to coordinate the utility grid, solar panels, battery storage, and domestic appliances, the controller uses these inputs to determine the best course of action. Reducing peak demand, increasing self-consumption of renewable energy, minimizing power costs, and maintaining residential comfort are the primary goals. Recent DRL-based predictive energy management strategies have also been applied to fuel cell buses by integrating speed and passenger prediction to improve energy management performance and maintain battery state-of-charge stability while improving fuel economy and system health management [10]. Real-time price forecasting with deep reinforcement learning has been applied for predictive home energy management systems to improve scheduling decisions under dynamic pricing conditions to both forecasted prices and demand response. Through a TD3 agent, it learns optimal scheduling policies for household appliances and energy storage, balancing cost savings with user comfort signals [11].

Reinforcement learning has also been applied for environmentally sustainable HVAC management in smart buildings to improve energy efficiency and comfort-aware control [12].

Intelligent control systems that can adjust to unpredictable and dynamic operating situations are necessary for modern household energy management. Because traditional methods rely on precise and fixed forecasts, they are frequently inadequate. An efficient model-free method for learning adaptive control policies directly from experience is provided by DRL, especially PPO. This makes it possible for smart homes to achieve lower energy expenditures, better use of renewable energy, lower peak demand, and dependable operation even in the face of volatility in the real world. Figure 1 shows the proposed PPO-based smart-home energy management system framework.

Figure 1. Proposed PPO-based smart-home energy management system framework.

The objective of this research is to create an intelligent and flexible energy management system for smart homes that incorporates battery energy storage and renewable energy sources like solar photovoltaic or wind production systems. By deciding whether to use stored battery energy, schedule flexible appliances, draw electricity from the utility grid, or modify controllable loads like heating, ventilation, and air conditioning systems, the proposed EMS aims to optimize household energy flow in real time. This decision-making process must be carried out in the face of fluctuating electricity prices and unpredictable environmental factors, especially those that impact the production of renewable energy. The environment has a significant impact on solar and wind energy outputs; therefore, the EMS needs to be able to react quickly to changes in home demand, power costs, and generation. Additionally, the system takes into account several domestic appliance classifications, each of which has unique operational features. Appliances that can be shifted, including dishwashers and washing machines, can be postponed within user-specified timeframes. Appliances that cannot be moved, including refrigerators and necessary lighting, need a steady and instant power supply. It is possible to gradually modify controllable loads, like heating and cooling systems, while yet preserving appropriate comfort levels. To lower power costs, boost self-consumption of locally produced renewable energy, and maintain consumer comfort, these resources must be managed in concert. The specific research objectives of this study are summarized as follows:

• To develop a PPO-based DRL policy for real-time residential energy management that schedules flexible household loads under dynamic RTP signals, PV variability, and appliance operating constraints.
• To construct a multi-objective reward formulation that captures the trade-off among electricity cost reduction, peak-demand mitigation, user-comfort maintenance, EV charging satisfaction, and battery degradation cost.
• To evaluate the proposed PPO-based HEMS controller against base methods using performance metrics, and operating conditions.

The rest of this paper is structured as follows. The literature on DRL-based home energy management systems utilizing existing algorithms is reviewed in Section 2. The decision-making framework, the energy management problem, and the working principles of DRL methods are presented in Section 3. The results of the comparative simulation are shown and discussed in Section 4. The work is finally concluded in Section 5, which also suggests future research areas.

2. Existing Methodology

Intelligent and adaptable energy management in smart-grid systems is made possible by recent developments in artificial intelligence, especially deep reinforcement learning. In applications like battery energy storage control, demand response optimization, and renewable energy coordination under uncertain operating conditions, DRL algorithms like Deep Q-Networks, Deep Deterministic Policy Gradient, and Proximal Policy Optimization have shown great promise [13]. DRL-based strategies can lower power costs, increase the use of renewable energy, and strengthen system resilience in the face of erratic fluctuations in pricing, generation, and demand. Despite these benefits, there are still a number of obstacles to overcome in the real-world use of DRL for energy management. Slow convergence is still a drawback in training, as it frequently takes a lot of events for the agent to pick up a stable and useful policy. Designing incentive functions is a significant problem. To achieve benefits from smart-grid and smart-home applications a balance must be struck between a number of conflicting goals, such as maximizing the use of renewable energy, minimizing costs, and reducing peak demand and user comfort. It is difficult to design for such a powerful and balanced reward function, yet it is still a crucial factor in smart-home energy management.

Conventional residential energy optimization studies mainly used MILP, DP, and heuristic algorithms such as PSO and GA to minimize electricity costs under ToU tariffs while maintaining user comfort, but these methods often required accurate forecasts, suffered from computational complexity or parameter tuning issues, and became less reliable under real-time uncertainties such as price variations, weather changes, and fluctuating household demand. The constraints of model-based optimization led to RL-based HEMS such as Q-learning algorithms which decreased dependency on system models but suffered from state-space explosion and poor generalization under continuous and variable residential situations. Demand-side management based on deep reinforcement learning advances an essential path of exploration in the development of smart grids. For contemporary energy systems, these methods provide flexible, scalable, and privacy-conscious options. The works under review demonstrate the application of multi-agent and federated DRL systems to distributed energy management. Further research is needed to develop reliable learning models, establish fair compensation mechanisms, and explore the integration of deep reinforcement learning with emerging technologies such as quantum computing. DRL facilitates adaptive decision-making and real-time learning in home energy management systems, enabling residential loads to be adjusted in response to shifting electricity costs, the production of renewable energy, and user preferences. This can lower electricity costs, increase energy efficiency, and preserve occupant comfort. However, there are still a number of issues that need to be resolved before widespread adoption of DRL-based HEMS is possible, including computational complexity, training stability, data availability, cybersecurity, and real-time application [14]. To increase the dependability, scalability, and usefulness of DRL-based demand-side energy management systems, more research is necessary.

A data-driven deep reinforcement learning approach for optimizing home energy management system performance in unpredictable circumstances was presented by the authors in [15]. The proposed strategy is utilized to solve high-dimensional sequential decision-making difficulties, manage uncertain photovoltaic generation, adapt to fluctuating electricity prices, and improve user comfort, among other practical issues. The system forecasts solar power generation and energy prices by combining bidirectional gated recurrent unit networks with a thermal comfort rating model. The appropriate energy-management actions are then determined using these projected values in a soft actor-critic-based decision-making structure. A possible method for coordinated, intelligent control of microgrids and EVs was investigated in [16] using multi-agent PPO. Although actual deployment will necessitate advancements in data, communication infrastructure, and safe RL algorithms, it improves efficiency, stability, and renewable integration. Using the Enhanced Northern Goshawk Optimization method, the authors looked into how green hydrogen may be integrated into smart-home energy management.

An effective method for incorporating DERs into smart grids is optimization-based household energy management; according to the author in [17], costs are decreased, and grid stability and sustainability are enhanced. This approach enhances the algorithm’s abilities to search and prevents convergence toward local optimal solutions. HEMS can enable intelligent and flexible home electricity management, increasing energy efficiency, lowering electricity costs, and preserving occupant comfort by continuously learning from real-time data and adapting to changing circumstances, including dynamic power rates, changeable photovoltaic generation, and varying household demand. The wider adoption of DRL-based energy management systems is being supported by ongoing research and technology developments, though real-time implementation still faces practical challenges such as training complexity, computational requirements, and deployment dependability. Consequently, DRL presents a practical approach for creating more intelligent, adaptable, and economical domestic energy management solutions. To obtain optimal scheduling decisions under uncertain operating conditions, a model-free deep reinforcement learning framework is formulated as a Markov decision process. This approach is more appropriate for realistic home power management since it takes into account both active and reactive power, in contrast to a number of traditional HEMS techniques. But the study also found that consistent performance and efficient learning require a lot of training data. The hybrid EMS that the authors created in [18] improves load balancing, reduces reliance on diesel backup, and lowers operating costs when compared to standalone optimization or RL approaches. It also adjusts to various industrial microgrid sizes and configurations.

The smart-grid environment is inherently a large-scale, stochastic, and partially observable cyber-physical system, making it an ideal testbed for DRL’s sequential decision-making capabilities. The SG makes use of a Wi-Fi-enabled Internet of Things network that allows effective communication between linked devices. A Raspberry Pi serves as the main home server, a Jetson Nano is used for deep learning processing, and microcontrollers are used for the direct control of smart devices as each hardware unit in the HEMS prototype works as the central home server, the Jetson Nano handles deep learning-based forecasting, and the microcontrollers receive control commands and operate the smart devices. The system can efficiently monitor power use, forecast future energy demand, and coordinate smart devices, according to experimental validation using real household energy data. However, cybersecurity was also a significant drawback with regard to preventing threats such as unauthorized access, data privacy concerns, and communication vulnerabilities since it relies on IoT-based connectivity and linked devices.

Security and privacy issues are brought forth by this change, particularly because of the dependence on prosumers and centralized data storage as reported in [19]. Therefore, for the actual implementation of such IoT-based energy management systems, more robust security measures are needed. The author of [20] investigated how deep reinforcement learning methods, in particular, used in smart-grid systems, manage energy in a more flexible and dynamic manner. In applications like battery energy, demand–response scheduling, and renewable energy coordination under uncertain operating conditions, DRL techniques like TRPO, DDPG, and PPO have proven to be highly effective. In HEMS, DQN and DDQN are used for discrete ON/OFF appliance scheduling while continuous power control for batteries, EVs, and HVAC systems is handled by DDPG, TD3, and SAC. SAC enhances exploration under uncertain conditions. For hybrid discrete–continuous control, TRPO and PPO provide stable policy updates, with PPO preferred due to its lower complexity and easier implementation. Positive benefits in demand-side energy management must balance lowering electricity costs, maximizing the use of renewable energy, minimizing peak demand, and improving occupant comfort. It is ideal for contemporary home and distributed energy systems due to its capacity to offer scalable, adaptable, and privacy-focused energy management solutions.

The proposed framework considers both discrete appliance scheduling and continuous power-control decisions to coordinate appliance scheduling strategy learned by the DRL agent. The controller reduces operating cost, charging the ESS during off-peak or surplus-PV intervals, and discharging the ESS during high-price or peak-demand intervals. In addition, the EV charging schedule is adjusted according to availability and SOC requirements, while HVAC operation is controlled to maintain user comfort.

2.1. Deep Reinforcement Learning Techniques

Deep reinforcement learning has emerged as a successful method for making successive decisions in dynamic and unpredictable environments. It is ideal for demand-side energy management applications due to its adaptability and model-free learning capacity. Without a precise mathematical model of the system, a DRL agent may learn effective control tactics and create intricate scheduling rules through constant interaction with the environment. This is particularly useful when several conflicting objectives must be considered simultaneously, such as reducing electricity cost, minimizing peak demand, increasing photovoltaic energy utilization, and maintaining occupant comfort. The reinforcement learning process is commonly formulated as a Markov decision process, represented by the tuple $(S,A,P,R,\gamma )$. In this case, S stands for the state space, A for the action space, P for the state transition probability, R is for the reward function, and $\gamma$ for the discount factor. The main objective of the agent is to learn an optimal policy $\pi (a\mid s)$ that maximizes the expected cumulative reward over the decision horizon.

The state and action spaces of demand-side management challenges are frequently diverse, vast, and continuous. As a result, conventional table-based reinforcement learning techniques are unsuitable for real-world application. Additionally, the operational environment is typically non-stationary because of changes in family consumption patterns, battery state of charge, cost of electricity, and renewable energy output. In order to overcome these difficulties, deep reinforcement learning uses deep neural networks as function approximators for value estimation and policy representation [21]. This makes it possible for DRL-based controllers to function well in complicated, unpredictable, and nonlinear smart-grid systems.

2.1.1. Deep Deterministic Policy Gradient Algorithm

The Deep Deterministic Policy Gradient algorithm is an extension of deterministic policy gradient methods designed for high-dimensional continuous control problems. The author in [22] executed a multi-agent deep reinforcement learning (MADRL) framework using Deep Deterministic Policy Gradient (DDPG). Each agent represents a machine or production unit, learning to schedule tasks adaptively under time of use pricing. DDPG uses deep neural networks as function approximators, allowing the agent to learn effective policies in complex environments with continuous state and action spaces. DDPG combines deterministic policy gradient theory with deep learning and improves sample efficiency through off-policy experience replay, where previous experiences are stored and reused during training.

This capability makes DDPG highly suitable for energy management applications that require real-valued control decisions which include determining battery charging and discharging power, adjusting HVAC power levels, selecting electric vehicle charging rates, and managing flexible load operation. Since these decisions are continuous in nature, DDPG can provide more precise control compared with purely discrete action-based methods. The author in [23] proposed Equation (1), which represents how the deterministic policy relates to the objective of maximizing the expected return as follows:

$$J\left(\theta \right)={\mathbb{E}}_{s\sim {\rho }^{\mu }}\left[R\left(s,{\mu }_{\theta }\left(s\right)\right)\right]$$

(1)

where $J\left(\theta \right)$ represents the performance objective of the policy, ${\rho }^{\mu }$ denotes the discounted state distribution generated by policy $\mu$, and ${\mu }_{\theta }\left(s\right)$ is the deterministic action selected by the actor network for state s. According to the deterministic policy gradient theorem, the policy parameters are updated using the following gradient expression [15] in Equation (2):

$${\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{s\sim {\rho }^{\mu }}\left[{\nabla }_{\theta }{\mu }_{\theta }\left(s\right){\nabla }_{{\mu }_{\theta }}{Q}^{\mu }\left(s,{\mu }_{\theta }\left(s\right)\right)\right].$$

(2)

where $Q^{\mu }(s,a)$ is the action-value function estimated by the critic network. The actor network generates the continuous control action, while the critic network evaluates the quality of that action. Through this actor–critic structure, DDPG learns efficient control policies for residential energy management under uncertain and dynamic operating conditions.

2.1.2. Trust Region Policy Optimization

Trust Region Policy Optimization (TRPO) is an on-policy reinforcement learning algorithm designed to make policy-gradient training more stable. It acts as a stable policy-gradient method that constrains policy updates using a trust-region formulation to encourage monotonic policy improvement. It restricts each policy update within a trust region, so the new policy does not move too far from the previous one, as described in [24]. This helps avoid sudden performance drops caused by overly large updates. It maximizes an advantage-based objective while limiting policy variation using Kullback–Leibler divergence according to Equation (3)

$$\underset{\theta }{\max }\mathbb{E}\left[{\displaystyle \frac{{\pi }_{\theta }(a\mid s)}{{\pi }_{{\theta }_{\mathrm{old}}}(a\mid s)}}{A}_{{\theta }_{\mathrm{old}}}(s,a)\right]$$

(3)

subject to

$$\mathbb{E}\left[D_{\mathrm{KL}}\left({\pi }_{{\theta }_{\mathrm{old}}}\parallel {\pi }_{\theta }\right)\right]\le \delta$$

(4)

where ${\pi }_{\theta }$ is the updated policy, ${\pi }_{{\theta }_{\mathrm{old}}}$ is the previous policy, $A(s,a)$ is the advantage function, and $\delta$ controls the maximum allowed policy change. Battery charging and discharging, HVAC management, EV charging, and flexible appliance scheduling are all possible with DSM and smart grid. Its primary goals are to lower power costs, limit peak demand, increase the usage of renewable energy, and preserve customer comfort. Because of its steady and regulated policy improvement, TRPO is frequently regarded as a trustworthy benchmark despite being computationally costly and less sample-efficient than off-policy techniques like DDPG.

2.2. Load Classification

In HEMS, electrical loads cannot be managed uniformly in DSM due to the varying operational requirements, flexibility, and impact on user comfort of each equipment. As a result, creating an effective home energy management system requires careful load classification. The controller can determine which loads can be moved to a different time, which loads can be modified based on energy conditions, and which loads must run continuously without interruption by classifying appliances based on their degree of flexibility.

2.2.1. Shiftable Appliances

Shiftable appliances may be set to operate within a predetermined timeframe, making them adaptable home loads. Within the designated start and end periods, these appliances must finish the necessary functioning cycle. To guarantee correct task completion and user convenience, the appliance’s cycle should remain uninterrupted after it starts operating, as shown in Equation (5) [13]. The entire energy consumption for each shiftable appliance is represented as

$$E_i^{\mathrm{sh}}={\displaystyle \sum _{t=T_i^{\mathrm{Start}}}^{T_i^{\mathrm{End}}}}{P}_i^{\mathrm{sh}}{x}_{i,t}$$

(5)

where $E_i^{\mathrm{sh}}$ denotes the total energy consumed by the $i^{\mathrm{th}}$ shiftable appliance, $P_i^{\mathrm{sh}}$ represents its rated power, and $x_{i,t}\in \{0,1\}$ is the binary operating status of the appliance at time t. Here, $x_{i,t}=1$ indicates that the appliance is ON, while $x_{i,t}=0$ indicates that it is OFF. The parameters $T_i^{\mathrm{Start}}$ and $T_i^{\mathrm{End}}$ define the permissible operating time window within which the appliance must be scheduled.

2.2.2. Controllable Appliances

Controllable appliances are home appliances that can have their power usage continuously changed within certain technological bounds. These include battery energy storage systems, electric water heaters, HVAC systems, and electric car chargers. Because their power levels may be adjusted in real time, controllable appliances offer more flexibility than non-shiftable loads, which follow set demand patterns, or shiftable loads, which are primarily managed through ON/OFF scheduling. Controllable appliance power adjustment is dependent on a number of operational factors, such as the price of energy in real time, the user’s choice for comfort, the battery’s state of charge, the availability of solar power, and grid limitations. As a result, these appliances are crucial for lowering power costs, controlling peak demand, increasing the use of renewable energy, and improving overall energy efficiency. The power consumption of any controlled appliance must stay within its permitted working range. Equation (6) expresses this restriction.

$$P_i^{\min }\le P_{i,t}^{\mathrm{ctrl}}\le P_i^{\max }$$

(6)

where $P_{i,t}^{\mathrm{ctrl}}$ represents the power consumed by controllable appliance i at time t. The parameter $P_i^{\min }$ denotes the minimum allowable power level, while $P_i^{\max }$ indicates the maximum permissible power level of the appliance.

This constraint ensures that each controllable device operates safely and reliably within its technical limits. At the same time, it allows the home energy management system to dynamically allocate power according to cost, comfort, renewable energy availability, and system operating requirements.

2.2.3. Non-Shiftable Appliances

Non-shiftable loads are household appliances that operate according to fixed usage patterns and cannot be controlled or rescheduled by the home energy management system. These loads either operate continuously or run during predetermined time intervals. Examples include refrigerators, security systems, internet routers, and other essential appliances. Since their operation follows regular household behavior, their energy demand can usually be estimated in advance. In Equation (7), the total power consumed by all non-shiftable loads at time t is expressed as

$$P_t^{\mathrm{NS}}={\displaystyle \sum _{i=1}^{N_{\mathrm{ns}}}}{P}_i^{\mathrm{ns}}$$

(7)

where $P_t^{\mathrm{NS}}$ denotes the total non-shiftable load demand at time t, $P_i^{\mathrm{ns}}$ represents the rated power of the $i^{\mathrm{th}}$ non-shiftable appliance, and $N_{\mathrm{ns}}$ is the total number of non-shiftable appliances.

For an individual non-shiftable appliance $n_i$, the power consumption is defined as in Equation (8):

$$P_t^{\left(n_i\right)}=\left\{\begin{array}{cc}{P}_{\mathrm{rated}}^{\left(n_i\right)},\hfill & t\in \left[t_{\mathrm{ini}}^{\left(n_i\right)},t_{\mathrm{end}}^{\left(n_i\right)}\right]\hfill \\ 0,\hfill & t\notin \left[t_{\mathrm{ini}}^{\left(n_i\right)},t_{\mathrm{end}}^{\left(n_i\right)}\right]\hfill \end{array}\right.$$

(8)

where $P_{\mathrm{rated}}^{\left(n_i\right)}$ is the rated power of appliance $n_i$, while $t_{\mathrm{ini}}^{\left(n_i\right)}$ and $t_{\mathrm{end}}^{\left(n_i\right)}$ represent its starting and ending operating times. The photovoltaic generation at time t mainly depends on the solar irradiance and the conversion efficiency of the PV system. It is calculated as

$$P_t^{\mathrm{PV}}={\eta }_{\mathrm{pv}}{A}_{\mathrm{pv}}{G}_t$$

(9)

where $P_t^{\mathrm{PV}}$ denotes the photovoltaic output power, ${\eta }_{\mathrm{pv}}$ represents the conversion efficiency of the PV system, $A_{\mathrm{pv}}$ is the effective area of the PV panels, and $G_t$ indicates the solar irradiance at time t. The instantaneous power balance of the residential energy system is determined according to Equation (10):

$$P_t^{\mathrm{grid}}+P_t^{\mathrm{PV}}+P_t^{\mathrm{BSS}}=P_t^{\mathrm{load}}$$

(10)

where $P_t^{\mathrm{grid}}$ represents the power exchanged with the utility grid. A positive value indicates power purchased from the grid, while a negative value indicates power exported to the grid. The term $P_t^{\mathrm{PV}}$ denotes solar photovoltaic generation, and $P_t^{\mathrm{BSS}}$ represents battery storage power. A positive $P_t^{\mathrm{BSS}}$ value indicates battery discharging, whereas a negative value indicates battery charging. Finally, $P_t^{\mathrm{load}}$ represents the total household demand, including shiftable, controllable, and non-shiftable appliances. This power-balancing constraint guarantees that the home load demand is met at every time interval by the total power available from the grid, solar energy, and battery storage. It facilitates the dependable operation of the suggested energy management framework and preserves energy homeostasis inside the smart home.

2.3. Problem Formulation

In the proposed framework, a proximal policy optimization (PPO) agent is used to learn the optimal scheduling policy, denoted as ${\pi }_{\theta }(a\mid s)$. In addition to regulating excessive energy use, peak demand violations, and user discomfort, the incentive function promotes efficient power consumption. Equation (11) states that the overarching goal is to minimize the total operating cost over the whole scheduling horizon.

$$\min {\displaystyle \sum _{t=0}^{T_{\mathrm{end}}}}\left(C_t^E+C_t^{\mathrm{HVAC}}+C_t^{\mathrm{EWH}}+C_t^{\mathrm{EV}}+C_t^{\mathrm{ESS}}\right)$$

(11)

where $C_t^E$ represents the cost of electricity purchased from the grid at time t, $C_t^{\mathrm{HVAC}}$ denotes the operational cost of the HVAC system, $C_t^{\mathrm{EWH}}$ is the cost associated with the electric water heater, $C_t^{\mathrm{EV}}$ represents the electric vehicle charging cost, and $C_t^{\mathrm{ESS}}$ indicates the cost related to the energy storage system, including charging, discharging, and possible energy loss effects. Table 1 presents a summary of recent deep reinforcement learning (DRL)-based home energy management system (HEMS) approaches. The reviewed studies mainly focus on optimal appliance scheduling, energy storage management, electric vehicle charging, HVAC control, renewable energy utilization, and user comfort improvement. Different DRL algorithms such as TRPO, PPO, DDPG, DQN, DDQN, and A3C have been applied to handle dynamic energy management problems under uncertain operating conditions. From the comparison, PPO- and DDPG-based methods show strong potential for real-time residential energy optimization, while TRPO is often used as a reliable benchmark because of its stable policy improvement.

Table 1. Summary of DRL-based HEMS approaches reported in the literature.

Ref.	Methodology	Key Contribution
[25]	TRPO, DDPG, A3C, TAC	TRPO achieved the highest self-sufficiency rate, outperforming A3C, DDPG, and TAC by 0.72–3.5%.
[26]	PPO	Uncertainty-aware PPO-based HEMS with a GUI application. The framework integrates ESS, EV, PV, and wind generation and achieves 44% energy cost reduction with rapid training convergence for real-time deployment.
[27]	PPO, DDQN	Comparative analysis of PPO and DDQN for HEMS by balancing electricity cost and residential comfort. PPO demonstrated superior thermal comfort maintenance and cost reduction, particularly in colder seasons.
[28]	DDPG, DQN	Novel dual-agent approach combining DDPG for ESS optimization with PV forecasting error compensation and DQN for appliance scheduling. The method reduces electricity cost while increasing occupant satisfaction.
[29]	DDPG	IoT–Edge–Cloud computing system for smart-home temperature control. The framework uses DDPG agents for HVAC and ESS management with Average Opinion collaborative decision-making.

3. PPO-Based DRL for Energy Management

The predicted energy management framework is designed as a Markov decision process, where the agent gradually learns the best control approach by interacting with a simulated smart-home environment. The domestic energy system’s current operational conditions, such as the cost of electricity, solar energy production, the status of appliances, and the battery’s charge level, are represented by the state space. These findings provide the agent with the knowledge it needs to make intelligent scheduling choices in the face of shifting system conditions. The hybrid structure of the action space incorporates both discrete and continuous control decisions. While continuous activities relate to power regulation duties like battery charging or discharging, HVAC power adjustment, and flexible load control, discrete actions reflect ON/OFF scheduling choices for household appliances. This hybrid action formulation reflects practical smart-home operation, where both binary appliance scheduling and continuous power allocation are required. The learning method used is proximal policy optimization because of its robust performance in challenging continuous control situations, steady training behavior, and sample efficiency. A multi-objective reward function is used to establish the learning objective, which penalizes breaches of user comfort, peak demand, and high electricity costs. Through repeated training sessions and ongoing contact with the environment, the PPO agent develops an optimal scheduling strategy while simultaneously encouraging efficient energy use, better solar self-utilization, and smoother home load operation. Under unpredictable circumstances, like changeable power prices, fluctuating solar output, and shifting household demand, the learned policy allows the home energy management system to balance cost reduction, operational efficiency, renewable energy use, and occupant comfort. As a result, the suggested PPO-based framework offers a flexible and useful approach to intelligent smart-home energy management.

3.1. System Model

The HEMS monitors the smart home’s current state of operation. This comprises the amount of electricity consumed by households, the production of renewable energy, the condition of batteries, the current price of electricity, and the status of appliances. The PPO agent uses these data to choose appropriate control actions, like controlling controllable loads, charging or discharging the battery, and scheduling flexible appliances. An overview of the suggested PPO-based home energy management system functioning in a smart-grid setting is shown in Figure 2 (visualizations were generated using a graphics assistance tool). Multiple prosumers, battery energy storage devices, and adjustable home loads make up the structure. The system operates in discrete time steps $t\in \{0,1,\dots ,T\}$, where each interval has a duration of $\Delta t$.

Figure 2. DRL-PPO-based energy management architecture for smart-home demand-side management.

The main importance of the suggested methodology is to determine the best control measures that satisfy system restrictions, lower peak demand, and reduce overall operating costs. The framework not only reduces costs but also enhances the use of renewable energy, preserves occupant comfort, and facilitates grid-friendly operation. The PPO-based HEMS offers an efficient solution for intelligent residential energy management in smart-grid applications by adaptively coordinating dispersed household energy resources. The proposed HEMS operates with a 15 min control step, and the scheduling horizon is defined as 24 h, corresponding to 96 time steps per day.

3.2. Mathematical Modeling of the Smart-Home Energy System

The smart-home environment is implemented as a discrete-time, equation-based simulator. At each 15 min interval, the simulator receives the electricity price, PV generation, outdoor temperature, non-shiftable demand, appliance availability, EV connection status, and current EV/ESS SOC. The controller then determines the appliance-scheduling and continuous power-control actions. Before the reward is calculated, these actions are constrained by the physical operating limits of the ESS, EV, HVAC, EWH, appliances, and utility grid. The resulting ESS SOC, EV SOC, indoor temperature, water temperature, and grid power are subsequently updated using the following state-transition equations. A 24 h episode therefore contains 96 simulation steps. The exogenous profiles and initial conditions are kept identical for all controllers to ensure that their performance is evaluated under the same operating conditions.

3.2.1. Battery Energy Storage System (ESS) Dynamics

The state of charge (SOC) of the ESS evolves according to the charging and discharging powers and their respective efficiencies. At each discrete time step t, the ESS dynamics are expressed as

$$SO{C}_{t+1}^{\mathrm{ESS}}=SO{C}_t^{\mathrm{ESS}}+{\displaystyle \frac{\left({\eta }_{\mathrm{ch}}{P}_{\mathrm{ch},t}-{\displaystyle \frac{P_{\mathrm{dis},t}}{{\eta }_{\mathrm{dis}}}}\right)\Delta t}{E_{\mathrm{cap}}^{\mathrm{ESS}}}}$$

(12)

Here, $SO{C}_t^{\mathrm{ESS}}$ denotes the current state of charge of the ESS in per-unit, $P_{\mathrm{ch},t}$ and $P_{\mathrm{dis},t}$ are the charging and discharging powers in kW, respectively, and $E_{\mathrm{cap}}^{\mathrm{ESS}}=6.4\mathrm{kWh}$ is the rated battery capacity. To prevent irreversible battery degradation, the ESS state of charge is restricted to the safe operating range:

$$SO{C}_{\min }^{\mathrm{ESS}}\le SO{C}_t^{\mathrm{ESS}}\le SO{C}_{\max }^{\mathrm{ESS}}$$

(13)

The charging and discharging powers are constrained by

$$0\le P_{\mathrm{ch},t}\le P_{\max }^{\mathrm{ESS}},0\le P_{\mathrm{dis},t}\le P_{\max }^{\mathrm{ESS}}$$

(14)

The battery degradation cost is modeled using the Depth-of-Discharge (DoD) cycle-degradation model:

$$c_t^{\mathrm{ESS}}=C_0\left({\displaystyle \frac{kSO{C}_t^{\mathrm{ESS}}-\varphi }{F_{\max }\times 15\times 365\times 24}}\right){\displaystyle \frac{\Delta t}{60}}+C_0{\displaystyle \frac{\Delta L_t^{\mathrm{DoD}}}{f\left(\Delta L_t^{\mathrm{DoD}}\right)}}$$

(15)

The incremental DoD variation is determined by

$$\Delta L_t^{\mathrm{DoD}}={\displaystyle \frac{-P_{\mathrm{dis},t}\Delta t}{{\eta }_{\mathrm{dis}}{E}_{\mathrm{cap}}^{\mathrm{ESS}}}},$$

(16)

and the corresponding DoD cycle-life function is

$$f(\Delta L)=\left[1.06{(\Delta L)}^4-2.08{(\Delta L)}^3+2.66{(\Delta L)}^2-1.07\Delta L+0.17\right]\times {10}^5.$$

(17)

3.2.2. Indoor Thermal Dynamics of the HVAC System

The indoor temperature is modeled using a first-order equivalent thermal resistance–capacitance (RC) network. This model represents the thermal inertia of the building and the heat exchange between the indoor and outdoor environments:

$$T_{t+1}^{\mathrm{in}}=ϵT_t^{\mathrm{in}}+(1-ϵ)\left(T_t^{\mathrm{out}}+{\eta }_{\mathrm{HVAC}}{P}_{\mathrm{HVAC},t}{\displaystyle \frac{\Delta t}{A}}\right)$$

(18)

Here, $ϵ=0.968$ is the thermal inertia factor, $A=7.27\times {10}^{-3}$ kW/°C is the effective thermal conductivity, $T_t^{\mathrm{in}}$ and $T_t^{\mathrm{out}}$ are the indoor and outdoor temperatures, respectively, and $P_{\mathrm{HVAC},t}$ is the electrical power consumed by the HVAC system.

Application of Equation (18) enables the PPO agent to learn pre-cooling or pre-heating strategies during low-tariff periods while maintaining the thermal comfort of the occupants.

The HVAC thermal-comfort cost is defined as

$$c_t^{\mathrm{HVAC}}={\alpha }_1{\left({\displaystyle \frac{\left|T_t^{\mathrm{in}}-T_{\mathrm{set}}^{\mathrm{in}}\right|}{\Delta T_{\mathrm{set}}^{\mathrm{in}}}}\right)}^2$$

(19)

where

$${\alpha }_1=0.01,T_{\mathrm{set}}^{\mathrm{in}}={22}^{\circ }\mathrm{C},\Delta T_{\mathrm{set}}^{\mathrm{in}}=2^{\circ }\mathrm{C}$$

(20)

3.2.3. Electric Water Heater Dynamics

The hot-water temperature is represented using a lumped thermal-capacity model:

$$T_{t+1}^{\mathrm{wa}}=T_t^{\mathrm{wa}}\exp \left(-{\displaystyle \frac{\Delta t}{{\tau }_t}}\right)+Q_t\left[1-\exp \left(-{\displaystyle \frac{\Delta t}{{\tau }_t}}\right)\right]$$

(21)

The equivalent steady-state temperature $Q_t$ and thermal time constant ${\tau }_t$ are calculated as

$$Q_t={\displaystyle \frac{G{T}_t^{\mathrm{wa}}+\rho C_p{W}_t{T}_{\mathrm{co}}+P_t^{\mathrm{EWH}}}{G+\rho C_p{W}_t}}$$

(22)

and

$${\tau }_t={\displaystyle \frac{\rho C_{p}V}{G+\rho C_p{W}_t}}$$

(23)

Here, $\rho =1000\mathrm{kg}/{\mathrm{m}}^3$ is the density of water, $C_p=4.2\times {10}^3\mathrm{J}/\left(\mathrm{kg}\mathrm{K}\right)$ is the specific heat capacity of water, $G=2.60\mathrm{W}/\mathrm{K}$ is the thermal conductivity of the tank, $W_t$ is the hot-water flow rate, $T_{\mathrm{co}}=1^{\circ }\mathrm{C}$ is the cold-water inlet temperature, $V=10\mathrm{L}$ is the tank volume, and $P_t^{\mathrm{EWH}}$ is the electrical heating power. The EWH thermal-comfort cost is formulated as

$$c_t^{\mathrm{EWH}}={\alpha }_2{\left({\displaystyle \frac{\left|T_t^{\mathrm{wa}}-T_{\mathrm{set}}^{\mathrm{wa}}\right|}{\Delta T_{\mathrm{set}}^{\mathrm{wa}}}}\right)}^2$$

(24)

where

$${\alpha }_2=0.01,T_{\mathrm{set}}^{\mathrm{wa}}={60}^{\circ }\mathrm{C},\Delta T_{\mathrm{set}}^{\mathrm{wa}}=3^{\circ }\mathrm{C}$$

(25)

3.2.4. Electric Vehicle Battery Dynamics

The EV battery SOC evolves according to the supplied charging power:

$$SO{C}_{t+1}^{\mathrm{EV}}=SO{C}_t^{\mathrm{EV}}+{\displaystyle \frac{{\eta }_{\mathrm{EV}}{P}_{\mathrm{EV},t}\Delta t}{E_{\mathrm{cap}}^{\mathrm{EV}}}},\forall t\in [t_{\mathrm{arr}},t_{\mathrm{dep}}]$$

(26)

The EV charging power is set to zero outside the user-defined availability window:

$$P_{\mathrm{EV},t}=0,\forall t\notin [t_{\mathrm{arr}},t_{\mathrm{dep}}]$$

(27)

The charging-power constraint is

$$0\le P_{\mathrm{EV},t}\le P_{\max }^{\mathrm{EV}},P_{\max }^{\mathrm{EV}}=6\mathrm{kW}$$

(28)

Here, ${\eta }_{\mathrm{EV}}=0.95$ is the on-board charger efficiency and $E_{\mathrm{cap}}^{\mathrm{EV}}=24\mathrm{kWh}$ is the EV battery capacity. The charging schedule must ensure that the required SOC is reached before departure:

$$SO{C}_{\mathrm{dep}}^{\mathrm{EV}}\ge SO{C}_{\mathrm{req}}^{\mathrm{EV}},SO{C}_{\mathrm{req}}^{\mathrm{EV}}=0.80$$

(29)

The EV range-anxiety cost is defined as

$$c_t^{\mathrm{EV}}={\alpha }_3{\left({\displaystyle \frac{SO{C}_t^{\mathrm{EV}}-SO{C}_{\max }^{\mathrm{EV}}}{E_{\mathrm{cap}}^{\mathrm{EV}}}}\right)}^2$$

(30)

3.2.5. Grid Power Balance and Electricity Cost

The net power exchanged with the utility grid is expressed as

$$P_t^{\mathrm{grid}}=P_t^{\mathrm{load}}-P_t^{\mathrm{PV}}$$

(31)

where the aggregate residential load is

$$P_t^{\mathrm{load}}=P_t^{\text{non-shiftable}}+P_t^{\mathrm{shiftable}}+P_t^{\mathrm{controllable}}$$

(32)

When ESS charging and discharging are explicitly included, the complete grid power balance becomes

$$P_t^{\mathrm{grid}}=P_t^{\mathrm{load}}+P_{\mathrm{ch},t}-P_{\mathrm{dis},t}-P_t^{\mathrm{PV}}$$

(33)

The instantaneous electricity cost is calculated using an inclined block rate:

$$C_t^E=\left\{\begin{array}{cc}{P}_t^{\mathrm{grid}}{\rho }_t\Delta t,\hfill & 0\le P_t^{\mathrm{grid}}\le P_{\max }^{\mathrm{grid}},\hfill \\ P_t^{\mathrm{grid}}\xi {\rho }_t\Delta t,\hfill & P_t^{\mathrm{grid}}>P_{\max }^{\mathrm{grid}},\hfill \\ 0.9P_t^{\mathrm{grid}}{\rho }_t\Delta t,\hfill & P_t^{\mathrm{grid}}<0\hfill \end{array}\right.$$

(34)

Here, $P_{\max }^{\mathrm{grid}}=6\mathrm{kW}$ is the maximum grid-import threshold, $\xi =1.4423$ is the inclined block-rate factor, and ${\rho }_t$ denotes the real-time electricity price. A negative value of $P_t^{\mathrm{grid}}$ represents the export of surplus PV energy to the utility grid.

3.3. Action Space Formulation

The intake of control choices that the PPO agent may make to regulate home energy use at each time step is defined by the action space. Decisions about power distribution for controlled devices and appliance scheduling are also included. Equation (35) forms the action vector.

$$A_t=\left\{u_t^{\mathrm{sh}},P_t^{\mathrm{HVAC}},P_t^{\mathrm{EWH}},P_t^{\mathrm{EV}},P_t^{\mathrm{ESS}}\right\}$$

(35)

where $u_t^{\mathrm{sh}}$ represents the scheduling decision for shiftable appliances at time t, such as washing machines, dishwashers, or dryers. The term $P_t^{\mathrm{HVAC}}$ denotes the power control action of the heating, ventilation, and air-conditioning system, while $P_t^{\mathrm{EWH}}$ represents the power setting of the electric water heater. Similarly, $P_t^{\mathrm{EV}}$ indicates the charging or discharging decision of the electric vehicle, and $P_t^{\mathrm{ESS}}$ denotes the charging or discharging action of the energy storage system.

For the future generation state space, both the current operating variables and look-ahead scheduling information are included. For each time step t, the augmented state vector is defined as at t:

$$S_t=\left\{P_t^{\mathrm{price}},L_t^{\mathrm{ns}},L_t^{\mathrm{sh}},L_t^{\mathrm{ctrl}},SO{C}_t^{\mathrm{ESS}},P_t^{\mathrm{PV}},T_t^{\mathrm{in}},P_{t:t+H}^{\mathrm{price}},{\widehat{P}}_{t:t+H}^{\mathrm{PV}},W_t^{\mathrm{app}},D_t^{\mathrm{app}}\right\},$$

(36)

where $P_t^{\mathrm{price}}$ is the current electricity price, $L_t^{\mathrm{ns}}$, $L_t^{\mathrm{sh}}$, and $L_t^{\mathrm{ctrl}}$ represent the non-shiftable, shiftable, and controllable load states, respectively. $SO{C}_t^{\mathrm{ESS}}$ denotes the battery state of charge, $P_t^{\mathrm{PV}}$ represents the current photovoltaic generation, and $T_t^{\mathrm{in}}$ denotes the indoor temperature.

The term $P_{t:t+H}^{\mathrm{price}}$ represents future electricity price information over the look-ahead horizon H, while ${\widehat{P}}_{t:t+H}^{\mathrm{PV}}$ represents the predicted photovoltaic generation or irradiance information over the same horizon. The term $W_t^{\mathrm{app}}$ represents appliance operating windows, and $D_t^{\mathrm{app}}$ represents the remaining operation time and deadline information for shiftable appliances and EV charging.

This augmented state enables the PPO agent to identify low-price periods, PV intervals, and appliance deadline constraints before selecting control actions. As a result, the controller can schedule shiftable appliances before their deadlines, charge the ESS during low-price or surplus-PV intervals, discharge the ESS during high-price intervals, and coordinate EV charging within its availability window.

3.4. Hybrid Action Space Implementation

The proposed home energy management system (HEMS) requires both discrete and continuous control decisions. Therefore, the proximal policy optimization (PPO) policy is implemented using a hybrid action structure. A single actor–critic PPO framework is used, in which the actor network consists of shared hidden layers followed by two separate output branches. The first branch generates discrete scheduling decisions for shiftable appliances, while the second branch generates continuous power-control actions for controllable devices. The critic network estimates the state value $V\left(s_t\right)$ and evaluates the complete hybrid action.

The continuous control vector is expressed as

$$a_t^c=\left[P_{\mathrm{HVAC},t},P_{\mathrm{EWH},t},P_{\mathrm{EV},t},P_{\mathrm{ESS},t}\right],$$

(37)

where $P_{\mathrm{HVAC},t}$, $P_{\mathrm{EWH},t}$, $P_{\mathrm{EV},t}$, and $P_{\mathrm{ESS},t}$ represent the power-control decisions of the HVAC system, electric water heater, electric vehicle charger, and battery energy storage system, respectively.

The sampled continuous actions are then scaled according to the physical operating limits of each device before being applied to the environment as

$$P_{j,t}=\mathrm{clip}\left(P_{j,t},P_j^{\min },P_j^{\max }\right).$$

(38)

The joint hybrid policy is written as

$${\pi }_{\theta }(a_t\mid s_t)={\pi }_{\theta }^d(a_t^d\mid s_t){\pi }_{\theta }^c(a_t^c\mid s_t),$$

(39)

where ${\pi }_{\theta }^d$ is the discrete scheduling policy and ${\pi }_{\theta }^c$ is the continuous power-control policy.

The total log probability used during PPO training is computed as the sum of the discrete and continuous log probabilities as follows:

$$\log {\pi }_{\theta }(a_t\mid s_t)=\log {\pi }_{\theta }^d(a_t^d\mid s_t)+\log {\pi }_{\theta }^c(a_t^c\mid s_t).$$

(40)

Using this total log probability, the PPO probability ratio is calculated as

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t\mid s_t)}}.$$

(41)

The clipped PPO surrogate objective for the hybrid action policy is expressed as

$$L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_t\left[\min \left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right],$$

(42)

where ${\widehat{A}}_t$ is the estimated advantage function and $ϵ$ is the PPO clipping parameter.

Thus, the PPO optimizes appliance scheduling and power-control decisions within a single learning framework. The hybrid action implementation enables the proposed HEMS to handle practical residential energy management, where binary ON/OFF scheduling and continuous power allocation must be performed simultaneously.

3.5. Parameter Selection and Tuning

The simulator and controller parameters were selected using a combination of data-driven parameterization, residential device ratings, and preliminary sensitivity trials. PV generation and weather-related inputs were obtained from the solar/meteorological dataset, whereas device parameters such as ESS capacity, EV battery capacity, maximum charging/discharging power, HVAC rated power, water-heater power, SOC limits, and comfort temperature bands were selected according to typical residential operating ranges.

The reward-weighting parameters were tuned empirically through preliminary simulation trials. The electricity cost term was considered the primary objective with an implicit unit weight. The peak-demand penalty weight was set as ${\lambda }_1=0.40$, and the comfort-violation penalty weight was set as ${\lambda }_2=2$. These values were chosen because they provided stable training performance while maintaining a reasonable trade-off among energy cost reduction, peak-demand mitigation, EV charging satisfaction, and user comfort preservation. The PPO hyperparameter values used in the proposed HEMS are presented in Table 2.

Table 2. PPO hyperparameters used in the proposed HEMS.

Algorithm	Hyperparameter	Value
PPO	Learning rate	$3\times {10}^{-4}$
	Discount factor, $\gamma$	$0.99$
	GAE coefficient	$0.95$
	Clipping parameter	$0.20$
	Network architecture	$3\times 128$

3.6. Reward Function Design

The reward function is formulated to guide the PPO agent toward cost-effective, energy-efficient, and comfort-aware scheduling decisions. At each time step t, the reward penalizes three main factors: electricity cost, peak demand, and user comfort violation. In Equation (43), the reward is defined as

$$R_t=-\left(C_t^{\mathrm{grid}}+{\lambda }_1{P}_t^{\mathrm{peak}}+{\lambda }_2{P}_t^{\mathrm{comfort}}\right)$$

(43)

where $R_t$ represents the reward at time step t, and $C_t^{\mathrm{grid}}$ denotes the cost of electricity purchased from the utility grid. The term $P_t^{\mathrm{peak}}$ represents the penalty associated with excessive power demand, while $P_t^{\mathrm{comfort}}$ denotes the penalty imposed when user comfort limits are violated, such as deviations from the desired indoor temperature. The parameters ${\lambda }_1$ and ${\lambda }_2$ are weighting coefficients used to adjust the relative importance of peak demand reduction and comfort preservation.

Major home appliances like the HVAC system, electric water heater, electric car charger, and energy storage system are included in the grid electricity bill. The PPO agent learns to schedule appliances and distribute energy in a way that optimizes power costs while meeting operational and comfort restrictions by taking these costs into account.

The objective of the PPO agent is to maximize the expected cumulative discounted reward over the entire scheduling horizon. This objective function in Equation (44) is

$$J\left(\pi \right)=\mathbb{E}\left[{\displaystyle \sum _{t=0}^{T_{\mathrm{end}}}}{\gamma }^t{R}_t\right]$$

(44)

where $J\left(\pi \right)$ denotes the performance objective of policy $\pi$, $\gamma$ is the discount factor, and $T_{\mathrm{end}}$ represents the final time step of the scheduling period. The discount factor determines the importance of future rewards compared with immediate rewards. Better energy management performance, such as lower power costs, lower peak demand, and enhanced occupant comfort, is indicated by a bigger cumulative incentive.

Figure 3 shows the operating process of the suggested PPO-based smart-home energy management system. Within this framework, the agent monitors the condition of the smart home, chooses appropriate control actions, gets rewarded according to the effectiveness of those activities, and modifies its policy as necessary. The agent progressively picks up an efficient scheduling technique for domestic energy management through frequent contact with the environment.

Figure 3. Flowchart of the PPO-based optimization.

4. Results and Discussion

This section explains the way the suggested PPO-based home energy management system was evaluated using simulation. The proposed HEMS requires to lower power costs, enhance energy efficiency, and preserve occupant comfort in a smart residential system. Real-time monitoring, adaptive appliance control, and dynamic load scheduling are all carried out by the system to accomplish these goals. MATLAB is used in order to generate the suggested control framework, which is then compared to the two deep reinforcement learning algorithms, DDPG and TRPO.

To ensure a rigorous comparison among the three RL controllers, identical PV generation, electricity price, ambient temperature, base load, appliance windows, EV availability, and initial ESS/EV SOC values were used. The environment dynamics are ESS and EV SOC update, HVAC and EWH thermal models, appliance constraints, and the grid power balance. The control actions resulting in differences in cost, PAR, comfort violation, and SOC trajectories reflect policy quality.

In the proposed reward function, the weighting parameters control the trade-off between energy-cost minimization, peak-demand reduction, and user-comfort preservation. In this study, the peak-demand penalty weight is set as ${\lambda }_1=0.4$, the comfort-violation penalty weight is set as ${\lambda }_2=2$, the learning rate is 0.0003, the discount factor is $\gamma =0.99$, the generalized advantage estimation parameter is ${\lambda }_{\mathrm{GAE}}=0.95$, and the PPO clipping parameter is ${ϵ}_{\mathrm{clip}}=0.20$. The controller maintains the indoor temperature close to the desired comfort range while responding to variations in external conditions and household energy requirements. This confirms that the proposed energy management strategy can regulate HVAC operation effectively without causing significant discomfort to the occupants.

Figure 4 demonstrates the coordinated power exchange among PV generation, household load, grid import, and export in the HEMS. During day-time hours, PV generation supports local demand and reduces grid dependency, whereas grid import increases during low-solar periods such as morning and evening and surplus generation is exported to the grid through optimized energy scheduling.

Figure 4. PV generation and grid power exchange profiles.

Figure 5 shows the relationship between the real-time electricity price signal and the scheduled operation of controllable household appliances. Figure 5a presents the hourly electricity price variation over a 24 h scheduling horizon. The price is relatively low during the early morning period, decreases to its minimum value during off-peak hours, and increases significantly during the evening peak period. This price signal provides the economic input for the demand-side management controller. Figure 5b illustrates the optimized power consumption schedule of the dishwasher, washing machine, and cloth dryer. The scheduled appliance operation indicates how controllable loads are allocated within the available operating windows according to the electricity price signal and user constraints. This coordinated scheduling helps reduce unnecessary energy consumption during expensive periods and supports efficient residential energy management.

Figure 5. Electricity price signal and appliance power consumption. (a) Real-time electricity price signal; (b) Scheduled appliance power consumption.

The proposed method considers a 24 h scheduling horizon with 15 min control intervals. The control action is updated in a receding manner based on the system measurements, allowing the system to respond adaptively to dynamic operating conditions, starting at 8:00 a.m. and finishing at 8:00 a.m. the next day. Each of the 96 equal time segments represents 15 min. The HEMS records the present situation of the household at the start of each time step, including electricity price, photovoltaic generation, battery condition, and appliance operating status. The controller chooses a suitable scheduling action based on these data in order to minimize the overall daily energy consumption while meeting operational and comfort requirements.

The inclined block rate factor $\xi$ is set to $1.4423$, and the maximum grid power limit is fixed at $6\mathrm{kW}$. To represent realistic occupant behavior, appliance usage patterns are generated using a truncated normal distribution, denoted as $TN(\mu ,\sigma ,\min ,\max )$. The distribution randomly determines appliance start and end times within predefined practical limits. The photovoltaic system parameters are selected as ${\rho }_{\mathrm{pv}}=0.2$ and $A_{\mathrm{pv}}=15$. In addition, specific operating parameters are defined for the HVAC system, electric water heater, electric vehicle, and battery energy storage system.

Separate policy and value networks are used in the design of the PPO agent. Three hidden layers, each having 128 neurons, make up each network. The Tanh activation function is used by the policy network output layer to produce bounded control actions, whereas the ReLU activation function is used by the hidden layers. Practical time-varying tariff conditions are represented using hourly real-time energy pricing data from the National Renewable Energy Laboratory. Solar power, shifting appliances, controlled household loads, battery storage, electric car charging, dynamic electricity prices, and grid interaction are all included in the simulation environment.

The DRL agent is trained to minimize the cost of electricity, preserve customer comfort, manage peak demand, and optimize the utilization of renewable energy. Lastly, to show how the suggested PPO-based HEMS performs for intelligent domestic energy management, its performance is compared with DDPG and TRPO. The performance of the HVAC system, one of the primary controllable appliances in the suggested energy management framework is considered. The user-defined temperature setpoint, external temperature, and inside temperature are considered. Under varying outdoor temperature conditions, the controller successfully maintains the indoor temperature within the required comfort range throughout the day. Figure 6 shows the HVAC indoor temperature response obtained using the proposed PPO-based home energy management system.

Figure 6. HVAC indoor temperature response.

The indoor temperature remains within approximately ±0.8 °C of the desired setpoint, despite outdoor temperature variations of more than 6 °C. This illustrates how the PPO-based controller can maintain thermal comfort in a changing environment. Furthermore, the pre-heating behavior seen during low-tariff periods demonstrates the agent’s ability to predict future comfort needs and plan HVAC operation in an economical way. Figure 5 presents the scheduled power consumption of household appliances obtained using the proposed PPO-based home energy management system. The result shows how the controller coordinates the operation of flexible and controllable appliances over the scheduling horizon. This coordinated scheduling strategy helps to smooth the household load profile and reduce peak demand. It also supports cost-effective energy utilization by shifting flexible appliance operation toward flexible time periods. Therefore, the proposed PPO-based HEMS can improve residential energy efficiency, reduce unnecessary grid stress, and maintain reliable smart-home operation without significantly affecting user convenience. The coordinated EV charging strategy improves the economic performance of the smart home without compromising user mobility requirements. By considering electricity price, household load demand, battery condition, photovoltaic generation, and grid power limits, the PPO agent learns an adaptive charging policy. As a result, the proposed framework supports cost-effective EV integration, reduces peak demand, and enhances the flexibility of residential energy management in smart-grid applications. EV charging is optimized by considering the electricity price, EV availability, required SOC, charging limits, PV generation, and peak-load constraints. Therefore, the EV may not always charge exactly at the lowest price period, but the selected charging schedule satisfies user energy requirements while minimizing the overall operating cost. Figure 7 describes the coordinated response of EV charging and ESS operation with respect to the real-time electricity price signal. The EV charging profile is not determined only by the lowest electricity price; rather, it is optimized by considering the dynamic price signal, EV energy requirement, charging power limit, and allowable user-defined charging window. Therefore, the controller schedules EV charging during suitable periods while ensuring that the required battery energy is satisfied before the departure time. Similarly, the ESS response shows the charging and discharging behavior of the battery energy storage system according to price variation and system demand conditions. The ESS is charged during economically favorable periods or when surplus energy is available, and it is discharged during high-price or high-demand intervals to reduce grid import and operating cost. This coordinated operation demonstrates that the proposed PPO controller can manage EV charging and ESS power exchange by considering electricity price, load demand, storage constraints, and user requirements.

Figure 7. Coordinated EV charging and ESS power response with respect to the electricity price signal. (a) EV charging power response; (b) ESS charging and discharging response.

Figure 8 shows the impact of the household load profile. The results indicate that the optimized scheduling strategy reduces simultaneous appliance operation and shifts flexible demand to suitable time intervals. As a result, the peak load is reduced, the load curve becomes smoother, and the overall energy management performance is improved.

Figure 8. Comparison of household load profile.

Figure 9 shows the convergence performance of PPO, TRPO, and DDPG in terms of episode reward over training timesteps. The reward curves fluctuate during training because of the dynamic nature of the HEMS environment, including variations in load demand, renewable generation, storage operation, and exploration during policy learning. Among the compared algorithms, PPO shows comparatively stable reward behavior, indicating better learning consistency for the proposed energy management problem. TRPO also provides competitive performance, while DDPG exhibits larger reward variations due to its sensitivity to continuous-action exploration. Overall, the convergence comparison shows the suitability of PPO for stable policy learning in residential HEMS optimization.

Figure 9. Training convergence comparison.

This result confirms the learning stability and effectiveness of the proposed PPO-based energy management strategy and indicates that the suggested PPO-based controller is appropriate for real-time home energy management, as it learns consistently, maintains steady performance, and successfully adjusts to changing domestic energy conditions. This steady convergence reflects the PPO agent’s effective exploration, as shown in Table 2. DDPG and TRPO also provide feasible control performance. However, their reward values, peak demand, and convergence behavior are comparatively less stable than PPO. Therefore, the results show that PPO offers better convergence reliability and operational stability for residential smart-home energy management. Table 3 presents the overall performance comparison of PPO, DDPG, and TRPO.

Table 3. Overall performance comparison of PPO, DDPG, and TRPO.

Algorithm	Mean Reward	Average Daily Cost ($)	Peak Demand (kW)	Convergence Performance	Stability
PPO	$-\mathbf{3386.20}$	$\mathbf{1.062}$	$\mathbf{5.89}$	Best convergence	Best stability
DDPG	$-3573.60$	$1.158$	$6.21$	Moderate	More fluctuating
TRPO	$-3588.47$	$1.114$	$6.08$	Moderate	Moderate

The simulation results indicate that the proposed PPO-based HEMS outperforms DDPG and TRPO across multiple metrics, including cost reduction and peak demand. By leveraging stable policy learning and efficient exploration, the PPO agent effectively coordinates flexible loads, optimizes energy storage utilization, and maintains precise temperature control. These outcomes confirm the suitability of PPO for practical smart home energy management applications in environments characterized by dynamic pricing and stochastic user behavior.

5. Conclusions

For intelligent residential energy control, this study creates a home energy management system based on proximal policy optimization. The primary goals are to reduce power costs, improve energy efficiency, and preserve user comfort. The suggested PPO controller was implemented and evaluated using deep reinforcement learning benchmark algorithms, including Trust Region Policy Optimization and Deep Deterministic Policy Gradient. In this paper, a deep reinforcement learning approach based on PPO for residential energy management in smart-grid environments was presented. The proposed PPO-based HEMS demonstrates improved residential energy management by coordinating PV generation, ESS operation, EV charging, HVAC control, and shiftable appliance scheduling. The results show that the proposed controller reduces the electricity cost and peak demand while maintaining acceptable comfort constraints. However, the contribution should be interpreted beyond a simple comparison with fixed scheduling, since many existing EMS studies also perform optimized load scheduling. The strength of the proposed method lies in its model-free adaptive control structure, its ability to handle both storage and flexible-load decisions, and its fast decision-making after the training process.

The proposed framework has limitations in controller performance that depend on the reward weight and hyperparameter selection, and the present validation is simulation-based. Future work will extend the comparison to robust optimization and multi-agent DRL methods, together with seasonal datasets, multiple household profiles, and hardware-in-the-loop validation.