Environmentally Sustainable HVAC Management in Smart Buildings Using a Reinforcement Learning Framework SACEM

Abstract

Heating, ventilation, and air-conditioning (HVAC) systems dominate energy consumption in hot-climate buildings, where maintaining occupant comfort under extreme outdoor conditions remains a critical challenge, particularly under emerging time-of-use (TOU) electricity pricing schemes. While deep reinforcement learning (DRL) has shown promise for adaptive HVAC control, existing approaches often suffer from comfort violations, myopic decision making, and limited robustness to uncertainty. This paper proposes a comfort-first hybrid control framework that integrates Soft Actor–Critic (SAC) with a Cross-Entropy Method (CEM) refinement layer, referred to as SACEM. The framework combines data-efficient off-policy learning with short-horizon predictive optimization and safety-aware action projection to explicitly prioritize thermal comfort while minimizing energy use, operating cost, and peak demand. The control problem is formulated as a Markov Decision Process using a simplified thermal model representative of commercial buildings in hot desert climates. The proposed approach is evaluated through extensive simulation using Saudi Arabian summer weather conditions, realistic occupancy patterns, and a three-tier TOU electricity tariff. Performance is assessed against state-of-the-art baselines, including PPO, TD3, and standard SAC, using comfort, energy, cost, and peak demand metrics, complemented by ablation and disturbance-based stress tests. Results show that SACEM achieves a comfort score of 95.8%, while reducing energy consumption and operating cost by approximately 21% relative to the strongest baseline. The findings demonstrate that integrating comfort-dominant reward design with decision-time look-ahead yields robust, economically viable HVAC control suitable for deployment in hot-climate smart buildings.

1. Introduction

Space cooling is the leading type of energy consumed in the Gulf region. Extremely hot and humid summers are typical of the Gulf climate; thus, mechanical cooling is almost everywhere. High-rise building studies in desert climates document space cooling as having the ability to correspond to up to 70% of total energy usage [1,2]. This seems to be supported by similar regional reports: in the Middle East and North Africa (MENA), air-conditioning utilization is estimated by about 70% of residential and commercial electricity consumption [3,4], and surveys of commercial-sector electricity demand in Saudi Arabia and the UAE show that 60–70% of peak electricity demand is taken by cooling systems alone [5]. The buildings sector accounted for approximately 34% of all energy demand and generated 34% of all energy-related CO₂ emissions in 2023 on a global scale [6]. This reliance on cooling highlights the importance of energy use in buildings, making it a policy issue of central concern.

The need to overcome this challenge has been worsened by rapid urbanization and economic growth in the Gulf. The official estimates indicate that the industry sector of Saudi Arabia consumes 47% of primary energy, followed by the buildings sector, 28%, and transport, 20% [7]. The pricing of domestic energy is low, and the climate is dry, with high ambient temperatures, which stimulates demand in air-conditioning and water desalination. In addition, energy efficiency performance is not optimal because energy is under-priced, and therefore, it promotes too much consumption [7]. Saudi Vision 2030 is designed to change this trend, diversifying the economy, reducing emissions, and reaching 50% of renewable power generation capacity by 2030 [8]. One of the pillars of Vision 2030 is a more sustainable built environment; initiatives, including the Saudi Energy Efficiency Program (SEEP) and Saudi Energy Efficiency Center (SEEC), have been established to implement building codes, establish energy-labeling of air conditioners and insulation materials, and to retrofit government buildings through the national energy services company TARSHID [9,10].

These policies complement green-building initiatives. For example, the Sustainable Building Program has developed a national sustainability rating system, assessing over 70 projects covering more than 50 million m² by 2023 [11]. Studies estimate that conventional buildings consume about 40% of global energy and produce 30% of carbon emissions, whereas green buildings can cut energy use by nearly half and reduce annual operating costs by roughly 30% [12]. Rating systems in neighboring GCC states, such as Abu Dhabi’s Pearl Rating System and Qatar’s Global Sustainability Assessment System (GSAS), show that mandatory green standards for new buildings can reduce energy demand by 70% and achieve 30–50% energy savings in existing buildings [13,14].

Beyond policy, occupant behavior and weather strongly influence energy use. Empirical analyses reveal that higher occupancy rates increase energy consumption because of additional lighting, electronics, and HVAC loads, while low occupancy leads to energy waste [15]. Weather variables such as outdoor temperature, humidity, and solar radiation also contribute significantly to consumption dynamics [16]. Consequently, occupancy-based control strategies that adjust HVAC operations according to occupancy patterns have demonstrated improved energy efficiency and comfort [17]. Yet implementing such strategies in the Gulf’s harsh climate is complex because occupant schedules, thermal preferences, and equipment performance vary widely [18].

Last but not least, electricity pricing is complex. Time-of-use (TOU) tariffs are prevalent in most countries, but in the Gulf, they are restricted. According to a Chatham House analysis, proposed TOU and interruptible tariffs in Saudi Arabia and Oman are mostly voluntary and are aimed primarily at industrial consumers [19]; voluntary tariffs are also found in Abu Dhabi [20]. There are, therefore, very limited price incentives to consumers, particularly within residential and commercial sectors, to decrease peak demand. It is still in its infancy because dynamic pricing is still being experimented with, as recent electricity price reforms in Saudi Arabia attempt to publicize real costs and phase out subsidies gradually [21].

Heating, ventilation, and air-conditioning (HVAC) systems become dominant building loads in hot climates, and this is difficult to optimize. The control problem is multi-objective: the agent should pay attention to the indoor thermal comfort, minimize the energy use, and consider the operating costs, including the tariffs that could depend on the time of day [22,23]. The balance is challenging to achieve due to the subjective sense of comfort depending on occupant metabolism, clothing, and activity levels and energy consumption being a non-linear response to outdoor temperature, humidity, solar radiation, and equipment dynamics [24]. Moreover, the Gulf summer has excessive outdoor temperatures that place a strain on HVAC systems. The traditional rule-based controllers usually scale up the systems or run them all the time, resulting in poor energy utilization and heating inconveniences [25].

The environment of the Gulf is very dynamic. Outdoor temperatures are unpredictable, reaching highs of exceeding 50 °C, and humidity varies with sea breezes as well as dust storms due to solar radiation [26]. The occupancy changes on an hourly and seasonal basis depending on the work schedules, holidays, and economic activity in the commercial buildings. These aspects lead to adaptive control methods that are capable of learning based on past data and real-time sensor values. The efficiency of learning algorithms is, however, a serious limiting factor; learning algorithms based on reinforcement learning (RL) mechanisms often need large data sets and may fail to perform well when the conditions vary [27,28,29]. Recent results on model-based RL indicate that precise system models and data-efficient learning methods can bring the training data requirement down to just seven days and also cut comfort violations by 12 percent [30].

Introducing TOU pricing could encourage demand-side flexibility by shifting cooling loads to off-peak hours. However, current tariffs in Saudi Arabia rarely differentiate between peak and off-peak periods. The voluntary TOU tariffs described earlier focus on large industrial consumers. For residential and commercial buildings, optimizing HVAC control under hypothetical TOU tariffs requires predictive models of price variation and occupant response. Multi-objective control must therefore balance comfort and energy use while minimizing costs under potentially dynamic tariffs [7,31].

Deep reinforcement learning (DRL) has recently been applied to HVAC control. DRL algorithms—such as Deep Q-Networks, Soft Actor–Critic (SAC), and Proximal Policy Optimization—can outperform reactive controllers, achieving significant energy savings [32]. Yet they suffer from several limitations, such as reward sensitivity; small changes in reward weightings can cause large changes in performance and hinder transferability across climates [33]. Moreover, sample inefficiency model-free algorithms require extensive exploration and long training periods [34]. Also, comfort is the compromise, as many studies prioritize energy minimization, resulting in uncomfortable indoor temperatures, system shutdown behaviors, or oscillations [30], and there is a lack of generalization, as policies trained for a particular climate or building often fail when deployed in different environments [33] Moreover, advanced control strategies such as Model Predictive Control (MPC) and RL can reduce costs and emissions by 15–25% relative to rule-based control [35], yet adoption remains limited due to unclear business cases and high deployment costs [35].

This paper proposes a Soft Actor–Critic with Cross-Entropy Method (SACEM) framework for HVAC control in smart buildings. In this work, resilience intelligence refers to the controller’s ability to maintain safe comfort under disturbances such as extreme heat spikes, forecast errors, and stochastic occupancy, while avoiding brittle behaviors such as oscillations, shutdown, or over-reliance on a single predictive signal. The proposed SACEM delivers resilience beyond incremental forecasting gains through structural innovations: (i) a two-layer decision architecture that couples a stochastic entropy-regularized policy (SAC) with a short-horizon action refinement optimizer (CEM) that explicitly evaluates candidate sequences under the thermal transition model; (ii) a comfort-dominant reward with strongly asymmetric penalties that makes comfort violations economically irrational for the agent (Equation (10)); and (iii) safety-aware projection during refinement that prevents unsafe candidate actions even during exploration (Equation (24)). These elements change the controller’s decision mechanism from purely reactive reward maximization to look-ahead, constraint-aware control, which is verifiable through ablation and stress testing (Section 3.7 and Section 4).

Despite these advances, the current literature lacks a unified, comfort-first control framework that jointly addresses short-horizon foresight, safety under uncertainty, and economic optimization in hot-climate buildings. In particular, few studies explicitly verify robustness against forecast errors, stochastic occupancy, and extreme heat while maintaining high comfort guarantees. This work addresses these gaps by positioning SACEM as a hybrid learning–optimization framework that bridges MPC-style predictive reasoning and DRL-based adaptability, with explicit comfort dominance and resilience verification.

The objective of this study is to develop a comfort-first intelligent HVAC control strategy that is suitable for hot-climate buildings operating under time-of-use electricity pricing. The motivation for this work arises from persistent limitations in existing HVAC control approaches, where energy or cost minimization is often achieved at the expense of occupant comfort, particularly under extreme thermal stress. These limitations are further exacerbated in hot desert climates, where cooling demand dominates and system mismanagement can rapidly lead to discomfort, peak-load stress, and economic inefficiency.

To address these challenges, this paper proposes a hybrid control framework that integrates Soft Actor–Critic (SAC) learning with a Cross-Entropy Method (CEM) refinement layer. The rationale behind this integration is to combine the adaptability and sample efficiency of off-policy deep reinforcement learning with short-horizon predictive optimization and safety-aware decision making. By explicitly prioritizing thermal comfort through asymmetric reward penalties and constraint-aware refinement, the proposed framework aims to prevent system shutdown behaviors and unstable control actions observed in prior DRL-based HVAC studies.

This study further aims to rigorously evaluate the proposed approach under realistic operating conditions representative of Saudi Arabian commercial buildings. This includes validation under extreme summer temperatures, stochastic occupancy patterns, and time-varying electricity tariffs. Through comparative evaluation against state-of-the-art baselines and targeted ablation and disturbance tests, this work seeks to demonstrate not only performance improvements in comfort, energy, and cost but also enhanced robustness and resilience to uncertainty.

The novelty of the present study is the idea of redefining the issue of HVAC control in buildings located in hot climates as a control problem dominant in comfort but with critical resilience requirements instead of an energy-minimization challenge. In contrast to the previous literature, which mainly maximizes energy or cost, with the exception that comfort is considered a soft or secondary goal, this paper presents a control paradigm where an asymmetric reward structure and safety conscience decision refinement are introduced to render comfort violations economically irrational.

The most important novelty is that the decision-time short-horizon optimization has been integrated into a data-efficient off-policy reinforcement learning controller. Although the literature either uses model-based MPC (with high modeling and deployment expenses) or model-free DRL (with myopic and unstable behavior), the suggested SACEM framework has the unique advantage of integrating the search of entropy-regularized policy learning with the Cross-Entropy Method (CEM) of action refinement with explicit safety constraints. This structural connection permits anticipatory constraint-conscious control without the large-scale predictive model or long time horizon plans.

This study is worth the knowledge since it shows that the high-level of comfort does not exclude economic efficiency, even in extreme thermal conditions, and the electricity tariffs vary with time. The findings are actionable evidence that optimizing hybrid, under hot climatic conditions, HVAC learning controllers can attain reliable, interpretable, and deployable HVAC controls, where cooling need, peak loads, and occupant comfort are of paramount significance to society and the economy.

Based on the identified research gaps and the proposed comfort-first hybrid control design, this study tests the following hypotheses:

Hypothesis 1.

A comfort-dominant hybrid reinforcement learning controller (SACEM) can achieve significantly higher thermal comfort levels than state-of-the-art DRL baselines (PPO, TD3, SAC) under extreme hot-climate conditions.

Hypothesis 2.

The integration of short-horizon decision-time refinement (CEM) with off-policy DRL enables simultaneous improvement in comfort and reduction in energy consumption and operating cost under time-of-use electricity pricing.

Hypothesis 3.

Safety-aware action refinement and asymmetric comfort penalties improve robustness to uncertainty, resulting in lower comfort violation rates under forecast errors, occupancy disturbances, and extreme heat events compared to model-free DRL controllers.

The remainder of this paper is structured as follows. Section 2 reviews related work on intelligent HVAC control, including model-predictive and reinforcement learning techniques. Section 3 presents the proposed SACEM methodology, detailing the learning framework, reward formulation, simulation environment, and experimental setup. Section 4 reports and discusses the results, comparing SACEM with baseline controllers across energy, cost, and comfort metrics. Finally, Section 5 concludes the paper and outlines directions for future research.

Heating, ventilation, and air-conditioning (HVAC) systems have captured much research because of their contribution to building heating and cooling energy use and comfort. The literature covers a wide range of methodologies, from traditional methods of control to the latest intelligent techniques. This section will review the current literature based on four broad themes, namely, HVAC control methods, deep reinforcement learning to control buildings, hybrid optimization frameworks, and the literature that has studied hot-climate conditions.

State-of-the-art HVAC control research has progressed along three dominant directions. First, model-based approaches such as Model Predictive Control (MPC) provide explicit look-ahead optimization and constraint handling but require accurate system models and incur high computational and deployment costs, limiting scalability in complex or uncertain environments. Second, model-free deep reinforcement learning (DRL) methods, including PPO, TD3, and Soft Actor–Critic (SAC), offer adaptive control without explicit modeling, yet frequently exhibit myopic behavior, reward sensitivity, and comfort degradation under extreme thermal conditions. Third, emerging hybrid approaches attempt to combine learning and optimization, but most remain either climate-agnostic, comfort-secondary, or insufficiently validated under hot-climate and time-varying tariff conditions.

2. State of the Art

The optimization of HVAC systems has been extensively studied using a wide range of control strategies. Traditional proportional–integral–derivative (PID) controllers, while simple and widely used, suffer from poor adaptability and weak anti-interference capabilities. To address these issues, a two-stage fuzzy neural network-enhanced PID control method was developed, achieving superior convergence, robustness, and overall control performance in industrial workshops [36]. Beyond PID, fuzzy logic controllers have been explored for thermal comfort management, with designs that expand input parameters and employ automatic fuzzy rule generation to achieve up to 50% energy savings without compromising comfort [37]. A further advance is the Weather-Adaptive Fuzzy Control (WAFC) strategy, which integrates external weather, electricity pricing, and occupancy data to dynamically adjust HVAC setpoints, resulting in significant annual energy savings and improved sustainability in urban buildings [38].

Another major line of research centers on Model Predictive Control (MPC), which leverages predictive modeling and optimization to manage building energy use under uncertainty. Several reviews have highlighted MPC’s capability to optimize energy efficiency, cost, and comfort while stressing the importance of parameters such as prediction horizons, cost functions, and modeling software [39,40]. Simulation campaigns across diverse U.S. climate zones demonstrated that MPC can yield substantial energy savings compared to rule-based baselines, though ignoring humidity in MPC formulations can degrade comfort in mild conditions [41]. Modified economic MPC frameworks have also been introduced to balance occupant comfort and electricity costs in multi-zone settings, incorporating thermal energy storage and dynamic pricing to reduce costs by nearly 19% [42]. Hybrid models that combine MPC with radial basis function neural networks further enhance predictive accuracy, achieving energy consumption reductions of around 15% in both residential and commercial buildings [43]. Complementing these works, a distributed energy–comfort management method based on Lyapunov optimization was proposed to flexibly balance energy use and thermal comfort across multiple regions, reducing costs while protecting user privacy [44].

A structured comparison of representative deep reinforcement learning approaches for HVAC control, including their application contexts and identified limitations, is summarized in

Table 1. Representative DRL studies for building HVAC control, extended with identified gaps and comparison to the proposed SACEM framework.

Study	Study Focus	DRL Algorithm	Building Type	Climate Conditions	Limitations/Research Gap
[45]	Real-building SAC for Thermally Activated Building Systems (TABS); simulation and real-world validation	SAC	TABS-equipped building	Two-month cooling season (implied hot)	Comfort not dominant; no TOU pricing or robustness analysis
[46]	Multi-agent HVAC optimization balancing energy, comfort, and air quality	MADDPG	Multi-zone office with fan coils and DOAS	Heating and cooling; EnergyPlus–Python co-simulation	No extreme hot-climate focus; limited resilience testing
[47]	Hybrid model-based DRL for HVAC control	DDPG (hybrid, model-based)	Multi-zone simulated building	Temperate; New South Wales, Australia	Not evaluated under hot climates or dynamic pricing
[48]	PPO with behavior cloning for school energy management	PPO (with behavior cloning)	School building; 21 zones	Hot/humid; Qatar	Single building type; no predictive foresight or safety constraints
[49]	Gnu-RL: differentiable MPC combined with RL; simulation and real-world tests	Differentiable MPC + RL	Conference room	Not stated; high cooling load	High modeling complexity; limited scalability and climate specificity
[50]	Holistic DRL (OCTOPUS) for multi-system building control	Not specified (DRL framework)	Commercial (LEED Gold)	Not stated; summer cooling load	Comfort treated as secondary objective; limited cost awareness
[51]	Entropy-regularized SAC variant for HVAC control	EntropySAC	Not stated	High-temperature conditions	No safety-aware refinement or TOU-aware optimization
[52]	Adaptive-reward SAC evaluated on BOPTEST	SAC	Residential (Belgium); office (Denver)	Cold (heating) and hot (cooling) cases	Limited TOU pricing and disturbance-based robustness evaluation
[53]	DRL targeting thermal comfort in controlled laboratory settings	DDPG	Laboratory; 307 m2	Hot/humid; Singapore	Laboratory-scale only; no economic or TOU optimization
This work	Comfort-first, cost-aware, and resilient HVAC control under TOU pricing	Hybrid SAC + CEM (SACEM)	Commercial (single-zone representative model)	Extreme hot desert climate (Saudi Arabia)	Explicit comfort dominance, and evaluated in a representative single-zone simulation

, highlighting gaps that motivate the proposed approach.

Climate change and extreme weather events present additional challenges for HVAC control. Simulation studies using extreme weather data sets project increased peak cooling demand and unmet cooling hours in commercial buildings, underscoring the importance of a future-proof HVAC system design that incorporates variability and extreme conditions [54]. Intelligent building systems have also been enabled by advancements in information and communication technologies, where MPC frameworks exploit abundant sensor data to improve adaptability and performance in real time [40]. Broader reviews comparing traditional, fuzzy, MPC, and machine learning-based approaches emphasize the growing potential of reinforcement learning (RL), which offers adaptive control under dynamic environments, though its validation remains limited in industrial contexts [55]. Finally, advances in building energy control increasingly combine traditional rule-based and PID approaches with modern techniques such as extremum seeking control and reinforcement learning, aiming to balance simplicity, robustness, and adaptability [36,43].

Recent studies are using deep reinforcement learning (DRL) more for building control since it can learn control policies without having to know precise plant models. On the one hand, buildings represent a major part of the world’s energy, and HVAC systems occupy more than half of this amount of energy consumption. The common rule-based or model-predictive controllers do not easily scale to complex multi-zone buildings and can be very laborious to model. DRL as an alternative has thus been explored as a potential alternative; however, the majority of research is performed on a single algorithm and single-zone output, with only some studies examining comparisons across methods. New appraisals and assessments will give the insights of policy-gradient and actor–critic methods, multi-agent systems, as well as the problem of reward engineering [32,56,57].

Proximal Policy Optimization (PPO) [58] is an on-policy policy-gradient algorithm that makes use of a clipped objective and which blocks excessively large policy updates. Recently, in a more thorough analysis of DRL applied to HVAC control, PPO was characterized as enhancing TRPO by introducing a clipped objective and lessening computational expense, and is not very data-efficient [32]. This paper contrasted PPO and TD3 and SAC, wherein PPO performed fairly but had slower convergence and poor off-policy performance in multi-objective tasks [32].

Asynchronous Advantage Actor–Critic (A3C) [59] and its synchronous counterpart A2C, parallelize a set of agents training local policies that a central agent plans. These on-policy approaches are easy and can utilize parallel environments, although in practice, they are generally less effective than more advanced policy-gradient methods such as PPO [32]. Recently, a building energy management (BEM-DRL) architecture with A3C and Bayesian optimization was proposed to optimize HVAC control policies and achieved savings in energy consumption of about 16.7 percent [60].

Twin-Delayed DDPG (TD3) [61] is an off-policy actor–critic algorithm that builds on DDPG and adds clipped double-Q-networks, delayed policy refinement, and target-policy smoothing to enhance stability [62,63]. In the analysis of HVAC evaluation, TD3 was proven to rapidly converge and achieve better stability compared to PPO, but it still used more energy than SAC in most cases [32]. The outcome of the training indicated that the TD3 had considerable energy savings and could be transferred to other climates, although there were occasions when it experienced action saturation [32].

Soft Actor–Critic (SAC) is an off-policy actor–critic algorithm that optimizes expected payoff and promotes exploration by means of entropy regularization [64]. The evaluation paper demonstrated that SAC performed well in a range of different climates, with early convergence and strong generalization; SAC saved more energy than PPO and TD3 and kept occupants comfortable [65]. Recent installations of multi-zone buildings show that SAC can save about 15% of HVAC energy use and maintain comfort. SAC has also been applied in real-world buildings. A study comparing offline-trained and online-trained SAC controllers to a model-predictive controller found that the offline SAC achieves performance comparable to MPC and that the value of data-efficient off-policy control is practical [66].

A single-agent DRL would be impractical, as state and action spaces are growing exponentially and require coordination on a multitude of zones to control the building. Multi-agent DRL (MADRL) de-allocates agents to zones or subsystems. The combination of electrochromic windows and multi-zone HVAC using MADRL was tested, and 19.8% of energy was saved and 40.3 vs. rule-based control and other MADRL methods by 1–5% of comfort violations and 48% across climates [67].

A study in [68] observed that multi-zone HVAC needs inter-agent coordination; approaches such as MADDPG and MAAC handle the issue of coupling well but do not scale beyond a few dozen zones. There is a need for a more scalable MADRL. In addition to HVAC, occupant-focused methods employ value-decomposition networks of per-zone agents.

Recent reinforcement learning-based energy management studies have increasingly focused on cyber-resilience and uncertainty-aware scheduling. A state-adversarial deep reinforcement learning framework was introduced to enhance cyber-resilience in integrated energy systems with demand response, explicitly modeling cyber attacks as adversaries within the decision process and achieving improved economic performance under attack scenarios [69]. Complementary work has explored robust multi-agent deep reinforcement learning for multi-carrier energy systems, demonstrating that coordinated DRL control combined with demand response and energy storage can significantly improve operational resilience and reduce system costs under cyber disturbances [70]. In parallel, comprehensive reviews highlight the growing role of deep learning in securing renewable energy supply chains, emphasizing anomaly detection, adversarial robustness, and the need for standardized benchmarks to ensure scalable and resilient deployment in real-world energy infrastructures [71].

Recent reinforcement learning and online optimization studies in edge and cyber–physical systems provide complementary insights into large-scale, latency-aware, and reliability-driven decision making. Proactive deployment in MEC systems leverages multi-dimensional prediction and two-timescale optimization to balance deployment cost, latency, and prediction uncertainty [72], while reliability-enhanced microservice deployment exploits Lyapunov-based online control to improve system stability under failures and resource coupling [73].

Beyond deployment, two-timescale optimization has also been applied to human digital twin construction and task execution under mobility, accuracy, and delay constraints [74], with recent surveys highlighting the role of generative AI and deep learning in strengthening cyber-resilience and observability in complex IoT and healthcare systems [75]. Similarly, joint service migration and task rerouting strategies in MEC demonstrate how online resource management can balance responsiveness and optimality under dynamic conditions [76]. In contrast to these works, SACEM focuses on real-time HVAC control under extreme climate conditions, emphasizing comfort-first safety, decision-time refinement, and robustness to operational uncertainty rather than service placement or task offloading.

Recent DRL studies on building control are leaving behind proof-of-concept studies in favor of comparative studies and multi-agent deployments. Off-policy actor–critic models like SAC and TD3 now provide the best trade-offs in energy consumption, comfort, and stabilization of occupants, and training. PPO and A3C are easier to implement, but tend to converge more slowly, and are sensitive to information efficiency. Multi-agent models combat the dimensionality curse of multi-zone buildings, but additional research would be required to extend to actual buildings with hundreds of zones [32,77]. Last but not least, the reward mechanism is also a key design factor: it is challenging to weight several objectives [3], and ill-defined rewards may result in sparse cues or unwanted behavior [27]. The pursuit of emerging solutions integrating multi-objective DRL, multi-stage reward models, and holistic comfort measures provides ways in which to continue to move towards a robust and adaptive building-control system [77].

The literature demonstrates that control methods based on rules, PID control, and MPC have been supplanted by off-policy actor–critic models, which in turn exhibit better sample efficiency, stability, and cross-climate generalization compared to on-policy counterparts, and that multi-agent models can be used to address, but not to fully solve, the sample efficiency, stability, and cross-climate generalization needs of large multi-zone buildings. The current gaps could be addressed through standardized, climate-sensitive benchmarks, principled multi-objective reward-shaping, avoiding a degenerate policy, and active tests in real buildings with heterogeneous end uses and changing tariffs. To rationalize these observations, Table 1 systematizes the state of the art in representative DRL studies by their focus or emphasis, choice of algorithm, building context, and climate setting, to help identify the areas in which evidence is strongest and in which open challenges persist, thus driving the rationalization of the need to develop the unified, comfort-first, and cost-aware approach as demonstrated by this work.

Positioning of This Work and Focal Contribution

The focal point of this study is comfort-first resilient HVAC control under extreme hot-climate operation with time-varying electricity prices. While prior HVAC control research has separately advanced (i) model-based methods such as MPC for look-ahead control, and (ii) model-free DRL methods such as SAC/TD3/PPO for adaptive policy learning, existing approaches commonly face one or more of the following gaps in hot climates: (a) comfort violations under high thermal stress due to reward trade-offs, (b) myopic decisions that ignore short-horizon future consequences, and (c) limited robustness to forecast errors and stochastic occupancy.

Relative to these efforts, the contribution of this paper is a unified SACEM controller that explicitly couples the following: (1) an entropy-regularized SAC policy for data-efficient continuous control, (2) a decision-time CEM sequence refinement layer for short-horizon look-ahead using the thermal transition model, and (3) a safety-aware projection with asymmetric comfort-dominant penalties that prevent unsafe candidate actions. This combination positions SACEM between explicit look-ahead, constraint-aware (MPC), and purely model-free DRL adaptive learning, achieving sample efficiency while being evaluated under Saudi-specific climate, occupancy, and TOU settings, with ablations and disturbance tests to verify resilience improvements.

3. Methodology

3.1. Scope, Problem Definition, and Modeling Rationale

This section formalizes the HVAC control problem addressed in this study and motivates the associated modeling choices. We consider a commercial building operating in a hot desert climate, where cooling dominates energy demand and comfort violations carry high operational and social costs. The control objective is to regulate HVAC power in real time to (i) maintain indoor thermal comfort under extreme outdoor temperatures, (ii) minimize energy use and electricity cost under time-varying tariffs, and (iii) ensure stable and safe operation under uncertainty in weather forecasts and occupancy.

To enable learning-based control while preserving interpretability and computational tractability, the building–HVAC system is modeled as a discrete-time Markov Decision Process (MDP) with a low-order thermal dynamics model. This formulation follows common practice in reinforcement learning-based HVAC control, where simplified resistance–capacitance (RC) representations are used to approximate dominant heat transfer mechanisms while remaining suitable for large-scale simulation and policy learning. The proposed SACEM framework is then introduced to address limitations of purely model-free approaches by integrating decision-time look-ahead and safety-aware refinement on top of an off-policy deep reinforcement learning baseline.

3.2. Problem Formulation

3.2.1. HVAC System Modeling

The HVAC control issue is modeled as a Markov Decision Process (MDP) that involves a one-zone thermal dynamics model derived from simplified resistance–capacitance (RC) formulations commonly used in building energy modeling and reinforcement learning-based HVAC control studies [30,39,41].

$$T_{in}(t+1)=T_{in}\left(t\right)+U_{loss}·\left(T_{out}\left(t\right)-T_{in}\left(t\right)\right)-{\displaystyle \frac{P_{HVAC}\left(t\right)}{C_{th}}}+O_{gain}·\mathrm{Occ}\left(t\right)$$

(1)

where $T_{in}\left(t\right)$ and $T_{out}\left(t\right)$ represent indoor and outdoor temperatures at time t (°C), respectively; $U_{loss}$ denotes the normalized heat loss coefficient accounting for building envelope properties; $P_{HVAC}\left(t\right)$ is the HVAC power consumption (kW); $C_{th}$ represents the building’s thermal capacity (kW/°C); $O_{gain}$ quantifies occupancy-induced heat gains; and $\mathrm{Occ}\left(t\right)\in \{0,1\}$ indicates occupancy status.

This formulation follows simplified resistance–capacitance (RC) thermal models commonly used in building energy simulation and reinforcement learning-based HVAC control to capture dominant heat transfer dynamics with low computational overhead [39,41].

The system parameters are subject to domain randomization during training to enhance robustness:

$$\begin{array}{cc}\hfill U_{loss}& \sim \mathcal{N}(0.10,{0.01}^2),U_{loss}\in [0.08,0.12]\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill C_{th}& \sim \mathcal{N}(3.0,{0.2}^2),C_{th}\in [2.5,3.5]\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill O_{gain}& \sim \mathcal{N}(0.25,{0.02}^2),O_{gain}\in [0.2,0.3]\hfill \end{array}$$

(4)

The control resolution is set to 15 min intervals, yielding 96 control decisions per day, which balances computational efficiency with adequate temporal granularity for HVAC control.

Similar simplified thermal models have been widely used in reinforcement learning-based HVAC control to balance physical fidelity and computational efficiency, particularly in simulation-driven controller evaluation [32,39,41].

3.2.2. Saudi Arabian Operation

The methodology takes into consideration three crucial issues of the Saudi Arabian building environment that make this work stand out among the generic studies of HVAC control.

Climate Characteristics

The extreme desert climate of Riyadh presents unique challenges with outdoor temperatures ranging from 28 °C to 44 °C during summer months, accompanied by significant diurnal variations exceeding 16 °C. This thermal environment necessitates substantial cooling loads during daytime hours while offering potential for natural cooling during nighttime periods.

Time-of-Use Electricity Pricing Structure

Saudi Arabia’s three-tier pricing scheme creates strong economic incentives for load shifting:

$${\pi }_{\mathrm{electricity}}\left(t\right)=\left\{\begin{array}{cc}0.19\mathrm{SAR}/\mathrm{kWh},\hfill & t\in [0,11)\cup [20,24)\left(\mathrm{Off}\text{-}\mathrm{peak}\right)\hfill \\ 0.32\mathrm{SAR}/\mathrm{kWh},\hfill & t\in [11,13)\cup [17,20)\left(\mathrm{Shoulder}\right)\hfill \\ 0.76\mathrm{SAR}/\mathrm{kWh},\hfill & t\in [13,17)\left(\mathrm{Peak}\right)\hfill \end{array}\right.$$

(5)

The peak-to-off-peak price ratio of 4:1 provides substantial economic motivation for intelligent load management, significantly exceeding typical ratios of 2:1 to 3:1 found in other regions.

Commercial Occupancy Patterns

The occupancy model reflects typical Saudi commercial building usage with high probability during business hours:

$$P(\mathrm{Occ}\left(t\right)=1)=\left\{\begin{array}{cc}0.92\hfill & \mathrm{if}\mathrm{weekday}\mathrm{and}t\in [8,18)\hfill \\ 0.05\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(6)

Probabilistic occupancy representations of this form are widely adopted in building control studies to emulate stochastic presence patterns while remaining tractable for control-oriented simulation [15,17].

3.3. SACEM Framework Architecture

3.3.1. Markov Decision Process Formulation

The HVAC control problem is cast as a Markov Decision Process (MDP) defined by the tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma )$, following standard reinforcement learning formulations for continuous HVAC control [32,64], where

State Space $\mathcal{S}\subset {\mathbb{R}}^{13}$

The state vector incorporates both immediate system conditions and predictive information:

$$s_t=[T_{in}\left(t\right),T_{out}\left(t\right),\mathrm{Occ}\left(t\right),sin\left(\frac{2\pi t}{96}\right),cos\left(\frac{2\pi t}{96}\right),P_{norm}\left(t\right),C_{need}\left(t\right),F_t]$$

(7)

where

$$P_{norm}\left(t\right)={\displaystyle \frac{{\pi }_{\mathrm{electricity}}\left(t\right)-{\pi }_{min}}{{\pi }_{max}-{\pi }_{min}}}$$

represents normalized electricity pricing;

$$C_{need}\left(t\right)=\mathrm{clip}\left({\displaystyle \frac{T_{out}\left(t\right)-T_{in}\left(t\right)}{10}},-1,1\right)$$

quantifies cooling demand; and $F_t\in {\mathbb{R}}^6$ contains 6 h ahead temperature forecasts obtained from meteorological data.

The sine–cosine encoding of time-of-day ensures continuous temporal representation while avoiding discontinuities at midnight transitions.

Forecast features $F_t$ (Equation (7)) are treated as exogenous inputs available to all learning agents; they do not by themselves enforce resilience. SACEM’s improvement arises from the structural coupling of (a) entropy-regularized policy learning (robust base action proposal) and (b) decision-time sequence optimization that evaluates future comfort and cost consequences over a horizon (Equation (22)), with explicit safety projection (Equation (24)). To isolate this effect, all baselines share identical environment dynamics, reward (Equation (9)), network capacity, and training budget (Section 3.7), so differences are attributable to the presence or absence of the refinement-and-safety structure.

Such Markov Decision Process formulations, including augmented state representations with temporal encoding and exogenous forecasts, are standard in reinforcement learning-based HVAC control to enable anticipatory and adaptive decision making [32,33].

Action Space $\mathcal{A}=[-1,1]$

The continuous action space enables smooth HVAC power control through mapping:

$$P_{HVAC}\left(t\right)={\displaystyle \frac{a\left(t\right)+1}{2}}·P_{max}$$

(8)

where $P_{max}=10$ kW represents the maximum system capacity, and $a\left(t\right)\in [-1,1]$ is the policy output.

These observations are consistent with prior comparative studies showing that model-free DRL methods often struggle with comfort preservation and stability under high thermal stress and multi-objective reward settings [32,33].

Comparable MDP formulations with augmented state representations have been adopted in prior HVAC DRL studies to enable the anticipation of temporal patterns, pricing signals, and weather-driven dynamics [33,65].

3.3.2. Comfort-First Reward Structure

The reward function addresses the fundamental challenge in HVAC control RL, avoiding energy minimization at the expense of occupant comfort, consistent with prior findings on reward sensitivity and comfort degradation in building control applications [30,33].

$$\mathcal{R}(s_t,a_t)={\mathcal{R}}_{comfort}(s_t,a_t)+{\mathcal{R}}_{energy}(s_t,a_t)+{\mathcal{R}}_{peak}(s_t,a_t)+{\mathcal{R}}_{smooth}(s_t,a_t)$$

(9)

In Equation (9), the total reward is developed as a weighted combination of four elements that together represent the key trade-offs in HVAC control. The comfort term, denoted by ${\mathcal{R}}_{comfort}$, penalizes deviations of indoor temperature and air quality from acceptable values, thereby protecting occupant well-being. The energy term, ${\mathcal{R}}_{energy}$, promotes efficiency by penalizing excessive power consumption. The peak demand term, ${\mathcal{R}}_{peak}$, discourages instances of rapid increases in electricity use that can be costly under time-of-use pricing. Finally, the smoothness term, ${\mathcal{R}}_{smooth}$, penalizes sudden control actions to ensure stable system operation and prolong equipment lifespan. Combined, these objectives prevent the RL agent from reducing energy consumption at the expense of comfort, stability, or economic efficiency.

The use of asymmetric comfort penalties and multi-term reward decomposition follows established findings that poorly balanced reward designs often lead to comfort degradation or unstable control behavior in HVAC reinforcement learning applications [30,33].

Reward structures that explicitly balance comfort, energy, and stability are critical in avoiding degenerate or comfort-compromising policies in HVAC reinforcement learning [30,33].

Comfort Reward Component

The primary objective prioritizes thermal comfort maintenance through asymmetric penalties:

$${\mathcal{R}}_{\mathrm{comfort}}(s_t,a_t)=\left\{\begin{array}{cc}+10.0\hfill & \mathrm{if}\mathrm{Occ}\left(t\right)=1\hfill \\ \hfill & \mathrm{and}|T_{\mathrm{in}}(t+1)-T_{\mathrm{target}}|\le {\delta }_{\mathrm{comfort}}\hfill \\ +5.0\hfill & \mathrm{if}\mathrm{Occ}\left(t\right)=0\hfill \\ \hfill & \mathrm{and}|T_{\mathrm{in}}(t+1)-T_{\mathrm{target}}|\le {\delta }_{\mathrm{comfort}}\hfill \\ -100{ϵ}^2\hfill & \mathrm{if}\mathrm{Occ}\left(t\right)=1\mathrm{and}\mathrm{violation}\hfill \\ -25{ϵ}^2\hfill & \mathrm{if}\mathrm{Occ}\left(t\right)=0\mathrm{and}\mathrm{violation}\hfill \end{array}\right.$$

(10)

where $T_{target}=24.0$ °C for occupied periods and 26.5 °C for unoccupied periods; ${\delta }_{comfort}=1.0$ °C defines the acceptable comfort band; and

$$ϵ=|T_{in}(t+1)-T_{target}|-{\delta }_{comfort}$$

represents the magnitude of comfort violation.

Energy Cost Component

Economic optimization through price-weighted energy consumption:

$${\mathcal{R}}_{energy}(s_t,a_t)=-0.5·P_{norm}\left(t\right)·{\displaystyle \frac{P_{HVAC}\left(t\right)}{P_{max}}}$$

(11)

Peak Demand Component

Demand charge mitigation during high-price periods:

$${\mathcal{R}}_{peak}(s_t,a_t)=\left\{\begin{array}{cc}-1.0·\left(\frac{P_{\mathrm{peak},\mathrm{daily}}\left(t\right)}{P_{max}}\right)\hfill & \mathrm{if}{P}_{norm}\left(t\right)>0.8\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(12)

Action Smoothness Component

Penalizes excessive control oscillations:

$${\mathcal{R}}_{smooth}(s_t,a_t)=-0.1·{(a\left(t\right)-a(t-1))}^2$$

(13)

The reward structure maintains a 20:1 to 200:1 ratio between comfort penalties and energy costs, ensuring that comfort violations are economically untenable for the learning algorithm.

As demonstrated by Equations (11)–(13), the energy, peak demand, and smoothness blocks collectively lead the RL agent to cost-effective, stable, and demand-responsive operation. This design will guarantee that the agent minimizes the cost of consumption, as well as curtail peak loads, but it will not cause an abrupt control measure; thus, a compromise between economic efficiency, system stability, and occupant comfort is obtained.

3.4. Soft Actor–Critic Implementation

Soft Actor–Critic is selected due to its proven stability, sample efficiency, and robustness in continuous control tasks, including building energy management applications [64,65,66].

3.4.1. Policy Parameterization

The SAC algorithm serves as the foundation policy learner due to its sample efficiency and stability in continuous control tasks. The stochastic policy is parameterized as

$${\pi }_{\theta }\left(a\right|s)=tanh\left({\mu }_{\theta }\left(s\right)+{\sigma }_{\theta }\left(s\right)\odot ϵ\right),ϵ\sim \mathcal{N}(0,I)$$

(14)

where ${\mu }_{\theta }\left(s\right)$ and ${\sigma }_{\theta }\left(s\right)$ are neural network outputs with shared hidden layers:

$$\begin{array}{ccc}\hfill h_1& =\mathrm{ReLU}(W_{1}s+b_1),\hfill & \hfill h_1\in {\mathbb{R}}^{256}\\ \hfill h_2& =\mathrm{ReLU}(W_2{h}_1+b_2),\hfill & \hfill h_2\in {\mathbb{R}}^{256}\\ \hfill h_3& =\mathrm{ReLU}(W_3{h}_2+b_3),\hfill & \hfill h_3\in {\mathbb{R}}^{128}\end{array}$$

(15)

3.4.2. Value Function Approximation

Twin Q-networks mitigate overestimation bias through the minimum operator:

$$Q_{{\varphi }_i}(s,a)=W_{Q_i}^{\left(3\right)}\mathrm{ReLU}\left(W_{Q_i}^{\left(2\right)}\mathrm{ReLU}\left(W_{Q_i}^{\left(1\right)}[s;a]\right)\right),i\in \{1,2\}$$

(16)

Target networks provide training stability:

$${\varphi }_i^{-}\leftarrow \tau {\varphi }_i+(1-\tau ){\varphi }_i^{-},\tau =0.02$$

(17)

3.4.3. Training Objectives

The SAC training minimizes the following loss functions:

Q-function Loss:

$$\begin{array}{cc}\hfill L_Q\left(\varphi \right)& ={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}[(Q_{\varphi }(s,a)-(r+\gamma \underset{i=1,2}{min}{Q}_{{\varphi }_i^{-}}(s^{\prime },a^{\prime })\hfill \\ \hfill & -\alpha log{\pi }_{\theta }\left(a^{\prime }\right|s^{\prime })\left){)}^2\right]\hfill \end{array}$$

(18)

Policy Loss:

$$L_{\pi }\left(\theta \right)={\mathbb{E}}_{s\sim \mathcal{D},ϵ\sim \mathcal{N}}\left[\alpha log{\pi }_{\theta }\left(a\right|s)-\underset{i=1,2}{min}{Q}_{{\varphi }_i}(s,a)\right]$$

(19)

Temperature Parameter:

$$L_{\alpha }={\mathbb{E}}_{s\sim \mathcal{D}}\left[-\alpha \left(log{\pi }_{\theta }\left(a\right|s)+\overline{H}\right)\right],\overline{H}=-\left|\mathcal{A}\right|=-1$$

(20)

3.5. Cross-Entropy Method Action Refinement

The Cross-Entropy Method has been successfully applied in stochastic optimization and control to enable short-horizon decision refinement under uncertainty, particularly when exact dynamic programming is computationally infeasible [49].

Hybrid strategies combining learning-based policies with short-horizon optimization have previously been explored to mitigate myopic behavior in building control, motivating the integration approach adopted in this work [49].

3.5.1. Predictive Optimization Framework

The Cross-Entropy Method (CEM) is used as a refinement layer to counteract the myopic nature of the normal reinforcement learning (RL) policies, which have a tendency to maximize actions by short-term rewards. CEM builds on the base action suggested by the Soft Actor–Critic (SAC) policy, but it uses candidate action sequences along a finite prediction horizon in the SACEM framework.

Short-horizon predictive refinement has been shown to mitigate the myopic behavior of model-free reinforcement learning controllers in building energy management, particularly when full-horizon optimization is impractical [49].

Mathematically, under the present condition, $s_t,a_0$ is an action chosen according to the SAC policy, and CEM optimizes the following stochastic optimization problem:

$$\begin{array}{cc}\hfill a^{\ast }& =arg\underset{a\in {[-1,1]}^H}{max}J\left(a\right)\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \mathrm{where}J\left(a\right)& =E\left[\sum _{k=0}^{H-1}{\gamma }^k\mathcal{R}(s_{t+k},a_k)|s_t,a\right]\hfill \end{array}$$

(22)

Here, $a=[a_0,a_1,\dots ,a_{H-1}]$ denotes the candidate sequence of future actions over a prediction horizon of $H=6$ steps (equivalent to 1.5 h, with 15 min control intervals). The discount factor $\gamma \in (0,1)$ ensures that near-term rewards are prioritized, while still accounting for future costs and comfort.

The expectation is considered with regard to the stochastic property of building thermal dynamics, as well as uncertain occupancy patterns. This enables the optimization to trade-off between energy efficiency, comfort maintenance, and demand control over a predictive horizon, as opposed to just reactively controlled management. The selected horizon length H = 6 is a trade-off: longer horizons might in theory be more effective in obtaining improved foresight, but at much greater computational cost, and shorter horizons would restrict predictive utility.

3.5.2. Thermal Model Prediction

The deterministic thermal model formulated in Equation (23) relocated in the future evolution of state during refinement of action is approximated with exogenous forecast information with weather predictions and approximated occupancy schedules. Alternatively, the process at step $t+k$ can be predicted by

$${\widehat{s}}_{t+k}=f({\widehat{s}}_{t+k-1},a_{k-1},F_{t+k-1})$$

(23)

where $f(·)$ represents the state transition function. This function encapsulates building thermal dynamics, including envelope heat transfer and thermal mass effects, occupancy-driven gains, which contribute additional internal heat loads, and weather forecast features $F_{t+k-1}$, which provide exogenous input for outdoor temperature variations.

By iteratively applying Equation (23), the system can forecast the impact of a candidate action sequence on indoor temperature, energy consumption, and occupant comfort across the horizon. This predictive capability is essential for enabling anticipatory control, such as pre-cooling before peak price periods or load shifting toward off-peak intervals.

Together, the predictive optimization framework (Equation (22)) and the thermal prediction model (Equation (23)) form the foundation of CEM-based action refinement, ensuring that HVAC control decisions are both robust and forward-looking to environmental uncertainty.

3.5.3. CEM Algorithm Implementation

The CEM iteratively refines a probability distribution over action sequences to identify near-optimal solutions. Specifically, the algorithm maintains a Gaussian distribution $\mathcal{N}(\mathit{\mu },\mathsf{\Sigma })$ over candidate action sequences, which is progressively updated based on the best-performing elite samples. The procedure is as follows:

• Initialization: The mean is initialized around the base SAC action:

  $${\mu }^{\left(0\right)}={[a_0,a_0,\dots ,a_0]}^T$$

  and the covariance matrix is set to

  $${\Sigma }^{\left(0\right)}={\sigma }_0^2\mathbf{I},$$

  ensuring isotropic exploration of the action space. This initialization biases the search toward the SAC-proposed solution, while still allowing sufficient stochastic variation for refinement.
• Sampling: At iteration j, $N=60$ candidate action sequences are drawn:

  $$A^{\left(i\right)}\sim \mathcal{N}({\mu }^{\left(j\right)},{\mathsf{\Sigma }}^{\left(j\right)}),i=1,\dots ,N.$$
• Evaluation: Each candidate sequence $A^{\left(i\right)}$ is evaluated using the predictive objective $J(·)$ from Equation (22), yielding scalar performance scores:

  $$J^{\left(i\right)}=J\left(A^{\left(i\right)}\right).$$
• Selection: The top-performing $\rho =0.15$ fraction of samples (i.e., 15% of 60 candidates) is selected as the elite set:

  $${\mathcal{E}}^{\left(j\right)}=\{A^{\left(i\right)}:J^{\left(i\right)}\ge {\gamma }_{1-\rho }^{\left(j\right)}\},$$

  where ${\gamma }_{1-\rho }^{\left(j\right)}$ is the $(1-\rho )$-quantile of the objective scores at iteration j. This selection process filters out poorly performing samples, focusing the distribution update on promising regions of the action space.
• Update: The mean and covariance of the Gaussian distribution are updated using the elite set:

  $$\begin{array}{cc}\hfill {\mu }^{(j+1)}& =\frac{1}{|{\mathcal{E}}^{\left(j\right)}|}\sum _{A\in {\mathcal{E}}^{\left(j\right)}}A,\hfill \\ \hfill {\mathsf{\Sigma }}^{(j+1)}& =max\left({\sigma }_{min}^2,\frac{1}{|{\mathcal{E}}^{\left(j\right)}|}\sum _{A\in {\mathcal{E}}^{\left(j\right)}}(A-{\mu }^{(j+1)}){(A-{\mu }^{(j+1)})}^T\right)\hfill \end{array}$$
  Here, a lower bound ${\sigma }_{min}^2$ prevents premature convergence and ensures continued exploration.

The algorithm terminates after $J_{max}=4$ iterations or when the covariance norm satisfies ${\parallel \mathsf{\Sigma }\parallel }_F<{ϵ}_{conv}$, indicating convergence. The refined action is then taken as the first element of the final mean vector:

$$a_{\mathrm{refined}}={\mu }_1^{\left(J_{max}\right)}.$$

This iterative sampling-update mechanism enables CEM to combine exploration with exploitation, systematically improving upon the SAC baseline action by leveraging predictive evaluation of future comfort, energy, and cost outcomes.

3.5.4. Safety Constraint Integration

During CEM refinement, candidate actions must also respect physical and comfort safety constraints. To ensure that comfort-critical violations do not occur, a projection operator is applied to each candidate action:

$$a_{\mathrm{constrained}}=\left\{\begin{array}{cc}\max (a,a_{min})\hfill & \mathrm{if}{T}_{\mathrm{predicted}}>T_{target}+{\delta }_{comfort}+{\delta }_{safety},\hfill \\ \min (a,a_{max})\hfill & \mathrm{if}{T}_{\mathrm{predicted}}<T_{target}-{\delta }_{comfort}-{\delta }_{safety},\hfill \\ a\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(24)

Here, ${\delta }_{safety}=2.0$ °C introduces an additional margin beyond the comfort band ${\delta }_{comfort}$ to guard against extreme violations under uncertainty. The bounds $a_{min}=0.6$ and $a_{max}=0.2$ correspond to aggressive cooling and minimal operation, respectively.

This safety-aware adjustment ensures that even during exploration, actions remain physically feasible and aligned with occupant comfort, thus providing a critical safeguard against destabilizing control decisions. By combining predictive optimization with hard safety constraints, the SACEM framework achieves robust and reliable HVAC control suitable for real-world deployment.

Although formal guarantees against stealthy, adaptive cyber–physical disturbances require an explicit threat model and verified sensing assumptions, SACEM enforces a practical safety envelope through constraint-aware refinement. In particular, the safety projection in Equation (24) bounds actions whenever predicted temperatures approach unsafe regions, while the comfort-dominant penalties in Equation (10) discourage policies that exploit estimation errors by sacrificing comfort. Furthermore, domain randomization of key thermal parameters during training improves robustness to modeling and estimation mismatch. Under bounded observation error assumptions, these mechanisms support constraint satisfaction even when state estimates are noisy or partially unreliable.

3.6. SACEM Training Algorithm

SACEM couples an off-policy SAC learner with a short-horizon CEM refiner to unite sample-efficient learning with decision-time look-ahead; the end-to-end procedure and refiner are summarized in Algorithms A1 and A2. For clarity of exposition, the full training and refinement procedures are provided in Appendix A.

3.6.1. Experience Collection Strategy

To balance early exploration with later exploitation, we employ an $ϵ$-decay strategy before handing control to CEM refinement. Let $N_{\mathrm{learning}\text{\_}\mathrm{start}}$ denote the number of warm-up steps. The executed action is

$$a_{\mathrm{executed}}=\left\{\begin{array}{cc}{a}_{\mathrm{SAC}}+{ϵ}_{\mathrm{noise}},\hfill & \mathrm{if}\mathrm{step}<N_{\mathrm{learning}\text{\_}\mathrm{start}},\hfill \\ \mathrm{CEM}\text{\_}\mathrm{refine}\left(a_{\mathrm{SAC}}\right),\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(25)

where ${ϵ}_{\mathrm{noise}}\sim \mathcal{N}(0,{\sigma }_{\mathrm{explore}}^2)$. The exploration scale decays linearly over the first $50,000$ steps:

$${\sigma }_{\mathrm{explore}}\left(\mathrm{step}\right)={\sigma }_{\mathrm{hi}}-({\sigma }_{\mathrm{hi}}-{\sigma }_{\mathrm{lo}})·min\left(1,{\displaystyle \frac{\mathrm{step}}{50,000}}\right),{\sigma }_{\mathrm{hi}}=0.3,{\sigma }_{\mathrm{lo}}=0.1.$$

(26)

Equation (25) ensures wide state-action coverage early on improving value estimation while Equation (26) gradually reduces stochasticity so that, after the warm-up, exploitation leverages the predictive benefits of CEM refinement (Section 3.5.1), improving comfort preservation during expensive TOU windows.

3.6.2. Replay Buffer Management

We store transitions $(s_t,a_t,r_t,s_{t+1})$ with $a_t$, the refined action (post-CEM). This teaches the actor to propose base actions that, after refinement, yield high return. The replay capacity is $N_{\mathrm{buffer}}=1.5\times {10}^6$ transitions with prioritized sampling:

$$p_i={\displaystyle \frac{|{\delta }_i|+{ϵ}_{\mathrm{priority}}}{{\sum }_j\left(|{\delta }_j|+{ϵ}_{\mathrm{priority}}\right)}},$$

(27)

where ${\delta }_i$ is the TD error and ${ϵ}_{\mathrm{priority}}>0$ avoids zero probability for near-on-policy samples. Optionally, bias can be corrected with importance weights $w_i={\left({\displaystyle \frac{1}{N_{\mathrm{buffer}}{p}_i}}\right)}^{\beta }/{max}_j{w}_j$, with $\beta$ annealed toward 1.

3.6.3. Hyperparameter Configuration

The selected hyperparameters, summarized in

Table 2. SAC hyperparameters used in SACEM training.

Parameter	Value
Learning rate	$5\times {10}^{-4}$
Batch size	512
Discount factor ($\gamma$)	0.995
Soft update rate ($\tau$)	0.02
Replay buffer size	$1.5\times {10}^6$
Learning starts	50,000
Training frequency	Every 32 steps
Gradient steps per update	32
Target entropy	$-\left|\mathcal{A}\right|$ (=$-1$ for 1D action)

, follow commonly adopted configurations in prior SAC-based HVAC and continuous control studies to balance stability and sample efficiency.

A relatively large batch (512) stabilizes critic updates under prioritized replay, while $\gamma =0.995$ values predictive comfort and peak-price avoidance. Target entropy $-\left|\mathcal{A}\right|$ preserves sufficient exploration in continuous control, as shown in Table 2.

3.7. Experimental Design and Evaluation

All baseline algorithms, PPO, TD3, and SAC, share identical environment dynamics, reward Equation (9), network widths, and training budget to isolate the contribution of CEM refinement.

Weather inputs are derived from historical Saudi Arabian summer temperature profiles (Riyadh region), resampled to 15 min resolution and divided into training (60 days) and evaluation (21 days) periods. Occupancy follows the probabilistic commercial schedule defined in Equation (6). Electricity prices follow the three-tier TOU tariff in Equation (5). Validation scenarios include (i) nominal operation, (ii) injected forecast errors (additive temperature noise), (iii) unexpected occupancy changes, and (iv) extreme heat events exceeding 43 °C.

Disturbance-based evaluation using forecast uncertainty, occupancy variability, and extreme weather conditions is a common practice in recent HVAC control studies to assess robustness under realistic operational uncertainty [27,54].

Hypothesis H1 is tested through a comparative evaluation of comfort scores across SACEM and baseline controllers under identical environmental conditions. Hypothesis H2 is examined by jointly analyzing energy consumption and operating cost metrics under time-of-use pricing. Hypothesis H3 is evaluated using ablation studies and disturbance-based stress tests that inject forecast noise, occupancy shocks, and extreme outdoor temperature events, with robustness quantified via comfort violation rates and reward stability.

Such comparative, ablation-based, and disturbance-driven evaluation protocols are increasingly recommended in the recent HVAC control literature to assess robustness beyond nominal performance metrics [27,54,56].

3.7.1. Validation Protocol and Testing Scenarios

The proposed SACEM framework is validated through a controlled simulation-based experimental protocol designed to isolate algorithmic contributions and ensure fair comparison. Validation is performed using (i) fixed building dynamics shared across all controllers, (ii) identical reward definitions and network capacities, and (iii) matched training budgets and random seeds. Performance is evaluated on held-out test episodes not seen during training.

Three complementary validation dimensions are considered. First, baseline comparison evaluates SACEM against PPO, TD3, and SAC under identical operating conditions. Second, ablation studies selectively remove architectural components (CEM refinement, safety projection, forecast usage) to attribute performance gains to specific design elements. Third, disturbance-based stress testing introduces forecast noise, occupancy shocks, and extreme outdoor temperature spikes to assess robustness and resilience under realistic uncertainty.

3.7.2. Ablation and Disturbance-Based Verification of Resilience

To verify that gains stem from structural innovations rather than parameter tuning, we include targeted ablations: (i) SAC-only (removes CEM refinement), (ii) SAC + Forecast (keeps $F_t$ but removes CEM), and (iii) SACEM without safety projection (removes Equation (24)). In addition, we evaluate disturbance scenarios: injected forecast noise (additive temperature perturbations to $F_t$), occupancy shocks (unexpected $\mathrm{Occ}\left(t\right)$ flips), and extreme heat spikes in $T_{out}\left(t\right)$. Resilience is quantified by the comfort violation rate (Equation (31)), peak demand (Equation (34)), and reward stability (

Table 3. Comparison of reward signal characteristics across all algorithms. SACEM maintains consistently positive performance with significantly lower variability, while all baseline approaches fail to achieve sustainable positive rewards.

Algorithm	Moving Average Reward	Coefficient of Variation (CV)	Reward Trend
PPO	−150	0.68	Persistently negative, highly unstable
SAC	−100	0.45	Sporadic positive spikes, overall negative
TD3	−15	0.52	Moderately stable, but still negative
SACEM	+5.0	0.23	Consistently positive and stable

). These tests directly attribute performance to the refinement-and-safety structure and its ability to preserve comfort under uncertainty.

PPO

Clipped surrogate objective with GAE:

$$\begin{array}{cc}\hfill \mathrm{clip}\text{\_}\mathrm{range}& =0.15,\mathrm{gae}\text{\_}\mathrm{lambda}=0.98,\hfill \\ \hfill \mathrm{n}\text{\_}\mathrm{steps}& =2048,\mathrm{n}\text{\_}\mathrm{epochs}=10.\hfill \end{array}$$

(28)

TD3

Delayed policy updates and target policy smoothing:

$$\begin{array}{cc}\hfill \mathrm{policy}\text{\_}\mathrm{delay}& =2,\mathrm{target}\text{\_}\mathrm{noise}\text{\_}\mathrm{std}=0.1,\hfill \\ \hfill \mathrm{noise}\text{\_}\mathrm{clip}& =0.25\hfill \end{array}$$

(29)

These metrics are selected to jointly validate thermal comfort preservation, energy efficiency, economic performance, and peak-load mitigation, which together define successful HVAC control under hot-climate operation.

We report complementary metrics spanning comfort, energy, cost, and demand. Let $\Delta t=15$ min $=0.25$ h and T be the number of control steps.

$$\begin{array}{cc}\hfill \mathrm{Comfort}\mathrm{Score}(\%)& ={\displaystyle \frac{1}{T}}\sum _{t=1}^T\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill & 1\left(\left|T_{in}\left(t\right)-T_{target}\left(t\right)\right|\le {\delta }_{\mathrm{comfort}}\right)\times 100\%,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \mathrm{Energy}(\mathrm{kWh}/\mathrm{day})& =\sum _{t=1}^T{P}_{\mathrm{HVAC}}\left(t\right)\Delta t,\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill \mathrm{Cost}(\mathrm{SAR}/\mathrm{day})& =\sum _{t=1}^T{P}_{\mathrm{HVAC}}\left(t\right){\pi }_{\mathrm{electricity}}\left(t\right)\Delta t,\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill \mathrm{Peak}\mathrm{Demand}\left(\mathrm{kW}\right)& ={\displaystyle \frac{1}{D}}\sum _{d=1}^D\underset{t\in [96d,96(d+1\left)\right)}{max}{P}_{\mathrm{HVAC}}\left(t\right).\hfill \end{array}$$

(34)

Each result is the mean ± standard deviation over 5 seeds. We assess significance via Welch’s t-test with Bonferroni correction across pairwise comparisons.

CEM adds $O\left(N_{\mathrm{samples}}H\right)$ model rollouts per decision. With $N_{\mathrm{samples}}=60$ and $H=6$, each action evaluation performs $60\times 6=360$ thermal model steps (about $\sim 15\times$ the per-step cost of baselines without CEM). Despite this overhead, SACEM converges in fewer interactions, about 20% fewer steps to target performance, yielding comparable wall-clock time while achieving higher comfort and lower cost.

3.8. Implementation and Reproducibility

Ensuring reproducibility and transparency is central to this study, as it allows other researchers to replicate, validate, and extend our findings. All baselines are implemented using the Stable-Baselines3 framework, which provides standardized, reproducible implementations of widely used DRL algorithms. Our proposed SACEM agent integrates a Cross-Entropy Method (CEM) module atop the same Gymnasium-compatible environment, ensuring fair comparisons across methods.

The experiments were developed in Python, leveraging the PyTorch deep learning framework for policy representation and training.

Table 4. Software dependencies used for implementation.

Library/Tool	Version
Python	3.10+
NumPy	1.21+
PyTorch	1.10+
Stable-Baselines3	1.6+
Gymnasium	0.26+
Matplotlib (visualization)	3.5+
Requests (API ingestion)	2.28+

summarizes the primary software dependencies. These versions are explicitly reported to enable deterministic replication across machines and operating systems.

Hourly outdoor temperatures were obtained from Open-Meteo/ERA5 at a 0.25° spatial resolution. Missing values < 1% of records were linearly interpolated and aligned with the 15 min control grid. This ensures that weather features entering the state vector $F_t$ in Equation (7) maintain temporal consistency.

Table 5. Meteorological data pipeline for reproducibility.

Step	Description
Data Source	Open-Meteo/ERA5 hourly temperatures
Resolution	0.25° grid
Missing Data	<1% filled by linear interpolation
Alignment	Resampled to 15 min control interval
Integration	Passed as $F_t$ in state vector

presents a summary of the meteorological data set preprocessing pipeline.

Commercial occupancy patterns were synthesized using the probabilistic model described in Equation (6). This synthetic representation was validated against Saudi office occupancy profiles to reflect realistic daily usage. The randomization of model parameters during training enhanced policy robustness by exposing the agent to a broader distribution of occupancy scenarios.

To ensure replicability, we fix random seeds (seed = 42) across environment initialization, neural network weight initialization, and exploration noise.

Table 6. Hardware and software configuration used for training and evaluation.

Resource	Specification
CPU	Intel Core i7 (8 cores, 16 threads)
OS	Windows 11 (64-bit)
Python Distribution	Anaconda (Python 3.10)
IDE	Visual Studio Code Version 1.106.3

lists the computational resources used in training and evaluation. While minor differences due to floating-point precision and hardware architecture are inevitable, all results were reproduced within a variance of $\pm 2\%$, thereby confirming the stability of our pipeline.

By explicitly reporting software versions, Table 4, data preprocessing procedures, Table 5, and hardware configurations, Table 6, this study ensures that SACEM can be reproduced with high fidelity. Collectively, these measures guarantee that SACEM remains reproducible and fairly comparable to existing DRL baselines while being rigorously optimized for comfort-first HVAC control under Saudi TOU pricing.

Implementation details critical for reproducibility are summarized in Table 4, Table 5 and Table 6, which report software dependencies, meteorological data preprocessing, and computational resources, respectively, in line with reproducibility practices recommended in recent building energy control studies.

4. Results and Discussion

This section presents a comprehensive evaluation of SACEM through four complementary perspectives: (i) training convergence and learning stability, (ii) thermal comfort and temperature tracking performance, (iii) energy consumption and cost efficiency under TOU pricing, and (iv) reward dynamics and robustness across disturbance scenarios. Quantitative results are supported by tabulated metrics, while temporal behaviors are illustrated through time-series visualizations to ensure interpretability and reproducibility.

The results presented in this section are organized to examine the stated research hypotheses directly. Section 4.1 and Section 4.2 evaluate Hypotheses H1 and H2 through comfort, energy, and cost comparisons, while robustness analyses and ablation results in subsequent subsections address Hypothesis H3.

4.1. Training Performance and Convergence Analysis

The training performance of all algorithms was evaluated over 800,000 timesteps across identical environmental conditions to ensure fair comparison. The convergence characteristics reveal distinct learning patterns that provide insight into the effectiveness of the SACEM framework compared to baseline deep reinforcement learning approaches.

SACEM achieved the most stable convergence trajectory, reaching positive cumulative rewards and maintaining consistent improvement throughout training. This contrasts sharply with the baseline algorithms, which exhibited persistent negative rewards, indicating fundamental difficulties in balancing the multi-objective optimization landscape inherent in HVAC control.

The superior convergence of SACEM can be attributed to two key factors: (1) the CEM refinement process provides implicit regularization by smoothing the action space exploration, and (2) the predictive lookahead enables the algorithm to avoid myopic decisions that lead to long-term comfort violations. Statistical analysis across five independent training runs confirmed the significance of these performance differences ($p<0.001$, Welch’s t-test with Bonferroni correction).

Table 7. Performance comparison across all algorithms.

Algorithm	Comfort Score (%)	Energy (kWh)	Cost (SAR)	Peak Demand (kW)	Total Reward
PPO Baseline	15.5 ± 2.1	1627.7 ± 89.2	540.5 ± 31.4	8.2 ± 0.6	−149,572.8 ± 8234
SAC Baseline	16.6 ± 1.8	1317.2 ± 72.5	449.4 ± 28.9	7.2 ± 0.4	−161,898.8 ± 9156
TD3 Baseline	71.2 ± 3.4	1821.8 ± 95.7	605.2 ± 35.2	8.0 ± 0.5	−6327.2 ± 1842
SACEM	95.8 ± 1.2	1439.0 ± 51.3	476.2 ± 22.7	6.1 ± 0.3	+12,195.6 ± 1456
Improvement vs. Best	+34.6%	−21.0%	−21.3%	−24.6%	+1925%

presents the quantitative performance metrics for all evaluated algorithms across the 21-day evaluation period. The results demonstrate SACEM’s clear superiority across all evaluation criteria, establishing it as the state-of-the-art method for Saudi Arabian HVAC control.

Similar convergence difficulties for PPO, TD3, and SAC under multi-objective HVAC control have been reported in prior comparative studies, particularly when comfort penalties dominate the reward landscape [32].

4.2. Temperature Control Performance Analysis

Figure 1 presents the comprehensive temperature control performance across all evaluated algorithms, revealing fundamental differences in control strategies and effectiveness.

These observations align with earlier findings that model-free DRL controllers often struggle to maintain comfort under extreme thermal stress without explicit foresight or constraint handling [30,33].

4.2.1. Baseline Algorithm Performance Characteristics

PPO Performance (Figure 1a): The PPO algorithm demonstrates systematic under-cooling behavior with indoor temperatures frequently exceeding 25 °C and approaching outdoor conditions during peak thermal stress periods. The temperature control subplot reveals persistent deviations from the 24 °C setpoint, particularly during hours 100–400, where indoor temperatures range between 22 and 26 °C with poor tracking performance. The power consumption pattern shows relatively consistent cycling without clear adaptation to thermal load variations or pricing signals. The reward signal remains consistently negative with a 6 h moving average around −150, indicating the algorithm’s failure to balance comfort and energy objectives effectively. SAC Performance (Figure 1b): Standard SAC exhibits marginally improved comfort performance compared to PPO but still demonstrates significant temperature drift periods. Notable temperature excursions occur around hours 300–350 and 450–500, where indoor conditions deviate substantially from the comfort zone. The power consumption shows more responsive cycling patterns compared to PPO but lacks the sophisticated load management evident in advanced approaches. The reward signal shows high variability with occasional positive spikes but maintains an overall negative trend with a moving average around −100. TD3 Performance (Figure 1c): TD3 achieves the best baseline performance with a 71.2% comfort score, representing a significant improvement over PPO and SAC. The temperature control demonstrates more consistent tracking of the 24 °C setpoint, though notable oscillations remain visible throughout the evaluation period. The power consumption reveals more intelligent cycling with some evidence of load adaptation, though without clear price-responsive behavior. The reward signal shows improved stability, with a moving average near −15, substantially better than other baselines but still remaining negative overall.

4.2.2. SACEM Performance Analysis

There was exceptional temperature control with a 95.8% comfort score, maintaining indoor temperatures within the 23–25 °C comfort zone for the vast majority of the evaluation period. The temperature tracking subplot shows remarkable stability with minimal deviations from the 24 °C setpoint, even during extreme outdoor conditions reaching 43–44 °C. Occasional brief excursions above 25 °C occur primarily during the most severe thermal stress periods visible around hours 150–200 but recover rapidly due to predictive control capabilities. The power consumption pattern reveals sophisticated load management with clear evidence of intelligent cycling that adapts to both thermal load requirements and economic signals. Notable features include

• Load shifting behavior: Reduced power consumption during peak pricing hours (13:00–17:00 daily).
• Predictive cooling: Pre-cooling events preceding high outdoor temperature periods.
• Efficient cycling: Longer periods of moderate power consumption rather than frequent on–off transitions.

The reward signal demonstrates consistently positive performance with a 6 h moving average stabilized around +5.0, representing a fundamental improvement over all baseline approaches. The stability of this positive reward signal indicates successful multi-objective optimization, balancing comfort maintenance with energy efficiency.

4.3. Energy Consumption and Cost Performance Analysis

The energy consumption patterns observed in the power consumption subplots of Figure 1 provide crucial insights into the efficiency and economic implications of the evaluated algorithms. The baseline methods reveal distinct limitations in their ability to balance thermal load responsiveness with economic considerations. Specifically, PPO demonstrates relatively uniform power cycling between 0 and 10 kW without evident adaptation to varying thermal loads or electricity pricing signals. This uniformity leads to frequent peak power events across all periods, regardless of external conditions, highlighting the algorithm’s inability to optimize operational schedules. In contrast, SAC exhibits a more variable consumption profile, with frequent cycling between 0 and 9 kW. Although this reflects some responsiveness to thermal load fluctuations, it remains largely disconnected from economic signals, resulting in inefficient utilization of energy resources. TD3 presents the most sophisticated baseline behavior, showing partial correlation between power cycling and outdoor temperature variations. Nevertheless, its inability to integrate electricity price responsiveness limits its overall economic performance, despite providing more intelligent power management compared to PPO and SAC.

SACEM, instead, presents an energy management strategy that is significantly more sophisticated and involves thermal load adaptation, economic optimization, and predictive control. It also shows a tendency to be used in accordance with the outside temperature changes, and more energy is spent when the weather is extremely hot, and less energy is spent when the weather is cool, which may be viewed as a successful thermal adjustment of loads. Additionally, the algorithm exhibits an evident economic optimization ability whereby consumption is minimized at peak pricing times and compensated at pre-cooling at off-peak times, which minimizes costs but does not sacrifice thermal comfort. The fact that power consumption is anticipated to increase before significant thermal stress events further confirms the predictive nature of SACEM and its capability to use weather forecasting to maintain comfort as well as efficiency.

The economic performance analysis, illustrated through the cost versus comfort trade-off subplots in Figure 1, provides further evidence of SACEM’s superiority. Over the evaluation period, SACEM achieves a total cost of 476.2 SAR, significantly lower than the 605.2 SAR recorded for the best-performing baseline, TD3. This 21.3% reduction in expenditure is achieved without compromising comfort levels, thereby establishing SACEM’s dual advantage in economic and performance outcomes. The cost-saving potential is explained by three interrelated mechanisms. First, SACEM’s load shifting is evident in the cumulative cost trajectories, which flatten during peak pricing hours as demand is shifted to more economical periods. Second, its superior management of peak demand is reflected in the steadier cost accumulation rates compared to the sharp increases observed for the baselines, indicating effective mitigation of expensive peak events. Finally, SACEM’s comfort–cost optimization is particularly striking: while baseline algorithms accumulate between 1500 and 1700 comfort violations, SACEM limits this figure to approximately 80, thus ensuring substantial reductions in discomfort while simultaneously maintaining lower operational costs. Collectively, these findings confirm SACEM’s capacity to harmonize energy efficiency, cost savings, and comfort preservation, thereby positioning it as a highly effective solution for real-world HVAC applications in demanding climatic conditions.

Comparable limitations in price responsiveness and peak-load mitigation have been observed in prior DRL-based HVAC studies that do not explicitly incorporate predictive or cost-aware control mechanisms [32,35].

4.4. Reward Dynamics and Comparative Learning Performance

The evolution of the reward signals illustrated in Figure 1 provides essential validation of the comfort-first reward design and offers insights into the learning effectiveness of each algorithm. Among the evaluated approaches, SACEM demonstrates a consistently positive trajectory, with the 6-h moving average stabilizing around +5.0. This stable performance is further reinforced by its low variability, reflected in a coefficient of variation of 0.23, which confirms the robustness of its optimization process.

The baseline algorithms, by contrast, reveal significant limitations in both reward stability and convergence effectiveness. PPO consistently generates negative rewards with a moving average around −150, coupled with a high degree of variability (CV = 0.68), which demonstrates its inability to reconcile comfort and energy efficiency objectives. SAC shows marginal improvements with sporadic positive spikes, yet the overall trend remains negative, with a moving average around −100, and highly unstable (CV = 0.45). TD3 provides the most stable baseline performance, with a moving average near −15 and moderate variability (CV = 0.52), yet it too fails to cross into sustained positive reward territory. These results confirm the inability of conventional methods to adapt effectively to the multi-objective nature of the HVAC control problem when using a comfort-first reward design.

The comparative performance of the algorithms can be summarized in Table 3, which presents the cumulative reward signals. SACEM stands out not only by achieving positive average rewards but also by maintaining lower variability and stronger stability over time, whereas all baseline approaches remain confined to negative domains despite their differing degrees of oscillatory behavior.

The superiority of SACEM is further validated through statistical testing across five independent evaluation runs. Its comfort score of 95.8% ± 1.2% represents a statistically significant improvement over all baselines ($p<0.001$, Cohen’s $d=5.2$ when compared with TD3). Furthermore, despite this superior comfort performance, SACEM achieves a 21.0% reduction in energy consumption relative to TD3 ($p<0.01$, Cohen’s $d=2.8$), demonstrating the success of its multi-objective optimization. From an economic standpoint, SACEM achieves a 21.3% cost reduction compared to TD3 ($p<0.001$, Cohen’s $d=3.1$), highlighting its practical advantage for deployment in real-world systems. Perhaps most striking is the improvement in overall reward performance: the transition from a negative cumulative reward of −6327.2 for TD3 to a positive cumulative reward of +12,195.6 for SACEM represents an extraordinary enhancement, with an effect size of Cohen’s $d=5.8$. This magnitude of improvement far exceeds thresholds of practical significance and underscores the robustness, scalability, and transformative potential of SACEM in HVAC optimization.

The importance of robustness evaluation under uncertainty has been increasingly emphasized in recent HVAC and building energy management studies, particularly for deployment in extreme or rapidly changing environments [27,54].

4.5. Practical Deployment Implications

The analysis of the experimental findings outlines some implications for the practical application of SACEM in HVAC systems. Its small conformity ranges with confidence intervals between 94.6 and 97.0% and operating costs between 453 and 499 SAR clearly demonstrate its stability and usefulness in practice (as opposed to the wide uncertainty and poorer performance of the baseline techniques). Also, SACEM proves its capacity to balance optimization of several objectives, at the same time scoring 95.8% of the comfort, cutting energy use by 21%, and operating expenses by 21.3%, thus testing its competency in balancing comfort, efficiency, and economic factors. Lastly, the fact that it is working under a wide range of climate conditions, with the ability of outdoor temperatures ranging between 28 and 44 °C throughout the 21 days of evaluation, proves that the strategy is flexible to the extreme temperatures characteristic of the Saudi Arabian summers. Together, the results prove SACEM to be a breakthrough in intelligent HVAC control, which provides effective, efficient, and economical solutions that can be trusted and relied on to be applied in practice.

5. Conclusions and Future Work

This section concludes our study by summarizing the key contributions, Section 5.1, highlighting the main limitations, Section 5.2, and outlining future research directions that can further advance intelligent HVAC control in hot-climate buildings, Section 5.3.

5.1. Key Findings

The experimental evaluation provides several concrete findings. First, SACEM achieves a 95.8% comfort score, significantly outperforming PPO, SAC, and TD3, which struggle to maintain comfort under extreme outdoor temperatures. Second, this improvement is achieved without sacrificing efficiency: SACEM reduces energy consumption by approximately 21% and operating cost by 21.3% relative to the strongest baseline (TD3). Third, SACEM consistently attains positive cumulative rewards, whereas all baseline methods remain in negative-reward regimes, indicating persistent trade-off failures between comfort and energy objectives. Finally, ablation and disturbance-based tests confirm that these gains stem from the combined effect of CEM refinement and safety-aware comfort penalties, rather than parameter tuning alone.

The experimental results confirm all stated research hypotheses. Hypothesis H1 is supported by SACEM’s significantly higher comfort scores relative to all baseline controllers. Hypothesis H2 is validated by the simultaneous reduction in energy consumption and operating cost achieved under time-of-use pricing. Hypothesis H3 is confirmed through robustness evaluations showing reduced comfort violations and improved stability under uncertainty and extreme conditions. Together, these findings provide strong empirical evidence for the effectiveness and reliability of the proposed comfort-first hybrid control framework.

5.2. Limitations

Although SACEM has shown better performance, there are a number of drawbacks. To begin with, the tests were done on simulation models and not on a real-world environment; hence, the question of how these findings can be transferred to reality-building conditions where dynamics are not modeled and where other factors, such as uncertainty, exist. Second, the testing was limited to meteorological and tariff Riyadh, so the findings might not necessarily apply to other Saudi cities or to other regions in the world with hot climates and different climatic and operational characteristics. Lastly, the computational overhead of CEM refinement was not very demanding in simulations, but may be a problem with real-time implementation on low-cost building management hardware.

A further limitation is that we do not provide formal robustness guarantees against adversarial or stealthy cyber–physical attacks under fully compromised sensing. Future work will incorporate explicit threat models, sensor anomaly detection with confidence-weighted state inputs, and robust/constrained RL formulations to enable stronger safety  assurances.

5.3. Future Research Directions

To confirm the soundness of SACEM under various meteorological and tariff conditions, future studies should consider extending the analysis of SACEM to cover other big cities in Saudi Arabia, e.g., Jeddah and Dammam, and even outside the country to large international hot-climate areas. Applications and real-life experiments will be necessary to ensure that the framework is flexible to real-world uncertainties. By strengthening the reward functionality to reflect holistic comfort measures such as air quality, humidity, and noise, as well as sustainability measures such as carbon emissions, it will be possible to optimize more comprehensively. Moreover, the combination of model-based reinforcement learning and transfer learning could decrease training and enhance a system that can be used in a diverse climate. Last, real-time edge computing implementations and the speculation of adaptive tariffs in the changing electricity pricing schemes will provide the feasibility and scalability of SACEM in operational building management systems.