A Hybrid LSTM–PPO–RBC Edge Controller for Building HVAC: Sim-to-Real Validation on a Raspberry Pi 4

Abstract

Deep reinforcement learning (DRL) controllers for heating, ventilation and air conditioning (HVAC) are rarely deployed on real edge hardware or evaluated across climates, and published DRL-HVAC studies seldom report a physical deployment. The contribution here is integration rather than a new algorithm: we build an end-to-end Sim-to-Real deployment of a recurrent proximal policy optimisation (PPO) agent, whose policy network contains a long short-term memory (LSTM) core and is supervised by a rule-based safety shield via the Simplex architecture on a Raspberry Pi 4 and ESP32 edge stack. The recurrent policy is trained once on a Kuala Lumpur EnergyPlus/Sinergym profile of the five-zone VAV archetype and then evaluated without retraining on five further climate profiles (New York, Tokyo, Egypt, Iceland and Saudi Arabia), reaching a comfort-zone compliance rate (CZCR) of 100.00% on New York and Kuala Lumpur, 99.50% on Tokyo, 99.65% on Iceland, 97.24% on Saudi Arabia and 93.50% on Egypt (single-run training). A ten-seed re-run on the training climate confirms 100.00% CZCR and 0 violations in every seed, with energy varying by a 2.3% coefficient of variation (mean 29,332.68 ± 673.30 kWh). The cross-climate transfer figures above remain single-run. A matched-climate PPO baseline, retrained directly on Kuala Lumpur rather than zero-shot transferred from New York, also reaches 100.00% CZCR at 13.2% less energy than the hybrid, showing that the hybrid’s energy premium on this climate is larger, not smaller, once the training climate confound in the original baseline comparison is removed. A two-sided safety shield (23.2–25.8 °C triggers) is shown, against an archived upper-bound-only configuration, to reduce overcool violations from 4124 (88.20% CZCR) to zero on the same climate. On a physical benchtop hardware-in-the-loop (HIL) testbed, three repeated 30 min trials, selected post hoc as the cleanest window from each of three evening sessions (the same three sessions’ full-length CZCR values are 98.02%, 97.22%, and 91.50%), record a 100.00% CZCR within the selected windows and a predicted mean vote (PMV) of 0.043 ± 0.023, and the system self-heals from a 12 min 42 s router outage with a $+0.06$ °C thermal drift. On-device inference takes 112.9 ms on the Raspberry Pi 4 ($n=1392$ logged invocations, with 95th/99th-percentile latency of 116.3 ms/120.4 ms), roughly four orders of magnitude below the 15 min control step. A separate set of three trials, bracketed by the testbed’s own “30 min Test” start/end log events rather than selected after the fact, records a lower but more representative 86.42–92.59% CZCR (aggregate 90.12%). These are benchtop feasibility results from a single disclosed training run and a short physical trial. They represent a step toward narrowing, rather than a closed, Sim-to-Real and edge deployment gap in practical DRL-HVAC.

1. Introduction

Buildings account for approximately 28% of global final energy consumption [1], and within the operational energy of a building, heating, ventilation and air conditioning (HVAC) accounts for roughly 40–60% of total building energy consumption in most climate zones [2,3]. The decarbonisation pathway of the buildings sector therefore hinges on a step change in HVAC control efficiency.

The control problem is well posed but hard. The underlying thermal dynamics are multi-zone, time-varying and stochastic. Comfort requirements are occupant-specific and predicted mean vote (PMV) is bounded by ISO 7730 [4]. The energy tariff is time-of-use and increasingly volatile. Four control families have been deployed at scale: rule-based control (RBC), proportional–integral–derivative (PID) control, model-predictive control (MPC) and deep reinforcement learning (DRL). Each has well-documented limitations. RBC is simple but myopic. PID is single-loop and reactive. MPC requires an accurate grey-box model and is computationally expensive at the edge [5,6]. In contrast, DRL, whose control lineage runs from DQN [7] and AlphaGo [8] to continuous control policy-gradient methods [9,10], is sample-hungry, fragile under Sim-to-Real transfer, and rarely deployed on real hardware [11,12]. In a review of 135 identified DRL-HVAC studies, Al Sayed et al. [11] find that only 23% report a real-building deployment. The DRL-HVAC gap is therefore not a research gap alone. It is a deployment gap, compounded at the edge, where the controller must run on low-cost embedded hardware and tolerate sensor and network faults.

This paper asks whether a recurrent-PPO HVAC controller, supervised by a lightweight rule-based safety override and running on low-cost edge hardware, can maintain comfort-band compliance across climates it was not trained on, while keeping on-device inference fast enough for real-time control. It also asks whether this holds up when the controller is taken off simulation and run on physical hardware. This research question decomposes into three specific gaps in the published literature. The first is an integrated recurrent controller with a rule-based safety override, in which a proximal policy optimisation (PPO) agent whose policy network contains a long short-term memory (LSTM) recurrent core (recurrent PPO [10,13]) is supervised by an RBC safety monitor via the Simplex architecture [14,15], an instantiation of runtime shielding for safe reinforcement learning [16,17] so that, once the monitored temperature crosses a fixed threshold, the learned policy’s command is overridden by a verifiable rule. Section 3.3 states the exact guarantee that this override provides and the assumptions it relies on, which are weaker than an unconditional safety proof. The second is edge-deployable DRL with measured latency: inference runs on a Raspberry Pi 4 with the physical actuation on an ESP32 node, and every logged inference completes in 112.9 ms (mean, $n=1392$), about four orders of magnitude below the 15 min simulation control step and two orders below the roughly 30 s action cadence of the deployed edge loop. The third is zero-shot cross-climate transfer, in which the same policy, trained once on a Kuala Lumpur profile, is evaluated without any retraining on five further climate profiles (New York, Tokyo, Egypt, Iceland and Saudi Arabia) spanning hot, temperate, continental and cold climates. The contribution is the integration of these elements, building on the algorithmic lineage of Bai and Tan [18] and the Sinergym benchmark corpus [19]: the Sim-to-Real pipeline. It combines the rule-based safety-override shield, the MQTT–TLS edge stack, the Unity digital twin, and the physical benchtop validation. The paper is framed as an empirical engineering case study rather than as a new algorithm.

The main contributions of this paper are as follows.

• We present an end-to-end Sim-to-Real deployment of a hybrid LSTM–PPO–RBC controller on a Raspberry Pi 4 and ESP32 edge stack, integrating Sinergym/EnergyPlus training, MQTT–TLS dispatch, a Unity 6 digital twin interface, and a benchtop hardware-in-the-loop (HIL) testbed with a measured on-device inference latency of 112.9 ms ($n=1392$ LSTM policy inference invocations recorded in the on-disk log). Positioned against the closest single-paper analogue, the robust DRL-HVAC approach of Bai and Tan [18], this is an integration contribution. Although the algorithmic lineage (PPO plus an LSTM component) is shared, the Sim-to-Real pipeline, the rule-based two-sided safety-override shield, and the real-hardware validation are distinct (Section 2 and Section 4.4).
• We demonstrate zero-shot transfer of the Kuala Lumpur-trained recurrent policy to the New York, Tokyo, Egypt, Iceland and Saudi Arabia climate profiles without controller retraining. With the deployed two-sided safety shield, the hybrid records a 100.00% CZCR on the New York and Kuala Lumpur traces (0 violations in 34,944 post-warm-up steps each), 99.50% on Tokyo, 99.65% on Iceland, 97.24% on Saudi Arabia and 93.50% on Egypt. The energy outcome is a premium rather than a saving relative to a PPO baseline ($+1.7$% on New York, $+3.1$% on Tokyo, both zero-shot comparisons, and, on Kuala Lumpur, $+6.8$% against a zero-shot PPO baseline but $+15.2$% against a matched-climate PPO baseline trained directly on Kuala Lumpur, which reaches the same 100.00% CZCR at lower energy, Section 4.3). The figures are single-run estimates, except the Kuala Lumpur hybrid energy figure itself, which a ten-seed re-run (Section 4.2) shows varies by a 2.3% coefficient of variation (Section 4.1). The controller’s value proposition is therefore not lowest-cost comfort on a single climate. A correctly trained simple PPO baseline matches its comfort outcome at lower energy where that comparison is available. The value proposition is zero-shot transfer: the same trained policy reaches comparable comfort across six climates without per-climate retraining, which the matched-climate baselines do not attempt.
• We show that the safety shield’s lower bound is load-bearing: an archived evaluation configuration in which the shield triggered only on the upper boundary records 4124 overcool violations (88.20% CZCR) on the Kuala Lumpur trace, with excursions 0.49 °C below the 23.0 °C comfort floor, whereas the deployed two-sided configuration (upper trigger 25.8 °C, lower trigger 23.2 °C) records 0 violations on the same climate (Section 4.5).
• We validate the controller on a benchtop HIL testbed across three repeated 30 min trials: 100.00% CZCR in each trial, a PMV of 0.043 ± 0.023 across the three trial means, a 12 min 42 s router outage that self-heals with a $+0.06$ °C thermal drift, and a 112.9 ms mean inference latency. The trials undergo feasibility validation on a 0.5 m³ enclosure, not a real-building deployment (Section 4.7).
• We provide a Unity 6 digital twin visualisation layer that mirrors the live controller state in real time, enabling remote operator inspection with a live temperature trace and a human–machine interface (HMI) override panel (Section 3.6).
• We release the code, the trained recurrent PPO model, the Sinergym environment configuration, the per-step evaluation traces, the hardware telemetry log, and the analysis scripts under the MIT licence.

The remainder of the paper is structured as follows. Section 2 surveys related work on DRL for HVAC, MPC, LSTM forecasting, edge IoT, digital twins and cross-climate transfer. Section 3 details the three-tier architecture, the simulation environment, the recurrent PPO controller, the RBC safety shield, the edge stack and the Unity digital twin. Section 4 reports simulation, cross-climate, shield-bound, sensor-fault, ablation and physical testbed results. Section 5 discusses implications, limitations and comparison with prior literature. Section 6 concludes.

2. Related Work

This section positions the present system against representative studies from four strands of the literature: MPC for HVAC, DRL for HVAC, LSTM load forecasting, and edge IoT with digital twins. A comparison matrix is given in Table 1.

Two strands of infrastructure underpin the evaluation before any controller is trained. The simulation layer is EnergyPlus [20], the US Department of Energy building-energy simulation engine, wrapped by the Sinergym interface [21,22], which exposes EnergyPlus as a standard gymnasium environment with a defined observation and action space. This choice makes the present results directly comparable to the published Sinergym benchmark corpus. The Sim-to-Real dimension is a recognised failure point for DRL controllers as shown by Zhao et al. [23], who surveyed Sim-to-Real transfer in deep reinforcement learning and identified a mismatch between training and deployment dynamics. This “reality gap” was identified as the central obstacle, which is why the present paper validates on physical hardware (Section 4.7) rather than relying on simulation realism alone.

2.1. Model-Predictive Control for HVAC

Taheri et al. [3] provided a state-of-the-art review of MPC for HVAC, reporting that energy reductions vary widely across surveyed studies by building type, climate and baseline. Afram and Janabi-Sharifi [5] reviewed the theory and applications of MPC for HVAC control systems and identified model accuracy, forecast quality, and solve time as recurring practical constraints. A similar claim was made by Oldewurtel et al. [6], who demonstrated the canonical combination of MPC with weather forecasts for energy-efficient building climate control. The intelligent building control literature predates both, as the survey of Dounis and Caraiscos [24] on fuzzy-logic, agent-based and other advanced control approaches for joint energy and comfort management showed. In a systematic review of 112 papers on MPC in real-world buildings, Rastad et al. [25] reported energy savings ranging from a few percent up to 56.3%, with cost reductions of up to 55%. Mariano-Hernández et al. [26] reviewed building energy management system (BEMS) strategies and organised the field around four pillars: MPC, demand-side management, optimisation, and fault detection and diagnosis. The MPC family is mature but expensive at the edge because an MPC controller requires a calibrated grey-box model, weather forecasts, and a solve at every control step [5], a compute footprint that does not fit comfortably on a Raspberry Pi 4 class node. The MPC deployment gap on low-cost edge hardware therefore remains a barrier. This is why no MPC baseline is run in the present study: deriving and calibrating a grey-box model for the benchtop Peltier/PTC enclosure is itself a substantial undertaking, and the paper’s scope is an edge-deployable learning-based controller rather than a head-to-head MPC comparison. This is left as future work (Section 5.3).

2.2. Deep Reinforcement Learning for HVAC

DRL controllers for HVAC have been demonstrated in simulation across a range of building types, with PPO, DQN and actor–critic variants dominating the recent literature [12,27,28]. The algorithmic lineage is standard DRL: DQN [7] established value-based deep control on discrete actions with experience replay, while DDPG [9] extended actor–critic learning to continuous action spaces, the setting required by thermostat setpoints. TD3 [29] corrects the value-overestimation bias of the actor–critic construction. SAC [30] adds maximum-entropy exploration and the trust region guarantee of TRPO [31] was simplified by PPO [10] into a first-order clipped surrogate, whose per-update cost makes it the natural choice for edge inference. Manjavacas et al. [19] evaluate PPO, TD3 and SAC head-to-head on Sinergym buildings and find PPO the most stable of the three. This is the direct basis for the present paper’s choice of RecurrentPPO over SAC or TD3: on the same simulation benchmark family used here, PPO was the more stable of the three off-the-shelf options, and its on-policy, first-order clipped surrogate update (rather than the off-policy replay buffer mechanics of SAC/TD3) keeps the per-update compute and memory footprint small, which matters for a policy that must also be exported and run deterministically on a Raspberry Pi 4. A head-to-head re-run of SAC/TD3 specifically with the LSTM-recurrent, edge-deployed, safety-shielded configuration used here was not performed and is identified as future work (Section 5.3). Two further reviews frame the field, the work of Vázquez-Canteli and Nagy [32] for demand response, and Mason and Grijalva [33] for autonomous building energy management. Wang and Hong [12] identify three recurring barriers: time-consuming and data-hungry training, control security and robustness that need enhancement, and generalisation capabilities that need improvement via transfer learning. The present paper engages with all three approaches: the trained recurrent policy is small enough to run on a Pi 4, the Simplex safety shield constrains the controller’s command once the monitored temperature crosses a fixed threshold (Section 3.3), and the zero-shot five-climate evaluation directly exercises transfer. Gupta et al. [27] compare centralised and decentralised DRL heating control and find that decentralised control performs better as zone set-point diversity grows, with energy-cost reductions of 5–12% relative to a traditional thermostat. Gao and Wang [28] compare model-free DDQN and SAC controllers against their model-based counterparts and find the model-based variants mainly win on sample efficiency rather than closed-loop performance. Al Sayed et al. [11] find that only 23% of published DRL-HVAC studies report a real-building deployment, a figure that motivates the physical validation strand of this paper. Two further studies help position the present contribution. Tian et al. [34] systematically review 144 building energy management studies under the Artificial Intelligence of Things (AIoT) framing and find that most proposed systems remain architecturally fragmented, with sensing, optimisation and control treated as separate layers rather than an integrated edge-to-cloud pipeline. The present paper’s three-tier Sinergym-to-Pi-to-digital twin stack is one instance of the integrated pipeline that review calls for, evaluated on a single case rather than surveyed across many. Li et al. [35] propose Safe-AutoSAC, an AutoML-enhanced safe deep reinforcement learning controller for integrated energy system scheduling with demand response. Their safety mechanism is built into the SAC training objective itself, whereas the present paper’s safety mechanism is an external runtime shield (Section 3.3) layered on an otherwise-unconstrained PPO policy. The two approaches are complementary safe-RL strategies for different control problems (energy system scheduling versus thermal comfort control) rather than directly comparable baselines.

2.3. LSTM Models for Building Energy Prediction

LSTM networks and their variants are the dominant architecture for short-term building energy prediction. Li et al. [36] assess vanilla LSTM against its modifications (including bidirectional, convolutional, attention-based and Seq2Seq variants, alongside GRU) for 24 h-ahead building energy prediction using normalised RMSE metrics. Kim et al. [37] propose a sequence-to-sequence deep-learning prediction model driven by dynamically simulated building data. Li et al. [38] evaluate sequence-to-sequence attention models for short-term multi-step-ahead building energy prediction. The architecture itself dates to Hochreiter and Schmidhuber [39], whose constant-error-carousel construction overcomes the vanishing-gradient limits of recurrent training, with the forget gate of Gers et al. [40] extending the horizon over which state is retained. The present paper uses an LSTM in a different role, namely as the recurrent memory inside the PPO policy network (recurrent PPO), which lets the controller integrate thermal history over time (a property the forget gate was introduced to enable) rather than as an open-loop forecaster.

2.4. Edge IoT and Digital Twins

The three-tier topology of the present system follows the canonical edge-computing argument of Shi et al. [41], who position computation at the network edge as the response to the latency and bandwidth constraints of cloud round trips. Chen and Ran [42] reviewed deep-learning inference success specifically on edge devices, and Deng et al. [43] framed the resulting edge intelligence paradigm in which training and inference migrate toward the data source. The underlying protocol layer was standard IoT. Al-Fuqaha et al. [44] surveyed enabling technologies and protocols, of which the MQTT publish–subscribe transport used here is a representative. Yu et al. [45] applied a DDPG-based deep reinforcement learning algorithm [9] to smart-home energy management, reporting cost savings of 8.1–15.2% across the baselines they consider. On the digital twin strand, Tao et al. [46] surveyed digital twins in industry and identified real-time state mirroring as the core function. Stogia et al. [47] proposed a framework integrating IoT and digital twins for home energy management, arguing that traditional non-immersive dashboards fail to convey real-time state. The present paper closes this gap with a Unity 6 application that subscribes to the same MQTT topic hierarchy as the controller and renders the live state in 3D.

2.5. Positioning Against the Closest Analogue

The closest single-paper analogue is Bai and Tan [18], who combined a decoupled adversarial learning (DAL) architecture with PPO in a robust DRL controller for HVAC energy efficiency, thermal comfort and indoor air quality. The present paper differs from [18] in four respects.

1.

The LSTM here is the recurrent memory of the policy network itself (recurrent PPO), integrating state history at decision time, rather than a decoupled adversarial feature extractor as in the DAL construction.

2.

The Simplex architecture is used as a two-sided rule-based override with explicit temperature triggers (Section 3.3 states the exact guarantee and its assumptions), which DAL-PPO does not implement.

3.

A real-hardware validation on a benchtop testbed with measured on-device latency and a recorded outage-recovery event is reported, which DAL-PPO does not report.

4.

Training here is primarily a single disclosed run, with a ten-seed re-run on the training climate reported in Section 4.2 and multi-seed coverage of the remaining climates and baselines identified as further future work (Section 5.3).

• The two papers are not directly comparable numerically because [18] uses a different reward function and a different building model. A second positioning anchor is the Sinergym benchmark corpus [19,22]: the same five-zone building archetype and the same recurrent-PPO implementation are used, so the simulation results are directly comparable to that benchmark corpus. The 30 min HIL protocol is below the threshold for a real-building deployment, since the testbed is a 0.5 m³ benchtop enclosure rather than an occupied building. This gap is acknowledged explicitly in Section 5.

Table 1. Related-work comparison matrix. Columns are scope flags: DRL (deep RL used), LSTM (LSTM used), MPC (model-predictive control), RBC (rule-based baseline), IoT (edge IoT deployment), Edge (real-time edge inference), Transfer (cross-climate zero-shot evaluation) and DT (digital twin). Y = yes, N = no, P = partial.

Study	DRL	LSTM	MPC	RBC	IoT	Edge	Transfer	DT
Taheri et al., 2022 [3]	N	N	Y	Y	N	N	N	N
Mariano-Hernández et al., 2021 [26]	N	N	Y	N	Y	N	N	N
Olu-Ajayi et al., 2022 [48]	N	Y	N	N	N	N	N	N
Al Sayed et al., 2024 [11]	Y	N	Y	Y	N	N	N	N
Gupta et al., 2021 [27]	Y	N	N	Y	N	N	N	N
Gao and Wang 2023 [28]	Y	N	P	N	N	N	N	N
Ala’raj et al., 2022 [49]	Y	N	Y	Y	N	N	N	N
Homod et al., 2023 [50]	Y	N	N	N	Y	N	N	N
Li et al., 2021 [36]	N	Y	N	N	N	N	N	N
Kim et al., 2021 [37]	N	Y	N	N	N	N	N	N
Bai and Tan 2024 [18]	Y	Y	N	N	N	N	N	N
This work	Y	Y	P	Y	Y	Y	Y	Y

Table 1. Related-work comparison matrix. Columns are scope flags: DRL (deep RL used), LSTM (LSTM used), MPC (model-predictive control), RBC (rule-based baseline), IoT (edge IoT deployment), Edge (real-time edge inference), Transfer (cross-climate zero-shot evaluation) and DT (digital twin). Y = yes, N = no, P = partial.

Study	DRL	LSTM	MPC	RBC	IoT	Edge	Transfer	DT
Taheri et al., 2022 [3]	N	N	Y	Y	N	N	N	N
Mariano-Hernández et al., 2021 [26]	N	N	Y	N	Y	N	N	N
Olu-Ajayi et al., 2022 [48]	N	Y	N	N	N	N	N	N
Al Sayed et al., 2024 [11]	Y	N	Y	Y	N	N	N	N
Gupta et al., 2021 [27]	Y	N	N	Y	N	N	N	N
Gao and Wang 2023 [28]	Y	N	P	N	N	N	N	N
Ala’raj et al., 2022 [49]	Y	N	Y	Y	N	N	N	N
Homod et al., 2023 [50]	Y	N	N	N	Y	N	N	N
Li et al., 2021 [36]	N	Y	N	N	N	N	N	N
Kim et al., 2021 [37]	N	Y	N	N	N	N	N	N
Bai and Tan 2024 [18]	Y	Y	N	N	N	N	N	N
This work	Y	Y	P	Y	Y	Y	Y	Y

Table 2 complements Table 1’s qualitative scope flags with the quantitative figures each cited study actually reports. The comparison is necessarily asymmetric: Al Sayed et al.’s review finding that only 23% of 135 DRL-HVAC studies report a real-building deployment [11] is itself evidence that energy, comfort, hardware latency and transfer scope are rarely reported together in one study, which is the gap this paper’s results (Section 4.1, Section 4.2, Section 4.3, Section 4.4, Section 4.5, Section 4.6 and Section 4.7) are built to close. Cells marked “not reported” are not a methodological choice by the present authors. They reflect what the cited source does or does not publish, and no number is estimated or inferred on a cited study’s behalf where its source does not state one. The present paper’s own energy figures are a premium over a matched-building PPO baseline rather than a saving (Section 4.1), which should not be read as an apples-to-apples ranking against the energy-saving percentages reported by Gupta et al. and Yu et al.: those studies reported cost and energy reduction relative to a traditional thermostat under different building models, climates and reward formulations, so the figures bound the range that the literature reports rather than support a superiority claim in either direction.

Table 2. Quantitative comparison with key DRL-HVAC, MPC and Sinergym benchmark studies cited in Section 2, limited to figures each source actually reports. “Not reported” means the cited source does not publish that figure, not that it was not measured here.

Study	Energy/Comfort	Hardware	Inference Latency	Climate Transfer
This work	+1.7–6.8% energy vs. PPO at 100.00% CZCR (sim, KL/NY), 86–100% CZCR (HIL)	Raspberry Pi 4 + ESP32, <US$75 BOM	112.85 ms mean, 116.3/120.4 ms p95/p99	6 climates, zero-shot
Bai and Tan 2024 [18] (DAL-PPO)	Not reported (different reward/building, Section 2.5)	Not reported (sim. only)	Not reported	Not reported
Manjavacas et al., 2024 [19] (PPO/TD3/SAC)	PPO most stable, no single figure published	Not reported (sim. only)	Not reported	Not reported
Gupta et al., 2021 [27]	5–12% energy-cost reduction vs. thermostat	Not reported (sim. only)	Not reported	Not reported
Yu et al., 2020 [45]	8.1–15.2% cost savings (DDPG)	Not reported (sim. only)	Not reported	Not reported
Rastad et al., 2026 [25] (112 papers)	Up to 56.3% energy, 55% cost savings (aggregate)	Not reported (Section 2)	Not reported	Not reported

Table 2. Quantitative comparison with key DRL-HVAC, MPC and Sinergym benchmark studies cited in Section 2, limited to figures each source actually reports. “Not reported” means the cited source does not publish that figure, not that it was not measured here.

Study	Energy/Comfort	Hardware	Inference Latency	Climate Transfer
This work	+1.7–6.8% energy vs. PPO at 100.00% CZCR (sim, KL/NY), 86–100% CZCR (HIL)	Raspberry Pi 4 + ESP32, <US$75 BOM	112.85 ms mean, 116.3/120.4 ms p95/p99	6 climates, zero-shot
Bai and Tan 2024 [18] (DAL-PPO)	Not reported (different reward/building, Section 2.5)	Not reported (sim. only)	Not reported	Not reported
Manjavacas et al., 2024 [19] (PPO/TD3/SAC)	PPO most stable, no single figure published	Not reported (sim. only)	Not reported	Not reported
Gupta et al., 2021 [27]	5–12% energy-cost reduction vs. thermostat	Not reported (sim. only)	Not reported	Not reported
Yu et al., 2020 [45]	8.1–15.2% cost savings (DDPG)	Not reported (sim. only)	Not reported	Not reported
Rastad et al., 2026 [25] (112 papers)	Up to 56.3% energy, 55% cost savings (aggregate)	Not reported (Section 2)	Not reported	Not reported

3. Materials and Methods

The proposed system is a three-tier HVAC controller in which a digital twin, an edge AI agent, and a cyber-physical actuation loop cooperate to deliver thermal comfort and bounded energy use. This section is organised into five subsections that follow the system data flow, which include simulation environment (Section 3.1), recurrent PPO controller (Section 3.2), RBC safety shield (Section 3.3), edge-stack implementation (Section 3.5) and Unity digital twin interface (Section 3.6). Figure 1 summarises the overall control dataflow, from the recurrent policy’s decision to the RBC-mediated actuator command.

Figure 1. Conceptual dataflow of the hybrid LSTM–PPO–RBC controller. The recurrent PPO policy (whose policy network contains an LSTM core) decides the temperature setpoints, and the RBC layer enforces the final safety-bounded actuator command.

3.1. Three-Tier System Architecture and Sinergym/EnergyPlus Environment

The three tiers of the system consist of a sensor/actuator tier, an edge computing tier and an immersive monitoring tier. The first tier consists of the ESP32-WROOM-32 nodes (sourced via Shopee, Malaysia) with SHT30 temperature and humidity sensors (sourced via Shopee, Malaysia), an eight-channel 5 V relay board, and a 12 V Peltier plus PTC heat-pump load (sourced via Shopee, Malaysia). The second tier, an edge computing tier, is made of a Raspberry Pi 4 Model B, 4 GB RAM, BCM2711 SoC, running the Python 3.11 control loop and a Mosquitto MQTT broker over TLS 1.2. The third tier is an immersive monitoring tier that uses a Unity 6 application running on a tablet, receiving live MQTT state and dispatching HMI overrides. The tiers are connected by a TLS-secured MQTT topic hierarchy under the namespace wansoon/{home,status,actuators}. Figure 2 shows the resulting three-layer system architecture: a perception/physical layer, an edge intelligence layer running the Mosquitto broker and the Python control agent, and a virtual/application layer hosting the Unity digital twin.

Figure 2. Three-layer system architecture. The perception layer (sensor and ESP32 node) and the application layer (user and Unity digital twin) communicate with the edge intelligence layer (Raspberry Pi 4, Mosquitto MQTT broker, Python hybrid AI agent) over MQTT.

The simulation environment is the five-zone commercial building archetype, a single-storey small-office prototype of 463.6 m² (5ZoneAutoDXVAV.epJSON) distributed with Sinergym. EnergyPlus [20] is the simulation engine, and the Sinergym wrapper [19,21,22] exposes it as a gymnasium environment. The archetype is a packaged variable-air-volume (VAV) air-handling system with a two-speed direct expansion (DX) cooling coil, a heating coil, and VAV reheat terminals, served by a dual-setpoint thermostat. The building archetype is an office prototype, not a residential model, and the implication for scope is discussed in Section 5.3. The control actions are the heating and cooling thermostat setpoints of the representative zone (SPACE5-1), exercised through the thermostat schedules. This is the only zone the recurrent policy actuates. The other four zones of the archetype follow Sinergym’s default thermostat schedules, independent of the policy, consistent with the single-enclosure physical rig, which has one actuation point. Comfort and energy metrics are therefore reported for SPACE5-1 only. They do not characterise the other four zones, which are not under RL control. The observation vector is a 17-element Sinergym state comprising the month, day and hour, the outdoor temperature, relative humidity, wind speed, wind direction, and diffuse and direct solar radiation, the representative zone’s indoor air temperature, relative humidity and occupant count, the current heating and cooling setpoints, the total CO₂ emission, the whole-building HVAC electricity demand rate, and a separate cumulative total-HVAC-electricity meter. An earlier draft of this manuscript stated 16 elements, omitting the cumulative meter. The policy’s saved observation space, verified directly from the archived checkpoints (Box(-5e7, 5e7, (17,)), i.e., bounds of $-5\times {10}^7$ to $5\times {10}^7$), confirms 17. Sinergym is configured for a 15 min simulation step over a one-year horizon (35,040 steps per run). The action space used by all AI controllers is the continuous dual setpoint vector $[T^{\mathrm{heat}},T^{\mathrm{cool}}]$ with $T^{\mathrm{heat}}\in [12.0, 23.25]$ °C and $T^{\mathrm{cool}}\in [23.5, 30.0]$ °C, which the recurrent policy addresses through a normalised ${[-1, 1]}^2$ encoding. Six climate profiles are used, one per evaluation run: the New York TMY3 (USA_NY_New.York-J.F.Kennedy.Intl.AP.744860_TMY3.epw, the default weather of this environment), a Kuala Lumpur climate file (KualaLumpur.epw, the training climate), a Tokyo IWEC profile (JPN_Tokyo.Hyakuri.477150_IWEC.epw), and three project-provided climate files (Egypt.epw, Iceland.epw, Saudi.epw). The repository vendors a locally modified copy of Sinergym 3.10.6 with four project-added weather files (KualaLumpur.epw, Egypt.epw, Iceland.epw, Saudi.epw) not present in the public upstream project. An earlier draft cited this as upstream commit f92ac94ecb9881e6a834f6b79853b5020ab19add, dated December 2024. The commit itself is dated 9 December 2025. It also differs from the vendored copy in one respect relevant to Section 5.3: the upstream commit adds an explicit np_random generator argument for reproducible weather-file sampling that the project’s vendored copy predates. This is consistent with, though not a full explanation of, the disclosed training-seed reproducibility issue. EnergyPlus is version 23.1.0 for all originally reported single-run results (Section 4). The ten-seed re-run (Section 4.2) and the matched-climate PPO/DQN retraining (Section 4.3), both added for this revision, were run under EnergyPlus 24.1.0 instead, the version the vendored Sinergym 3.10.6 actually requires. This version split, and its implications, are discussed in Section 5.3. The first 96 steps (24 h) are excluded as warm-up. All reported simulation metrics use the post-warm-up window of 34,944 steps. The comfort band is [23, 26] °C, and the comfort zone compliance rate (CZCR) is the percentage of post-warm-up steps within this band. Detailed environment parameters are summarised in Table 3.

Table 3. Sinergym/EnergyPlus simulation environment parameters and per-climate weather files.

Parameter	Value
Building archetype	5ZoneAutoDXVAV.epJSON (five-zone office archetype, VAV + two-speed DX cooling coil, electric heat, VAV reheat)
Representative control zone	SPACE5-1
Simulation step/horizon	15 min, 1 year (35,040 steps)
Warm-up steps excluded	96 (24 h), metrics on 34,944 post-warm-up steps
Comfort band (CZCR)	23.0–26.0 °C
Action space	Dual thermostat setpoints: $T^{\mathrm{heat}}\in [12.0, 23.25]$ °C, $T^{\mathrm{cool}}\in [23.5, 30.0]$ °C
Training climate	KualaLumpur.epw (KL)
Evaluation climates	New York TMY3 (USA_NY_New.York-J.F.Kennedy.Intl.AP.744860_TMY3.epw), Tokyo IWEC (JPN_Tokyo.Hyakuri.477150_IWEC.epw), and project files Egypt.epw, Iceland.epw and Saudi.epw
EnergyPlus version	23.1.0 for all originally reported single-run results; 24.1.0 for the ten-seed re-run (Section 4.2) and matched-climate retraining (Section 4.3) added in this revision (Section 5.3)
Sinergym version	3.10.6, locally modified (vendored in repository)

Table 3. Sinergym/EnergyPlus simulation environment parameters and per-climate weather files.

Parameter	Value
Building archetype	5ZoneAutoDXVAV.epJSON (five-zone office archetype, VAV + two-speed DX cooling coil, electric heat, VAV reheat)
Representative control zone	SPACE5-1
Simulation step/horizon	15 min, 1 year (35,040 steps)
Warm-up steps excluded	96 (24 h), metrics on 34,944 post-warm-up steps
Comfort band (CZCR)	23.0–26.0 °C
Action space	Dual thermostat setpoints: $T^{\mathrm{heat}}\in [12.0, 23.25]$ °C, $T^{\mathrm{cool}}\in [23.5, 30.0]$ °C
Training climate	KualaLumpur.epw (KL)
Evaluation climates	New York TMY3 (USA_NY_New.York-J.F.Kennedy.Intl.AP.744860_TMY3.epw), Tokyo IWEC (JPN_Tokyo.Hyakuri.477150_IWEC.epw), and project files Egypt.epw, Iceland.epw and Saudi.epw
EnergyPlus version	23.1.0 for all originally reported single-run results; 24.1.0 for the ten-seed re-run (Section 4.2) and matched-climate retraining (Section 4.3) added in this revision (Section 5.3)
Sinergym version	3.10.6, locally modified (vendored in repository)

3.2. Recurrent PPO Controller (LSTM–PPO)

The high-performance controller is a recurrent proximal policy optimisation agent [10], implemented with the RecurrentPPO (MlpLstmPolicy) class of stable-baselines3-contrib [13]. PPO itself is a first-order simplification of TRPO [31], where TRPO enforces a trust region constraint with a bound on the per-update policy divergence. The PPO replaces the constrained optimisation with the clipped surrogate objective of Equation (2), trading the monotonic improvement guarantee for per-update simplicity, which is the property that makes PPO practical on embedded hardware. The policy network is a multi-layer perceptron whose decision core is an LSTM recurrent layer [39,40] (hidden size 256, the stable-baselines3-contrib default), which carries thermal memory across control steps. At inference the policy is run deterministically, with its recurrent state (lstm_states) threaded across the episode. The hyperparameters are a learning rate of $3\times {10}^{-4}$, a rollout length of $n_{\mathrm{steps}}=2048$, a minibatch size of 64, an entropy coefficient of 0.01, and the stable-baselines3 defaults for the remaining parameters (discount factor $\gamma =0.99$, generalised advantage estimation parameter $\lambda =0.95$, clip ratio $ϵ=0.2$). Training runs for 300,000 timesteps on the Kuala Lumpur climate file. The headline figures throughout Section 4 and Section 5 come from a single disclosed training run: no random seed was threaded into the environment constructor in that run, and an archived multi-seed CSV purporting to cover this has ten identical rows, which the on-disk reproduction audit traces to a defect in the evaluation script that produced it (Section 5.3) rather than to the training itself. Section 4.2 reports a corrected ten-seed re-run (seeds explicitly threaded this time) on this training climate: CZCR and violations are seed-invariant (100.00%, 0, in all ten seeds) and energy varies by a 2.3% coefficient of variation. The cross-climate transfer results (Section 4.4) and the PPO/DQN baselines remain single-run.

The training reward, computed by the environment wrapper on every step, is the difference of a comfort term and a normalised energy penalty:

$$r_t=10·\exp \left(-0.5{\left({\displaystyle \frac{T_t-24.5}{1.0}}\right)}^2\right)-0.5·{\displaystyle \frac{P_t}{10,000}},$$

(1)

where $T_t$ is the representative-zone indoor air temperature at step t and $P_t$ is the whole-building HVAC electricity demand rate in watts. The comfort term is a Gaussian-shaped reward centred on 24.5 °C with a 1 °C width [51,52], and the energy term subtracts the normalised HVAC power draw as per-step power in the evaluation traces reaches approximately 20,000 W, so the energy term can dominate and no clipping is applied.

The PPO clipped surrogate objective is

$$L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_t\left[\min \left({\rho }_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left({\rho }_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right],$$

(2)

where ${\rho }_t\left(\theta \right)$ is the probability ratio between the new and old policies, and the advantage is estimated by generalised advantage estimation (GAE), as in Equation (3):

$${\widehat{A}}_t={\displaystyle \sum _{l=0}^{\infty }}{\left(\gamma \lambda \right)}^l{\delta }_{t+l},{\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right).$$

(3)

3.3. RBC Safety Shield (Simplex Floor)

The controller follows the Simplex architecture, originally proposed by Seto et al. [14] and recently adapted for learning-based autonomous systems by Salamini et al. [15], in which a high-performance but unverified AI agent is supervised by a simple, verifiable safety monitor. In the safe-reinforcement-learning literature this pattern corresponds to runtime shielding as Alshiekh et al. [16] formalise a shield that intercepts actions violating a safety specification before they reach the plant, and García and Fernández [17] formulate the safe exploration principle that a learning agent must be prevented from entering catastrophic states during training and deployment alike. Here the safety monitor is the two-sided temperature predicate

$$\mathrm{unsafe}\left(s_t\right)=\left(T_t^{\mathrm{in}}>25.8\right)\vee \left(T_t^{\mathrm{in}}<23.2\right).$$

(4)

On activation, the recurrent policy’s decision is overridden by a fixed dead-band rule that issues the action $[T^{\mathrm{heat}}, T^{\mathrm{cool}}]=[23.25, 24.0]$ °C when the room is too warm and $[23.25, 28.0]$ °C when the room is too cool, irrespective of the recommended AI action. Between the triggers, the AI action passes through unchanged. The dead-band is deliberately asymmetric relative to the 23–26 °C comfort band: both triggers sit 0.2 °C inside the band, so the shield fires before a violation occurs rather than after. The shield is evaluated on the perceived (sensor) temperature in both simulation and hardware, which is the property that makes it effective under sensor faults (Section 4.6).

What the shield guarantees, and under what assumptions. Equation (4) is a trigger predicate, not a formal proof of closed-loop safety, and the two should not be conflated. What it guarantees is narrower: whenever the sensed temperature $T_t^{\mathrm{in}}$ crosses a trigger, the controller’s command at the next control step is replaced by the fixed override, provided (i) the sensor reading is not itself biased away from the true zone temperature by more than the 0.2 °C margin the triggers sit inside the comfort band, and (ii) the actuator applies the override command within one control step (15 min in simulation, approximately 30 s on hardware). It does not guarantee that the zone temperature remains inside $[23.0, 26.0]$ °C after the override is issued: Section 4.4 shows that once triggered, the zone can continue drifting for a further 10–47 steps (up to 11.75 h) because of command-to-zone thermal lag, so the shield bounds when intervention begins, not the resulting excursion depth or duration. Phrases such as “safe by construction” or “hard safety floor” used elsewhere in earlier drafts of this work overstated this guarantee and have been removed accordingly.

3.4. Shield Pseudocode and Exact Boundary Behaviour

Algorithm 1 gives the shield as implemented, using strict inequalities throughout (Equation (4)). Four boundary temperatures illustrate its behaviour. At $T_t^{\mathrm{in}}=23.0$ °C (the comfort floor), the shield is already active, since $23.0<23.2$ is true and the too-cool override is in effect. At $T_t^{\mathrm{in}}=23.2$ °C (the lower trigger itself), the shield is inactive, since the strict inequality $23.2<23.2$ is false. The AI action therefore passes through unchanged exactly at the trigger value, and the override engages only strictly below it. Symmetrically, at $T_t^{\mathrm{in}}=25.8$ °C (the upper trigger itself), the shield is inactive for the same reason, engaging only strictly above it. At $T_t^{\mathrm{in}}=26.0$ °C (the comfort ceiling), the shield is already active, since $26.0>25.8$ is true. Both comfort-band edges (23.0 and 26.0 °C) therefore fall inside the shield’s active region, while the triggers themselves (23.2 and 25.8 °C) are the exact hand-off points between AI control and override, consistent with the 0.2 °C margin described in Section 3.3.

Algorithm 1 Two-sided Simplex shield (deployed configuration)

Require: perceived indoor temperature $T_t^{\mathrm{in}}$ and AI action $a_t^{\mathrm{AI}}=[T_{\mathrm{AI}}^{\mathrm{heat}},T_{\mathrm{AI}}^{\mathrm{cool}}]$ 1: if $T_t^{\mathrm{in}}<23.2$  then                                                                                       ▹too cool 2:     issue $[T^{\mathrm{heat}}, T^{\mathrm{cool}}]\leftarrow [23.25, 28.0]$ 3: else if $T_t^{\mathrm{in}}>25.8$ then                                                                           ▹ too warm 4:     issue $[T^{\mathrm{heat}}, T^{\mathrm{cool}}]\leftarrow [23.25, 24.0]$ 5: else                                                                  ▹$23.2\le T_t^{\mathrm{in}}\le 25.8$: shield inactive 6:     issue $a_t^{\mathrm{AI}}$ unchanged 7: end if

3.5. Edge-Stack Implementation (Raspberry Pi 4 and ESP32)

The edge stack runs on a Raspberry Pi 4 (BCM2711, 4 GB RAM, Raspbian GNU/Linux 12) with a Python 3.11 virtual environment. The control loop polls the SHT30 sensors at 1 Hz and issues a new controller action approximately every 30 s. This is reflected in the on-disk telemetry that shows the median interval between logged control actions as 33 s, and each logged inference completes in a mean of 112.85 ms (median 113.2 ms, standard deviation (SD) 19.1 ms, range 7.3–414.3 ms, $n=1392$ invocations). On-device recurrent PPO inference is implemented with stable-baselines3 2.7.1 [13] and PyTorch 2.14.1 (CPU). The archived recurrent-policy checkpoint is 7.29 MB (ppo_lstm.zip) with a 2.4 MB exported weight file. The MQTT [53] broker is Mosquitto 2.0 with TLS 1.2 [54] server certificates. The topics are wansoon/home/#, wansoon/status/# and wansoon/actuators/#. HMI overrides from the Android tablet are dispatched on wansoon/actuators/override. The 15 sampled override events in the telemetry complete in a mean of 16.3 ms (SD 7.5 ms). The on-disk telemetry logs application-level events and inference latency but does not record Raspberry Pi CPU/RAM utilisation, MQTT packet loss counters, or missed control commands. These were not instrumented during the May 2026 trials and are reported as a gap in Section 5.3 rather than estimated post hoc. The deployment code initialises the LSTM recurrent state (lstm_states) to None and the episode-start flag to True once, at process start. The flag is set to False after the first inference call and never reset thereafter, so the recurrent state persists continuously across the entire boot-to-shutdown session (not just the reported 30 min trial windows) rather than being reset at any fixed interval.

A 12 min combined outage (2026-05-02 22:44:16 to 22:56:58) recorded in the on-disk telemetry is analysed in Section 4.7. The router is power-cycled with the Wi-Fi link drops, and the MQTT client reconnects after a 12 min 42 s gap, and the indoor temperature moves by $+0.06$ °C across the outage (from 25.75 °C to 25.81 °C), i.e., the enclosure stays within the comfort band throughout the outage.

Figure 3 shows the wiring schematic of the testbed (Figure 3a) and the assembled physical HIL testbed (Figure 3b) used for all hardware trials: the 0.5 m³ enclosure houses the Peltier module with heatsink and fan, the SHT30 sensor, and the ESP32/Raspberry Pi control electronics, with a live LCD status readout.

Figure 3. The hardware-in-the-loop (HIL) testbed. (a) Wiring schematic of the Peltier module, SHT30 sensor, ESP32 nodes, relay board, and Raspberry Pi 4 control electronics. (b) The assembled 0.5 m³ enclosure containing the Peltier cooling module with heatsink and fan, the SHT30 temperature/humidity sensor, and the Raspberry Pi/ESP32 control electronics. The LCD panel displays live AI mode, temperature, and humidity readings.

3.6. Unity Digital-Twin Interface

The Unity 6 application subscribes to the same MQTT topic hierarchy and renders a real-time 3D scene of the controlled space with a live temperature trace and an actuator override panel. The application is implemented as a single C# script bound to the MQTT client. The 3D assets are lightweight primitives (cube rooms, sphere sensors, line connectors) to keep the GPU footprint small on the tablet. The bidirectional control loop proceeds as follows.

• The Unity scene receives the live state.
• The user touches an actuator in the scene.
• The touch event is published on wansoon/actuators/override.
• The Pi 4 receives the override and applies it.
• The Pi 4 publishes the new state.
• The Unity scene reflects the new state within one MQTT round trip.

Figure 4 shows the 3D model of the controlled space as rendered in the Unity digital twin. The lightweight primitive-based scene mirrors the live MQTT state of the enclosure and its actuators and serves as the immersive front end of the digital twin through which the tablet user monitors the room and dispatches HMI overrides.

Figure 4. The Unity 6 digital twin 3D model of the controlled space as rendered on the tablet: the lightweight primitive-based scene mirrors the live MQTT state of the enclosure and its actuators and hosts the actuator override panel through which HMI overrides are dispatched.

4. Results

This section reports the simulation, cross-climate transfer, shield-bound, sensor-fault, ablation, and physical testbed results for the proposed hybrid LSTM–PPO–RBC controller. All simulation energy and CZCR values are re-derived from the per-step evaluation traces over the post-warm-up window of 34,944 steps ($\Delta t=0.25$ h). Because training is a single run, except where Section 4.2’s ten-seed re-run on the Kuala Lumpur trace applies, every percentage in this section is a descriptive point estimate: no formal statistical hypothesis testing, confidence intervals, or multiple-comparisons correction are applied, and the differences between controllers should be read as single-run observations, not as statistically validated effects (Section 5.3).

4.1. Energy and Comfort Baselines

Table 4 reports the annual energy and CZCR for the four controller families on the New York, Kuala Lumpur and Tokyo traces. Each AI controller is identified by its training climate (the standard PPO and DQN baselines). The DQN [7] is included as the canonical value-based deep controller. It was trained on the New York profile (the environment default) and evaluated zero-shot on the Kuala Lumpur and Tokyo profiles, whereas the hybrid was trained on the Kuala Lumpur profile and evaluated zero-shot on New York and Tokyo. The RBC and PID baselines are rule-based and require no training. On the New York trace the standard PPO baseline consumes 36,376.16 kWh at 99.72% CZCR (98 violations) and the RBC baseline is 5.8% more energy efficient (34,268.07 kWh) but admits 227 violations. The hybrid records the highest energy of the New York controllers (36,982.98 kWh, $+1.7$% vs. PPO) but the best comfort: 100.00% CZCR with 0 violations. On the Kuala Lumpur trace the hybrid ties for best comfort (100.00%, 0 violations) at a $+6.8$% energy premium over the zero-shot PPO baseline. On the Tokyo trace, the pattern repeats: 99.50% CZCR (173 violations) at a $+3.1$% energy premium over the zero-shot PPO baseline. These zero-shot comparisons, however, are not the full picture on Kuala Lumpur, where a genuinely matched-climate baseline is now available (Section 4.3). A standard PPO policy trained directly on Kuala Lumpur, rather than zero-shot transferred from New York, also reaches 100.00% CZCR with 0 violations, and does so at 28,613.99 kWh, $13.2$% less energy than the hybrid’s 32,956.07 kWh for the identical comfort outcome. A matched-climate DQN policy reaches 99.32% CZCR (237 violations) at 28,016.62 kWh, $15.0$% less energy than the hybrid. On this climate, once the training climate confound is removed, the hybrid’s energy premium is not modest. It is a real cost relative to a correctly trained simple baseline, not an artefact of comparing against a baseline trained on the wrong climate. The hybrid’s claim to value in this comparison is therefore not best-comfort-at-lowest-cost on a single climate. It is that the same single trained policy, without retraining, reaches comparable comfort across six climates (Section 4.4), which the matched-climate PPO and DQN baselines, trained once per climate, do not attempt to do. The energy premium relative to a same-training climate PPO baseline is $+1.7$% on New York and $+3.1$% on Tokyo (both zero-shot comparisons, Section 5.3), and, on Kuala Lumpur, a $+15.2$% premium relative to the matched-climate PPO baseline rather than the previously reported $+6.8$% relative to the zero-shot one.

The five-configuration ablation in Table 5 (all configurations retrained on the Kuala Lumpur trace) confirms that the LSTM–PPO and full hybrid rows are numerically identical (33,066.99 kWh, 5 violations), because the safety shield never overrides the policy: the recurrent policy is already inside the safety envelope on its training climate. The $+24.4$% energy premium of the full hybrid over pure RBC is the cost of trading 3361 RBC violations for 5 LSTM–PPO violations in the same climate. The standard PPO and PPO + Shield rows are also identical (32,697.64 kWh, 5 violations), for the same reason. Two protocol differences with Table 4 should be noted when reading the two tables side by side: the ablation metrics are accumulated over the full 35,040-step episode without the 96-step warm-up exclusion used for the headline tables, and the ablation’s Pure RBC row is a fixed thermostat rule (heating 22 °C, cooling 25 °C) rather than the state-dependent RBC baseline of Table 4, which is why its energy and violation counts differ from the RBC row there. Under the archived per-step Kuala Lumpur trace, the five full-window violations of the full hybrid all fall within the excluded 96-step warm-up window, consistent with the 0 post-warm-up violations reported in Table 4.

Table 4. Annual simulation energy and CZCR by controller and climate, re-derived from per-step traces over the post-warm-up window of 34,944 steps. “ZS” marks a zero-shot evaluation of a policy trained on a different climate: PPO and DQN were trained on New York, the hybrid on Kuala Lumpur. “Matched” marks a new same-climate baseline, trained on Kuala Lumpur for this revision (Section 4.3), to de-confound the training climate effect from the algorithm effect on that trace. The bottom rows are hybrid-only deployments for which no baseline traces exist on disk.

Controller (Train Climate)	Energy (kWh)	CZCR (%)	Violations	$\Delta$ vs. PPO (%)
New York (NY) trace
RBC	34,268.07	99.35	227	$-5.8$
PID	34,566.82	94.73	1842	$-5.0$
DQN (NY)	35,633.44	99.44	195	$-2.0$
Standard PPO (NY)	36,376.16	99.72	98	0.0
Hybrid (KL, ZS)	36,982.98	100.00	0	$+1.7$
Kuala Lumpur (KL) trace
RBC	27,086.72	93.56	2251	$-12.2$
PID	27,557.38	88.76	3928	$-10.7$
DQN (NY, ZS)	28,519.86	91.94	2818	$-7.6$
DQN (KL, matched)	28,016.62	99.32	237	$-9.2$
Standard PPO (NY, ZS)	30,859.69	99.17	290	0.0
PPO (KL, matched)	28,613.99	100.00	0	$-7.3$
Hybrid (KL)	32,956.07	100.00	0	$+6.8$
Tokyo trace
RBC	32,872.95	98.59	493	$-4.5$
PID	33,350.93	94.59	1889	$-3.1$
DQN (NY, ZS)	33,758.37	96.82	1110	$-1.9$
Standard PPO (NY, ZS)	34,424.77	98.95	368	0.0
Hybrid (KL, ZS)	35,489.79	99.50	173	$+3.1$

Table 4. Annual simulation energy and CZCR by controller and climate, re-derived from per-step traces over the post-warm-up window of 34,944 steps. “ZS” marks a zero-shot evaluation of a policy trained on a different climate: PPO and DQN were trained on New York, the hybrid on Kuala Lumpur. “Matched” marks a new same-climate baseline, trained on Kuala Lumpur for this revision (Section 4.3), to de-confound the training climate effect from the algorithm effect on that trace. The bottom rows are hybrid-only deployments for which no baseline traces exist on disk.

Controller (Train Climate)	Energy (kWh)	CZCR (%)	Violations	$\Delta$ vs. PPO (%)
New York (NY) trace
RBC	34,268.07	99.35	227	$-5.8$
PID	34,566.82	94.73	1842	$-5.0$
DQN (NY)	35,633.44	99.44	195	$-2.0$
Standard PPO (NY)	36,376.16	99.72	98	0.0
Hybrid (KL, ZS)	36,982.98	100.00	0	$+1.7$
Kuala Lumpur (KL) trace
RBC	27,086.72	93.56	2251	$-12.2$
PID	27,557.38	88.76	3928	$-10.7$
DQN (NY, ZS)	28,519.86	91.94	2818	$-7.6$
DQN (KL, matched)	28,016.62	99.32	237	$-9.2$
Standard PPO (NY, ZS)	30,859.69	99.17	290	0.0
PPO (KL, matched)	28,613.99	100.00	0	$-7.3$
Hybrid (KL)	32,956.07	100.00	0	$+6.8$
Tokyo trace
RBC	32,872.95	98.59	493	$-4.5$
PID	33,350.93	94.59	1889	$-3.1$
DQN (NY, ZS)	33,758.37	96.82	1110	$-1.9$
Standard PPO (NY, ZS)	34,424.77	98.95	368	0.0
Hybrid (KL, ZS)	35,489.79	99.50	173	$+3.1$

Table 5. Five-configuration ablation, all configurations trained and evaluated on the Kuala Lumpur trace. Metrics are accumulated over the full 35,040-step episode without the 96-step warm-up exclusion used in Table 4. The full hybrid and LSTM–PPO rows are numerically identical because the safety shield never fires in this configuration.

Configuration	Energy (kWh)	Violations	$\Delta$ vs. Pure RBC (%)
Pure RBC (KL trace)	26,589.53	3361	0.0
Standard PPO (KL trace)	32,697.64	5	$+23.0$
PPO + Shield (KL trace)	32,697.64	5	$+23.0$
LSTM–PPO (KL trace)	33,066.99	5	$+24.4$
Full Hybrid (KL trace)	33,066.99	5	$+24.4$

Table 5. Five-configuration ablation, all configurations trained and evaluated on the Kuala Lumpur trace. Metrics are accumulated over the full 35,040-step episode without the 96-step warm-up exclusion used in Table 4. The full hybrid and LSTM–PPO rows are numerically identical because the safety shield never fires in this configuration.

Configuration	Energy (kWh)	Violations	$\Delta$ vs. Pure RBC (%)
Pure RBC (KL trace)	26,589.53	3361	0.0
Standard PPO (KL trace)	32,697.64	5	$+23.0$
PPO + Shield (KL trace)	32,697.64	5	$+23.0$
LSTM–PPO (KL trace)	33,066.99	5	$+24.4$
Full Hybrid (KL trace)	33,066.99	5	$+24.4$

4.2. Multi-Seed Re-Run on the Training Climate

The headline hybrid figures above come from the single disclosed training run (Section 3.2). To quantify seed sensitivity, ten additional training runs were carried out with the exact architecture, hyperparameters and reward function of Section 3.2 (300,000 timesteps each, Kuala Lumpur trace), differing only in an explicit random seed (101–110) passed to the policy constructor. This is the corrected version of the seeding step the single disclosed run omitted. Each resulting policy was evaluated deterministically for the same 35,040-step protocol with the deployed two-sided shield active, using the same post-warm-up convention as Table 4. This is distinct from, and supersedes for this purpose, an earlier on-disk multi_seed_models/ artifact: those ten checkpoints turned out to use a smaller, single-dimension action wrapper trained for only 80,000 timesteps, architecturally different from the deployed policy, and are not used here or elsewhere in this manuscript.

Table 6 reports the ten re-runs. CZCR is 100.00% and violations are 0 in every one of the ten seeds: on the training climate, comfort-band compliance with the two-sided shield active is not seed-sensitive within this sample. Energy is seed-sensitive but modestly so: mean 29,332.68 kWh, SD 673.30 kWh (coefficient of variation 2.3%), range 28,238.50–30,442.83 kWh across the ten seeds. The originally reported single-run hybrid (KL) energy figure in Table 4, 32,956.07 kWh, sits above this ten-seed range. The single disclosed run was not a typical draw from the seed distribution observed here, and readers should treat Table 4’s KL energy figure, and the other single-run energy comparisons that depend on it, with that in mind. This multi-seed re-run covers the Kuala Lumpur training climate only. The zero-shot transfer climates (Section 4.4) and the PPO/DQN baselines (Table 4) remain single-run estimates, and multi-seed coverage of those is identified as future work (Section 5.3). Total compute for the ten re-runs was 211.7 min, using EnergyPlus 24.1.0, single-threaded per run. See the repository’s repro/train_eval_new_seed.py.

Table 6. Ten independently seeded re-runs of the hybrid policy (identical architecture, hyperparameters and reward to the single disclosed run), trained and evaluated on the Kuala Lumpur trace with the two-sided shield active. Metrics use the same post-warm-up (34,944-step) convention as Table 4.

Seed	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
101	29,920.68	0	100.00	23.21–24.36
102	30,442.83	0	100.00	23.21–24.25
103	28,238.50	0	100.00	23.22–24.74
104	29,245.43	0	100.00	23.22–24.51
105	28,910.26	0	100.00	23.22–24.58
106	28,893.03	0	100.00	23.22–24.59
107	28,714.84	0	100.00	23.22–24.63
108	29,276.75	0	100.00	23.22–24.50
109	29,945.04	0	100.00	23.21–24.35
110	29,739.42	0	100.00	23.21–24.40
Mean ± SD ($n=10$)	29,332.68 ± 673.30	0 ± 0	100.00 ± 0.00	n/a

Table 6. Ten independently seeded re-runs of the hybrid policy (identical architecture, hyperparameters and reward to the single disclosed run), trained and evaluated on the Kuala Lumpur trace with the two-sided shield active. Metrics use the same post-warm-up (34,944-step) convention as Table 4.

Seed	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
101	29,920.68	0	100.00	23.21–24.36
102	30,442.83	0	100.00	23.21–24.25
103	28,238.50	0	100.00	23.22–24.74
104	29,245.43	0	100.00	23.22–24.51
105	28,910.26	0	100.00	23.22–24.58
106	28,893.03	0	100.00	23.22–24.59
107	28,714.84	0	100.00	23.22–24.63
108	29,276.75	0	100.00	23.22–24.50
109	29,945.04	0	100.00	23.21–24.35
110	29,739.42	0	100.00	23.21–24.40
Mean ± SD ($n=10$)	29,332.68 ± 673.30	0 ± 0	100.00 ± 0.00	n/a

4.3. Matched-Climate Baseline Retraining

Table 4’s Kuala Lumpur rows originally compared the hybrid, trained on Kuala Lumpur, against PPO and DQN baselines trained on New York and evaluated zero-shot. This conflates two effects in a single comparison: the algorithm (hybrid versus PPO versus DQN) and the training climate (matched versus zero-shot). To separate them, a standard PPO baseline and a DQN baseline were retrained for this revision, each trained directly on the Kuala Lumpur trace with the same wrapper, hyperparameters and timestep budget (200,000 timesteps) as the original New York-trained versions (Section 4.1), differing only in the training climate. Neither retrained baseline uses a random seed, matching the unseeded original PPO/DQN training runs and keeping the training climate variable isolated from the seed-variance question addressed separately in Section 4.2.

The matched-climate PPO baseline reaches 100.00% CZCR with 0 violations at 28,613.99 kWh. This matches the hybrid’s comfort outcome on this climate while using 13.2% less energy than the hybrid’s 32,956.07 kWh. The matched-climate DQN baseline reaches 99.32% CZCR (237 violations) at 28,016.62 kWh, 15.0% less energy than the hybrid. Both results are added to Table 4 alongside the original zero-shot rows. The implication is discussed where Table 4 is first introduced (Section 4.1): the hybrid’s reported energy premium over PPO on Kuala Lumpur is materially larger, not smaller, once the baseline is trained on the correct climate. This matched-climate retraining covers only the Kuala Lumpur trace. The New York and Tokyo zero-shot comparisons in Table 4 are unaffected and remain zero-shot. Both baselines were trained and evaluated using EnergyPlus 24.1.0, the same simulation-engine version as the ten-seed re-run (Section 4.2) and for the same reason (Section 5.3). Full training and evaluation code is in the repository’s repro/train_eval_matched_climate.py.

4.4. Cross-Climate Zero-Shot Transfer

The recurrent policy is trained once on the Kuala Lumpur climate and then deployed, without any retraining or fine-tuning, on five further climate profiles (Table 7). This is zero-shot transfer across thermal distributions, the building-control analogue of the Sim-to-Real transfer problem surveyed by Zhao et al. [23]: the policy is trained on one climate’s dynamics and evaluated on unseen ones, and robustness degrades as the target distribution departs from the training one. The per-step evaluation shows 100.00% CZCR on New York (0 violations in 34,944 post-warm-up steps, indoor temperature 23.17–23.75 °C), 99.50% on Tokyo (173 violations, 22.76–23.74 °C), 99.65% on Iceland (121 violations), 97.24% on Saudi Arabia (966 violations) and 93.50% on Egypt (2271 violations). The worst transfer outcome is the hot desert climate (Egypt), where the policy overcools below the 23.0 °C comfort floor (minimum 22.37 °C). The best are the training climate itself and New York, the temperate profile whose cooling-demand distribution is closest to the training one. Zero violations are recorded on two of the six evaluated climates (New York and Kuala Lumpur, with the shield active in both cases). Tokyo and Iceland come within 173 and 121 steps of zero, respectively. The interpretation is a per-climate decomposition: the KL-trained policy is robust in tropical, temperate and cold climates but is least compliant in the hot desert climate, where the cooling demand profile is furthest from the training distribution.

The residual violations admit a direct mechanistic explanation from the per-step trace logs, which record a mode flag (LSTM_AI or SHIELD_ACTIVE) at every step. Over the post-warm-up window the shield fires in 671 steps on Tokyo, 4073 on Egypt, 2058 on Saudi Arabia, 242 on Iceland and 5 on New York (0 on Kuala Lumpur), and every violation step in every climate falls inside a shield-active interval. None occurs while the policy is in control. The shield therefore detects the excursion immediately (median temperature at shield onset 23.16–23.18 °C, i.e., within 0.05 °C of the 23.2 °C trigger) but cannot arrest it: the fixed override (heating setpoint 23.25 °C, cooling 28.0 °C) acts on the thermostat command, and the zone temperature continues to fall for roughly ten further steps before turning around. In the longest Egyptian excursion, the shield fires at 23.195 °C and the zone continues cooling to 22.69 °C over the next ten steps, then requires a further forty steps to climb back above 23.0 °C, a 51-step (12.75 h) round trip in total under an active shield. Violations are consequently sustained runs rather than single-step overshoot: the median violation run lasts 11 steps on Tokyo and Iceland, 20 on Saudi Arabia and 26 on Egypt (maxima 25, 44, 40 and 47 steps, i.e., 6.25–11.75 h), which is why the hot desert climates, with the largest cooling demand and the deepest excursion into the override envelope, accumulate the most violation steps. The mechanism is thus the command-to-zone thermal lag between the 15 min shield intervention and the zone’s thermal response, not a failure of shield detection. A faster-acting or state-dependent override (for example, raising the heating setpoint above 23.25 °C in proportion to the violation depth) is the obvious engineering remedy, and is left as future work. These counts are descriptive single-run statistics (Section 5.3).

Table 7. Cross-climate zero-shot transfer of the Kuala Lumpur-trained hybrid policy, with no retraining in the target climate.

Climate	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
Kuala Lumpur (train)	32,956.07	0	100.00	23.20–23.74
New York	36,982.98	0	100.00	23.17–23.75
Tokyo	35,489.79	173	99.50	22.76–23.74
Iceland	46,907.45	121	99.65	22.43–23.76
Saudi Arabia	30,773.39	966	97.24	22.29–24.26
Egypt	29,273.39	2271	93.50	22.37–24.22

Table 7. Cross-climate zero-shot transfer of the Kuala Lumpur-trained hybrid policy, with no retraining in the target climate.

Climate	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
Kuala Lumpur (train)	32,956.07	0	100.00	23.20–23.74
New York	36,982.98	0	100.00	23.17–23.75
Tokyo	35,489.79	173	99.50	22.76–23.74
Iceland	46,907.45	121	99.65	22.43–23.76
Saudi Arabia	30,773.39	966	97.24	22.29–24.26
Egypt	29,273.39	2271	93.50	22.37–24.22

4.5. Shield-Bound Configuration: Why the Lower Trigger Is Load-Bearing

An archived evaluation trace on disk was produced with an earlier, upper-bound-only shield predicate: the shield fired when the perceived temperature exceeded 26.0 °C but had no lower trigger. On the Kuala Lumpur climate this configuration records 4124 comfort violations (88.20% CZCR), with the indoor temperature driven as low as 22.51 °C, 0.49 °C below the 23.0 °C comfort floor, because the recurrent policy’s overcool excursions fall just short of any lower-bound intervention. The deployed two-sided predicate (Equation (4)) eliminates this failure mode on the same climate: 0 violations and a narrower temperature range (23.20–23.74 °C), at essentially the same energy (32,956.07 vs. 33,003.47 kWh). The comparison is summarised in Table 8. The takeaway is a design lesson rather than a benchmark result: a one-sided Simplex monitor on an RL controller is safe only against the failure direction it monitors. The lower trigger, sitting 0.2 °C inside the comfort floor, is what removes the overcool failure mode.

Table 8. Shield-bound configuration comparison on the Kuala Lumpur trace. The upper-bound-only row is an archived earlier evaluation configuration, and the two-sided row is the deployed configuration.

Shield Configuration	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
Upper bound only (archived)	33,003.47	4124	88.20	22.51–24.60
Two-sided (deployed)	32,956.07	0	100.00	23.20–23.74

Table 8. Shield-bound configuration comparison on the Kuala Lumpur trace. The upper-bound-only row is an archived earlier evaluation configuration, and the two-sided row is the deployed configuration.

Shield Configuration	Energy (kWh)	Violations	CZCR (%)	T Range (°C)
Upper bound only (archived)	33,003.47	4124	88.20	22.51–24.60
Two-sided (deployed)	32,956.07	0	100.00	23.20–23.74

4.6. Sensor-Fault Stress Test

A sensor-fault stress test (the archived stress_test harness) injects a sudden $+4.0$ °C bias into the perceived indoor temperature for 501 consecutive steps (steps 1000–1500) of a 3000-step Kuala Lumpur evaluation, with the recurrent policy active outside the injection window and the safety supervisor active inside it. This is a deliberate worst-case comparison, not a fair head-to-head baseline: the test compares the deployed fixed dead-band supervisor (SmartShieldRBC) against a PID supervisor with a deliberately aggressive integral gain ($k_i=0.5$), chosen specifically to exhibit integral windup under sustained bias, so as to bound how badly an integral controller can behave under this fault rather than to claim the dead-band supervisor beats a well-tuned PID. Under the fixed dead-band supervisor, the actual indoor temperature stays within 23.24–24.00 °C throughout the injection window. Under the PID supervisor, the sustained bias winds up the integral term, the supervisor drives the cooling setpoint to its lower bound (23.5 °C), and the actual temperature oscillates over a 2.7 °C range (23.23–25.96 °C). Neither supervisor breaches the 26.0 °C comfort ceiling in this scenario (0 violations above 26 °C for both). The dead-band supervisor’s advantage is bounded overshoot rather than violation count. The result supports the design choice of a fixed dead-band safety monitor on the perceived temperature: its behaviour is invariant to the magnitude of a sustained sensor bias, whereas an integral controller’s behaviour is not.

4.7. Physical Hardware Validation

The physical testbed is a 0.5 m³ benchtop enclosure with a 12 V Peltier cooler and a PTC heater under closed-loop control. SHT30 sensors are polled at 1 Hz and controller actions are issued roughly every 30 s. Table 9 reports three repeated 30 min hybrid trials (6, 7 and 8 May 2026), filtered from the on-disk telemetry as the cleanest 30 min window per day. CZCR is 100% within each selected 30 min window. It is not 100% over the full evening session each window was drawn from. The full-session CZCR, computed over the complete logged recording for that day, is 98.02% (9 violations, 6 May), 97.22% (18 violations, 7 May) and 91.50% (29 violations, 8 May), all violations being mild upper-band excursions above 26.0 °C rather than overcooling. The selected windows are therefore a best-case slice of each session, not representative of the full recording, and should be read as a feasibility demonstration rather than a validated steady-state result. With only $n=3$ trial-level observations and a non-random (post-hoc, highest-CZCR) selection rule, no confidence interval is reported for the 100% figure. The mean PMV is 0.043 ± 0.023 across the three trial means (0.039, 0.068, 0.022), each within the ISO 7730 [4] neutral band ($-0.5<\mathrm{PMV}<+0.5$). The PMV is computed with the pythermalcomfort library’s ISO 7730 implementation from the logged air temperature and relative humidity, with mean radiant temperature assumed equal to air temperature (no separate radiant sensor), relative air velocity fixed at 0.1 m/s, metabolic rate fixed at 1.0 met (seated, quiet), and clothing insulation fixed at 0.5 clo (typical light indoor clothing). Air velocity, metabolic rate and clothing insulation are therefore nominal assumptions rather than measured quantities. The 30 min energy delta averages 0.156 kWh across the three trials, with a wide spread (0.0024–0.2522 kWh) dominated by the low-power state of the third trial, in which the controller held the actuator at its low state for the entire 30 min window. The trial windows are not pre-marked in the source log. They were identified by a sliding 30 min search across the three cleanest days (6–8 May 2026), scored by CZCR and then by the lowest temperature range (0.31–0.58 °C). The logger’s sampling interval on 8 May was 60 s versus 20 s on the two previous days, which is why trial 3 contains fewer rows in Table 9.

Table 9. Three repeated 30 min hybrid trials on the benchtop hardware-in-the-loop testbed, selecting within the three cleanest trial days (6–8 May 2026) the 30 min window with the highest CZCR and, on ties, the narrowest temperature range per day. The 30 min energy delta is the change in cumulative energy over the trial window.

Trial	Date/Time (MYT)	n Rows	T Range (°C)	CZCR (%)	PMV (Mean ± SD)	$\Delta \mathit{E}$ (kWh)
1	2026-05-06 21:27–21:57	85	25.56–25.95	100.00	0.039 ± 0.049	0.2137
2	2026-05-07 21:15–21:45	88	25.61–25.92	100.00	0.068 ± 0.043	0.2522
3	2026-05-08 21:38–22:08	30	25.16–25.74	100.00	0.022 ± 0.087	0.0024
Aggregate ($n=3$)	n/a	203	25.16–25.95	100.00	0.043 ± 0.023	0.156

Table 9. Three repeated 30 min hybrid trials on the benchtop hardware-in-the-loop testbed, selecting within the three cleanest trial days (6–8 May 2026) the 30 min window with the highest CZCR and, on ties, the narrowest temperature range per day. The 30 min energy delta is the change in cumulative energy over the trial window.

Trial	Date/Time (MYT)	n Rows	T Range (°C)	CZCR (%)	PMV (Mean ± SD)	$\Delta \mathit{E}$ (kWh)
1	2026-05-06 21:27–21:57	85	25.56–25.95	100.00	0.039 ± 0.049	0.2137
2	2026-05-07 21:15–21:45	88	25.61–25.92	100.00	0.068 ± 0.043	0.2522
3	2026-05-08 21:38–22:08	30	25.16–25.74	100.00	0.022 ± 0.087	0.0024
Aggregate ($n=3$)	n/a	203	25.16–25.95	100.00	0.043 ± 0.023	0.156

Figure 5 plots the on-disk indoor/outdoor telemetry from the 6 May 2026 trial, illustrating the tight indoor-temperature band maintained by the hybrid controller despite outdoor thermal cycling. Figure 6 plots telemetry from an earlier commissioning run on 30 April 2026, in which an initial thermal transient (indoor temperature at 30.1 °C) is brought back within the comfort band by the hybrid controller within approximately 20 min.

Figure 5. Indoor versus outdoor temperature telemetry from the physical testbed during the 6 May 2026 steady-state hybrid control trial. The indoor temperature is held within a narrow band despite outdoor oscillation.

Figure 6. Indoor versus outdoor temperature telemetry from the 30 April 2026 commissioning run, showing the hybrid controller’s response to an initial thermal transient. The indoor temperature is brought back within the comfort band within approximately 20 min.

Table 9 reports windows selected post hoc. Table 10 reports a separate, later set of three trials that answer the predefined-protocol concern directly. On 5 June 2026, the controller’s own operating log brackets three trials with explicit SYSTEM events: “Raspberry Pi AI 30 min Test Started” and, for two of the three, “AI Test Ended. Total kWh: <value>”. That is, the trial boundary and nominal duration were declared by the test protocol itself at the time, not selected afterward by searching the log for the best-looking window. (Trial A is bounded by the next test’s start rather than its own logged end, as no distinct shutdown message precedes it.) As an internal consistency check, the energy delta computed independently from the per-step log for Trials B and C matches the logged Total kWh figure exactly (0.2286 and 0.2625 kWh respectively), confirming these are genuine system-declared trials rather than a reconstruction. Across the three prospective trials, CZCR is 91.21%, 92.59% and 86.42% (aggregate 90.12%, 25 violations in 253 rows), materially lower than the 100.00% of the post-hoc-selected May windows and consistent with the full-session CZCR reported above. Manual HMI interactions (light, window and lock toggles) occurred during these trials as part of normal demo use of the testbed and were not suppressed, which is itself closer to an uncontrolled-disturbance setting than the quieter May sessions, though not a systematically varied disturbance protocol. These results should be read together with Table 9, not in its place: the two together bound the controller’s benchtop performance between a best-case (100.00%) and a more representative, prospectively declared case (86–93%).

Table 10. Three further hybrid trials from 5 June 2026, bracketed by the testbed’s own SYSTEM-level “30 min Test” start/end events rather than selected post hoc. Trial A’s window ends at the next test’s start (no distinct end-of-test message was logged for it). The $\Delta E$ values for Trials B and C match the logged Total kWh shutdown message exactly.

Trial	Date/Time (MYT)	n Rows	T Range (°C)	CZCR (%)	PMV (Mean ± SD)	$\Delta \mathit{E}$ (kWh)
A	2026-06-05 22:04–22:36	91	24.00–26.84	91.21	0.020 ± 0.110	0.2741
B	2026-06-05 22:36–23:06	81	24.00–26.02	92.59	0.033 ± 0.117	0.2286
C	2026-06-05 23:08–23:38	81	24.00–26.25	86.42	0.043 ± 0.120	0.2625
Aggregate ($n=3$)	n/a	253	24.00–26.84	90.12	0.032 ± 0.115	0.2551

Table 10. Three further hybrid trials from 5 June 2026, bracketed by the testbed’s own SYSTEM-level “30 min Test” start/end events rather than selected post hoc. Trial A’s window ends at the next test’s start (no distinct end-of-test message was logged for it). The $\Delta E$ values for Trials B and C match the logged Total kWh shutdown message exactly.

Trial	Date/Time (MYT)	n Rows	T Range (°C)	CZCR (%)	PMV (Mean ± SD)	$\Delta \mathit{E}$ (kWh)
A	2026-06-05 22:04–22:36	91	24.00–26.84	91.21	0.020 ± 0.110	0.2741
B	2026-06-05 22:36–23:06	81	24.00–26.02	92.59	0.033 ± 0.117	0.2286
C	2026-06-05 23:08–23:38	81	24.00–26.25	86.42	0.043 ± 0.120	0.2625
Aggregate ($n=3$)	n/a	253	24.00–26.84	90.12	0.032 ± 0.115	0.2551

The outage-recovery record and the edge-stack benchmark are summarised in Table 11. The system reconnected after a 12 min 42 s router outage (2026-05-02 22:44:16 to 22:56:58) and the indoor temperature moved by $+0.06$ °C across the outage (25.75 °C to 25.81 °C), well within the comfort band. This is a favourable but bounded result: the enclosure’s thermal time constant is long relative to the outage duration, so the drift is small by construction. The operative finding is the autonomous reconnection, not the drift magnitude.

Table 11. Edge-stack benchmark on Raspberry Pi 4 and ESP32 (rev 1.4, 4 GB). All latency values are measured from the on-disk telemetry log. The bill of materials (BOM) covers the bare controller board only. The Peltier and PTC heat-pump load and the AC unit are excluded.

Metric	Value
Controller board	Raspberry Pi 4 Model B (rev 1.4, 4 GB, BCM2711)
Sensor node	ESP32-WROOM-32 + SHT30 (I2C)
MQTT broker	Mosquitto 2.0 over TLS 1.2
Model size (recurrent PPO checkpoint)	7.29 MB (ppo_lstm.zip), 2.4 MB exported weights
Control-action cadence (measured)	median 33 s between logged actions
Inference latency (mean ± SD)	112.85 ± 19.1 ms ($n=1392$)
Inference latency (median)	113.2 ms
Inference latency (p95/p99)	116.3 ms/120.4 ms
Inference latency (range)	7.3–414.3 ms
HMI override latency (tablet to Pi)	16.3 ± 7.5 ms ($n=15$)
CPU/RAM utilisation, packet loss	not instrumented (Section 5.3)
MQTT reconnection after 12 min outage	12 min 42 s, T drift $=+0.06$ °C
BOM (controller board only)	<US$75

Table 11. Edge-stack benchmark on Raspberry Pi 4 and ESP32 (rev 1.4, 4 GB). All latency values are measured from the on-disk telemetry log. The bill of materials (BOM) covers the bare controller board only. The Peltier and PTC heat-pump load and the AC unit are excluded.

Metric	Value
Controller board	Raspberry Pi 4 Model B (rev 1.4, 4 GB, BCM2711)
Sensor node	ESP32-WROOM-32 + SHT30 (I2C)
MQTT broker	Mosquitto 2.0 over TLS 1.2
Model size (recurrent PPO checkpoint)	7.29 MB (ppo_lstm.zip), 2.4 MB exported weights
Control-action cadence (measured)	median 33 s between logged actions
Inference latency (mean ± SD)	112.85 ± 19.1 ms ($n=1392$)
Inference latency (median)	113.2 ms
Inference latency (p95/p99)	116.3 ms/120.4 ms
Inference latency (range)	7.3–414.3 ms
HMI override latency (tablet to Pi)	16.3 ± 7.5 ms ($n=15$)
CPU/RAM utilisation, packet loss	not instrumented (Section 5.3)
MQTT reconnection after 12 min outage	12 min 42 s, T drift $=+0.06$ °C
BOM (controller board only)	<US$75

5. Discussion

The results in Section 4 support three engineering claims and one design lesson.

The first engineering claim is that a Simplex-architecture hybrid recurrent-RL controller can run at the edge with inference latency far below the control cadence while a lightweight rule bounds when the controller’s command is overridden: every logged inference completes in a mean of 112.85 ms against a control-action cadence of roughly 30 s and a 15 min simulation step. The Simplex supervisor is the load-bearing component: without the lower trigger (the archived upper-bound-only configuration, Section 4.5), overcool excursions go unchecked. With the two-sided predicate, 0 violations are recorded on New York and Kuala Lumpur, with the residual 173 on Tokyo (Table 4). The shield is a two-line predicate (Equation (4)) that triggers an override within one control step of the threshold being crossed (Section 3.3). It is not a proof of closed-loop safety, and Section 4.4’s mechanistic account shows the zone can still drift for up to 11.75 h after triggering because of thermal lag. The role of the AI is to be approximately optimal within the envelope the shield constrains, not to be formally certified safe.

The second engineering claim is that a recurrent policy trained on a single climate transfers zero-shot to five further climates at 93.5–100% CZCR. The transfer is not free: the worst target climate (Egypt, 93.50%) sits 6.5 percentage points below the training climate, and the energy premium over a zero-shot PPO baseline is climate-dependent (1.7–6.8%). On Kuala Lumpur, where a matched-climate PPO baseline is now available (Section 4.3), the premium against that fairer baseline is $+15.2$%, not $+6.8$%: a PPO policy trained directly on the deployment climate matches the hybrid’s comfort outcome at materially lower energy. The controller’s value proposition is therefore not lowest-energy comfort on any single climate it has been fairly compared on. It is that one trained policy reaches comparable comfort across six climates without per-climate retraining, a property the matched-climate baselines, each trained once per climate, do not have by construction.

The third engineering claim is that the MQTT–TLS edge stack is reliable enough for benchtop deployment: the 12 min combined outage test (Table 11) shows that the system self-heals without a human in the loop, with a $+0.06$ °C thermal drift across the outage. The drift magnitude is a property of the enclosure’s thermal time constant and should not be generalised to a full-size building.

5.1. Comparison with Prior Literature

The closest single-paper analogue, Bai and Tan [18], combines a decoupled adversarial learning architecture with PPO. The present paper shares that algorithmic lineage but contributes the integration: the Sim-to-Real pipeline, the MQTT–TLS edge stack, the Unity digital twin, the two-sided safety shield with a bound-comparison result, and the physical benchtop validation. The paper is positioned as an empirical engineering case study, not as a new algorithm. The simulation numbers are directly comparable to the Sinergym benchmark corpus [19], on which the same recurrent-PPO implementation is evaluated. The broader positioning is against the simulation–deployment gap: Vázquez-Canteli and Nagy [32] and Mason and Grijalva [33] both note in their reviews that the RL-for-buildings literature is predominantly simulation-based, and the 23% real-building-deployment figure of Al Sayed et al. [11] quantifies the same gap. The benchtop hardware validation strand of this paper is a step across it, at benchtop scale (Section 5.3).

5.2. Engineering Implications

A bill of materials under US$75 for the bare controller board (Table 11) places the system in the same cost bracket as a domestic smart thermostat, with the added benefit of a recurrent policy that can be retrained per site. The measured 112.85 ms inference latency sits within the envelope that Chen and Ran [42] identify as the deep-learning-at-the-edge regime, in which inference is cheap relative to the control cadence but training remains off-device. Here, training is offline in Sinergym, and only inference ships to the Pi 4. The implied deployment model has four steps.

• Ship the controller with a default policy trained on a representative climate file.
• On first boot, run in RBC mode to collect site telemetry.
• Fine-tune the recurrent policy on the site telemetry.
• Lock the policy and run in inference mode indefinitely.

This is a benchtop feasibility pathway, not a demonstrated deployment model. Only the first step and the inference loop are validated in this manuscript.

5.3. Limitations

The manuscript has eleven substantive limitations.

• Building archetype and deployment scale. The simulated building is the five-zone commercial office archetype distributed with Sinergym, not a residential model. The physical testbed is a 0.5 m³ benchtop enclosure, not a habitable room. The title and claims are therefore scoped to building HVAC at benchtop scale, and no residential deployment is claimed.
• Single-run training (partially addressed). The originally reported training run used no explicit seed, and an archived multi-seed CSV purporting to cover this has ten identical rows, traced to a defect in the evaluation script that produced it rather than in training itself (Section 3.2). A corrected ten-seed re-run on the training climate (Section 4.2) shows CZCR and violations are seed-invariant (100.00%, 0, across all ten seeds) and energy varies by a 2.3% coefficient of variation (mean 29,332.68 ± 673.30 kWh), with the originally reported single-run energy figure (32,956.07 kWh) sitting above this ten-seed range. This resolves the seed-sensitivity question for the hybrid policy on its training climate. It does not resolve it for the zero-shot transfer climates (Section 4.4) or for the PPO/DQN baselines (Table 4), which remain single-run and for which multi-seed coverage is identified as future work. The sensitivity of reported DRL performance to random-seed variation is a well-documented reproducibility concern [55].
• Mixed training climates in cross-controller comparison (partially addressed). In Table 4, the original PPO and DQN baselines were trained on New York while the hybrid was trained on Kuala Lumpur. A matched-climate PPO and DQN baseline, each retrained directly on Kuala Lumpur (Section 4.3), has now been added to Table 4 for this climate. The matched-climate PPO baseline matches the hybrid’s comfort outcome (100.00% CZCR, 0 violations) at 13.2% less energy, and the matched-climate DQN baseline reaches 99.32% CZCR at 15.0% less energy than the hybrid. This resolves the training climate confound for Kuala Lumpur and shows the hybrid’s energy premium there is larger, not smaller, once the confound is removed. It does not resolve the confound for the New York and Tokyo rows, which remain zero-shot comparisons and for which matched-climate retraining is identified as further future work.
• Sim-to-Real gap. The simulation CZCR is 93.50–100.00% depending on climate and the physical CZCR is 100.00% within each of the three selected 30 min trials, but the physical validation covers 90 min of benchtop operation against an annual simulated horizon, and the 30 min windows were selected post hoc as the cleanest per day rather than sampled prospectively. The full evening sessions those windows were drawn from have a lower CZCR (98.02%, 97.22% and 91.50% on 6, 7 and 8 May 2026, respectively), confirming that the selected-window figure is a best case rather than a steady-state estimate. A separate set of three prospective trials from 5 June 2026 (Table 10), bracketed by the testbed’s own protocol-level start/end log events rather than selected post hoc, records 86.42–92.59% CZCR (aggregate 90.12%), consistent with the full-session figures above and a more representative estimate of benchtop performance than the post-hoc-selected 100.00% windows. This partially addresses, but does not fully resolve, the predefined-protocol request: a single evening’s three trials is still a small and short sample, and no systematically varied disturbance protocol was applied. The benchtop enclosure is also a materially different thermal system from the simulated office archetype. Narrowing this reality gap [23] would require a full-scale deployment.
• EnergyPlus version split between the original and revision results. The originally reported single-run results (Table 4’s zero-shot rows, Table 5, Table 7 and Table 8) were produced under EnergyPlus 23.1.0. The ten-seed re-run (Section 4.2) and the matched-climate PPO/DQN retraining (Section 4.3), both added for this revision, were instead run under EnergyPlus 24.1.0, the version the vendored Sinergym 3.10.6 copy actually requires. The two EnergyPlus releases share the same five-zone building archetype and weather-file handling, so the comfort-band and energy-accounting definitions are unchanged, but the two result sets were not produced on an identical simulation-engine build. This is a residual caveat on the direct numerical comparability of the new multi-seed and matched-climate figures against the original single-run figures they are compared with in Table 4, beyond the training climate and seed effects those sections already isolate. Re-running the original single-run baselines under EnergyPlus 24.1.0 for full consistency is identified as future work.
• No formal safety proof. The shield predicate (Equation (4), Section 3.3) guarantees only that an override is issued within one control step of a threshold crossing, conditioned on unbiased sensing and timely actuation. It does not bound the resulting excursion depth or duration, which the cross-climate thermal-lag analysis (Section 4.4) shows can reach 11.75 h. No formal verification of the closed-loop (shield + plant) system against a timed safety specification was performed.
• Edge-stack instrumentation gap. Raspberry Pi CPU/RAM utilisation, MQTT packet loss counters, and missed control commands were not logged during the May 2026 trials. Only inference latency and application-level events are on disk. Table 11 now reports p95/p99 inference latency (re-derived from the existing log) but these resource- and network-level metrics require re-instrumentation and a repeat trial.
• Single building and single actuator path. The policy is not evaluated on other building geometries or HVAC plants. The physical actuation path is a Peltier/PTC benchtop load, not a real air-handling unit.
• Sensor-fault test is simulation-only. The $+4.0$ °C sensor-bias injection (Section 4.6) runs in the simulation loop. No physical sensor-fault injection was performed, and the PID supervisor in that test is deliberately mis-tuned to exhibit windup, so it bounds the comparison rather than representing a well-tuned PID.
• No statistical hypothesis testing. All reported differences are descriptive single-run or single-trial observations. No formal hypothesis testing, confidence intervals, or multiple-comparisons correction are applied across the family of five controllers, three trials and several metrics.
• LSTM-attribution and auxiliary claims not isolated. The physical trials do not include a PPO-only ablation, so the marginal contribution of the LSTM recurrence on hardware is not isolated. Auxiliary hardware figures (per-controller energy under the 30 min protocol) beyond the three reported trials are not re-derivable from the on-disk log.

6. Conclusions

This paper presented an integrated Sim-to-Real LSTM–PPO–RBC controller for building HVAC, deployed on a Raspberry Pi 4 and ESP32 edge stack and evaluated in Sinergym/EnergyPlus on six climate profiles, with benchtop HIL validation. The contribution is integration: the Sim-to-Real pipeline (Sinergym/EnergyPlus → MQTT–TLS → Unity 6 digital twin), the two-sided rule-based safety-override shield, and the physical HIL validation together contribute toward practical DRL-HVAC, building on the algorithmic lineage of Bai and Tan [18] and the Sinergym benchmark corpus [19].

Five headline findings emerge. First, the Kuala Lumpur-trained recurrent policy transfers zero-shot to five further climates at 93.50–100.00% CZCR, with 0 violations on New York, the best-case transfer, and the worst transfer in the hot desert climate (Egypt, 93.50%). Second, the safety shield’s lower trigger is load-bearing: an archived upper-bound-only configuration records 4124 overcool violations on the Kuala Lumpur trace, whereas the deployed two-sided predicate records 0 on the same climate at essentially the same energy. Third, a ten-seed re-run on the training climate (Section 4.2) shows CZCR and violations are seed-invariant (100.00%, 0, in every seed) while energy varies by a 2.3% coefficient of variation (mean 29,332.68 ± 673.30 kWh). This is the first quantified seed-sensitivity estimate for this controller, covering the training climate only. Fourth, a matched-climate PPO baseline retrained directly on Kuala Lumpur (Section 4.3) matches the hybrid’s 100.00% CZCR, 0-violation outcome at 13.2% less energy, revising the hybrid’s reported energy premium on this climate from $+6.8$% (against a zero-shot baseline) to $+15.2$% (against the matched-climate one). The hybrid’s value on Kuala Lumpur is therefore not lowest-energy comfort. It is that the same policy, without retraining, also transfers to five further climates, which the matched-climate baseline does not attempt. Fifth, the benchtop HIL validation records a 100.00% CZCR within each of three selected 30 min windows (98.02–91.50% over the corresponding full evening sessions, Section 4.7, with a separate set of three prospectively protocol-declared trials recording 86.42–92.59%, aggregate 90.12%) with a PMV of 0.043 ± 0.023, a 112.85 ms mean on-device inference latency ($n=1392$, p95/p99 116.3/120.4 ms), and autonomous recovery from a 12 min 42 s router outage with a $+0.06$ °C thermal drift.

The future-work agenda is structured around the limitations in Section 5.3. It comprises the following items.

• Extending the ten-seed re-run (Section 4.2) to the zero-shot transfer climates and to the PPO/DQN baselines, which remain single-run.
• A physical PPO-only ablation on the same 30 min HIL protocol, to isolate the LSTM-attribution component.
• Matched-climate retraining of the PPO and DQN baselines on New York and Tokyo, extending the Kuala Lumpur matched-climate baselines already added in this revision (Section 4.3) to the remaining two climates in Table 4.
• Further prospective trials, beyond the three reported in Table 10, under systematically varied rather than incidental disturbances.
• An in-situ sensor-fault test on the physical testbed.
• A multi-building evaluation across alternative building archetypes.
• Formal verification of the shield-plant closed loop against a timed safety specification.
• CPU/RAM and network-level instrumentation (packet loss, missed commands) on the edge stack.
• A fairly tuned PID baseline for the sensor-fault comparison.
• A scaled-up physical enclosure closer to a habitable room volume.