Adaptive Lighting and Thermal Comfort Control Strategies in Digital Twin Classroom via Deep Reinforcement Learning

Abstract

With the advancement of smart education and carbon neutrality goals, optimizing Indoor Environmental Quality (IEQ) while minimizing energy consumption is critical. Traditional PID or rule-based strategies struggle with the strong non-linearity and time delays of photothermal coupling in high-density classrooms. This paper proposes an adaptive closed-loop control framework fusing Digital Twin (DT) and Deep Reinforcement Learning (DRL). A high-fidelity multi-physics model is constructed as a virtual testbed, utilizing the Proximal Policy Optimization (PPO) algorithm to learn multi-objective strategies. The trained agent is deployed to an edge gateway for real-time inference. Experimental results from a field study distinguish this work from pure simulations. Results demonstrate that compared to PID baselines, the proposed strategy reduces energy consumption by 28.4% while maintaining thermal comfort (PMV) and visual comfort compliance. Furthermore, the variance of PMV is reduced by 66.7%, and system recovery time under stochastic disturbances is shortened by 31.4%.

1. Introduction

Driven by the global digital transformation, smart building technology is swiftly advancing from mere automation to true intelligence [1]. Indoor environmental quality (IEQ) has emerged as a vital metric for balancing energy efficiency with occupant health [2]. This is especially important in educational buildings, where classroom thermal and humidity conditions, along with lighting, not only ensure physical comfort but also significantly impact students’ cognitive performance, attention, and long-term mental and physical health. In recent years, the widespread adoption of internet of things (IoT) heterogeneous sensing devices and edge computing technology [3] has made managing smart campus environments based on extensive real-time data a central focus of research in both academia and industry.

Existing environmental control methods face considerable limitations. Traditional approaches, such as PID control [4], depend heavily on predetermined fixed parameters or simple logic rules. This reactive mechanism struggles to adapt accurately to the complex and dynamic environments of classrooms [5], which are characterized by high randomness in occupant movement, significant thermal inertia, and pronounced nonlinear features. These inadequacies often result in system overshooting or oscillation. More critically, traditional methods typically prioritize comfort as the sole control objective, overlooking the optimization of energy efficiency in multi-objective scenarios [6,7]. Consequently, a pressing challenge in intelligent building control is to maintain high standards of environmental comfort while maximizing system energy efficiency [8,9], even in the face of sudden external weather changes and internal load disturbances.

Digital Twin (DT) technology acts as a crucial conduit for two-way data exchange between physical entities and virtual environments [10]. With its high-fidelity real-time mapping and comprehensive digitalization capabilities, DT shows significant potential in managing the entire lifecycle of buildings. While DT technology has matured in areas like building operation monitoring, facility maintenance, and fault diagnosis, its applications in real-time environmental closed-loop control and multi-objective collaborative optimization remain exploratory. As a result, its predictive and simulation capabilities have yet to be fully realized. In parallel, Deep Reinforcement Learning (DRL) [11], an advanced model-free adaptive decision-making paradigm [12], excels in optimizing strategies through trial-and-error interactions with the environment. It demonstrates exceptional performance and generalization ability in addressing complex control problems within high-dimensional continuous action spaces [13].

Recent studies show that combining the high-precision physical simulation capabilities of digital twins with the autonomous decision-making abilities of deep reinforcement learning effectively overcomes traditional control limitations and addresses the challenges of virtual-to-real transfer [14]. This integration is accomplished through low-cost strategy training and validation in a virtual environment. Building on this foundation, the present study aims to fill a research gap in photothermal coupling control [15] by proposing an adaptive classroom environment regulation framework that merges digital twins with reinforcement learning. This framework seeks to optimize both comfort and energy consumption [16].

Despite the extensive literature on intelligent control within the built environment, significant methodological gaps persist in tackling multi-objective collaborative optimization and dynamic adaptability in complex scenarios. To bridge these gaps, this study developed an adaptive regulation framework integrating digital twin technology with deep reinforcement learning algorithms. The primary contributions of this research are:

• Development of a Closed-Loop Digital Twin Framework: We construct a dynamic bidirectional feedback mechanism that utilizes Digital Twin as a safe training ground for RL agents, bridging the simulation-reality gap.
• Multi-Objective PPO Control Strategy: We propose a reward function that simultaneously optimizes conflicting goals: thermal comfort, visual comfort, and energy efficiency.
• Edge-Cloud Collaborative Deployment: We validate the system’s real-time feasibility, demonstrating superior robustness (31.4% faster recovery time) against stochastic disturbances compared to PID baselines.

2. Related Work and Literature Review

In the field of intelligent building and environmental engineering, the evolution of environmental control systems remains a central topic of academic interest. Traditional control strategies, such as PID control, fuzzy logic control, and model predictive control (MPC), maintain basic comfort levels in steady-state environments. However, these methods lack the real-time responsiveness required to manage the time-varying characteristics of dynamic environments and the challenges posed by nonlinear disturbances. With the advent of Industry 4.0, the exponential growth of IoT sensor networks and big data has driven a paradigm shift in building control from singular mechanistic approaches to proactive regulation based on intelligent sensing and data-driven methods. Current research increasingly focuses on leveraging multi-source heterogeneous data to achieve a precise understanding and adaptive decision-making in complex building environments.

2.1. Traditional Control Methods and Their Limitations

PID control and its variants have long been the standard in building automation. However, they frequently fall short in robustness when faced with random disturbances like fluctuating occupancy and sudden weather changes. Although Model Predictive Control (MPC) is excellent for optimizing energy efficiency [17], it depends heavily on accurate physical models and requires substantial online computational resources. This dependency restricts its use on low-cost edge devices. Furthermore, Rule-Based Control (RBC) [18], while simple, lacks adaptive learning capabilities making it difficult to optimally balance comfort with energy consumption [19].

2.2. Applications of Digital Twin Technology

Digital twin technology offers a dynamic virtual representation of building systems through real-time data interaction [20]. Almadhor [21] et al. developed a digital twin-based framework for smart building operation and maintenance, significantly improving facility management efficiency. In energy management [22], digital twins have been used to create highly accurate energy consumption prediction models, supporting demand-side response [23]. Recent studies have integrated digital twins with Building Information Modeling (BIM) and Internet of Things (IoT) data to enable real-time visualization and monitoring of thermal comfort in indoor environments. However, most current digital twin applications remain focused on [24] monitoring and diagnosis, lacking the transition to a closed-loop system for autonomous control.

2.3. Application of Reinforcement Learning in Intelligent Environmental Control

Reinforcement learning (RL) has become a vital tool in intelligent environmental control, enabling systems to learn optimal strategies through environmental interaction. This adaptability allows systems to adjust to changes and enhance performance over time. Recent advancements in RL have significantly broadened its use in managing complex environmental systems, such as smart grids and automated climate control in buildings. Previous studies employed RL algorithms to optimize energy consumption in smart buildings, achieving notable reductions in energy use while maintaining occupant comfort [25]. These algorithms utilized historical data and real-time feedback to dynamically adjust control strategies, accommodating variations in occupancy and weather conditions. Additionally, RL has been effectively integrated into smart grid management [26], efficiently balancing energy supply and demand. By predicting energy usage patterns and adjusting resource allocation, RL systems have contributed to more sustainable and cost-effective energy management solutions. The application of RL in these areas highlights its potential to transform environmental control systems. As RL techniques continue to evolve, they promise to deliver even more sophisticated and adaptive solutions, paving the way for enhanced sustainability and efficiency across various sectors. Reinforcement Learning (RL) learns optimal strategies through trial and error by interacting with the environment. Recently, it has achieved groundbreaking progress in HVAC (Heating, Ventilation, and Air Conditioning) control. Zhang et al. employed the DQN algorithm to optimize temperature control strategies in office buildings, which resulted in significant energy savings. For continuous control challenges, the PPO and SAC [27] algorithms have shown greater stability in managing complex building thermal dynamics. In lighting control, RL has been used to dynamically adjust artificial lighting based on natural light variations [28], thereby enhancing visual comfort. Despite these advancements, most RL studies have been conducted in static simulation environments, creating a significant sim-to-real gap [29]. This gap often reduces the effectiveness of strategies when applied in real-world settings. However, a critical limitation in these studies is the decoupling of visual and thermal systems [25,26,27]. Most existing approaches optimize these subsystems independently, ignoring the significant heat gain from lighting or the impact of solar radiation on both illumination and temperature. Furthermore, the lack of hardware-in-the-loop validation often leads to performance degradation when theoretical models are applied to real-world hardware. The comparison of different environmental control strategies is shown in Table 1.

Table 1. Comparison of Environmental Control Strategies.

Strategy	Model Dependency	Adaptability	Multi-Objective	Computation Cost (Inference)	Limitations
PID	Low (Model-free)	Poor (Fixed gains)	No (SISO)	Very Low	Struggles with lag & coupling
Rule-based	Low	Low (Static rules)	Limited	Very Low	Rigid, cannot handle dynamics
MPC	High (Physics model)	High	Yes	High	Sensitive to model mismatch
Standard RL	Low (Black box)	High	Yes	Low	Unsafe training on real hardware
Ours (DT-RL)	Hybrid (DT training)	Very High	Yes	Low	Safe Sim-to-Real transfer

2.4. Recent Advances in the Synergy of Digital Twins and Reinforcement Learning

In recent years, the integration of digital twins with reinforcement learning has attracted considerable attention in both academia and industry. Digital twins, which are virtual replicas of physical systems, offer a dynamic environment for testing and optimizing strategies without the risks of real-world experimentation. Reinforcement learning, a branch of machine learning, excels in decision-making tasks by learning optimal policies through trial and error. The collaboration between these technologies has led to significant advancements. Researchers have successfully used digital twins to simulate complex environments, enabling more efficient training of reinforcement learning algorithms. This synergy has been particularly beneficial in manufacturing, where digital twins model intricate processes, allowing reinforcement learning to optimize production lines and reduce operational costs. Furthermore, digital twins have enhanced the adaptability and robustness of reinforcement learning models. By providing a controlled and replicable environment, digital twins facilitate the continuous refinement of learning algorithms, resulting in improved performance and reliability in real-world applications. Recent studies have shown that this collaborative approach not only accelerates the development of reinforcement learning models but also broadens their applicability across various domains. Consequently, the integration of digital twins and reinforcement learning is poised to drive further innovations, offering new opportunities for optimizing complex systems and processes. To enhance training efficiency and safety, the use of Digital Twins (DTs) as environments for Reinforcement Learning (RL) has emerged as a leading research direction. Chen [30] and colleagues developed a data-driven DT model that significantly shortened the training cycle for RL agents. In multi-objective optimization, Li’s team [31] introduced a DT-RL framework to optimize thermal comfort and air quality in a coordinated manner. However, existing research has not yet achieved multi-objective adaptive control in light-thermal coupled environments for classrooms with dynamic occupancy and complex demands [32].

2.5. Research Innovation

Unlike conventional studies that rely solely on simulation or static physical models, this research proposes a closed-loop Digital Twin-Deep Reinforcement Learning (DT-DRL) framework. By integrating high-fidelity multiphysics modeling with real-time sensor data, the system creates a virtual training environment that mirrors the stochastic nature of the physical world. The study innovatively addresses the physical coupling between visual and thermal comfort, which is often treated independently in the existing literature. Recognizing the limitations of value-based DRL algorithms (like DQN or DDPG) in sensitive building control tasks, this study applies the Proximal Policy Optimization (PPO) algorithm. Scientific originality is further established through rigorous experimental validation in a real-world classroom environment rather than purely theoretical simulation.

3. Methods

3.1. Research Framework Overview

This study presents an adaptive closed-loop control framework that combines Digital Twin (DT) technology with Deep Reinforcement Learning (DRL) to overcome the limitations of traditional control strategies in handling stochastic disturbances and multi-objective conflicts in classroom settings. This framework advances beyond traditional single-objective optimization by enhancing global system efficiency while considering individual variations in thermal, humidity, and visual comfort. This research focuses on three main components: constructing a high-fidelity digital twin, designing DRL-based control strategies, and integrating and experimentally validating the cyber-physical system. The subsequent sections will elaborate on the methodological principles and implementation steps for each component.

To visualize the complex interaction between the physical classroom and the virtual training environment, the complete workflow of the proposed DT-RL framework is illustrated in Figure 1. As shown in the diagram, the system operates in three hierarchical phases: physical sensing, offline policy optimization in the cloud, and real-time inference on the edge gateway.

Figure 1. The overall workflow of the proposed DT-RL framework.

3.2. Digital Twin Modeling

Digital twin technology underpins precise environmental control by creating a virtual mirror that accurately maps and tracks the real-time state of physical classrooms. This model functions as a digital replica of the physical world, offering a cost-effective, risk-free environment for dynamic interaction and the training of reinforcement learning agents.

3.2.1. Model Construction and Simulation

Creating a digital twin classroom model entails developing a highly accurate dynamic map of various physical quantities within a physical space. This model views the classroom as a complex system that integrates thermodynamics, optics, and fluid dynamics. Key state variables include illuminance distribution, three-dimensional indoor temperature-humidity fields, airflow velocity vectors, the spatiotemporal distribution of occupants, and metabolic rates. Precisely defining these variables ensures the model thoroughly addresses the building energy consumption.

The model construction process integrates physical mechanisms with data-driven methods. At the physical level, a multiphysics coupling model was developed using thermodynamic heat transfer equations and the Radiance lighting mode [33]. The thermal environment module, based on the law of energy conservation, employed the EnergyPlus engine to calculate wall heat transfer, convective heat exchange, and radiative heat gain. Concurrently, the lighting environment module utilized the Radiance engine to simulate interactions between natural and artificial lighting and their spatiotemporal dynamics. To improve real-time performance, surrogate modeling techniques using Gaussian Process Regression (GPR) [34] or neural networks [35] were introduced to accelerate computations. This hybrid modeling strategy maintained the interpretability of mechanistic models while harnessing the efficient inference capabilities of data-driven models, thereby significantly enhancing simulation realism in complex dynamic scenarios.

To calibrate model parameters, this study used historical data from high-resolution sensors for reverse correction. Specifically, spectral power distribution data from light sensors calibrated the photometric parameters of lighting fixtures. High-frequency sampling sequences from temperature and humidity sensors adjusted thermal capacity and thermal resistance coefficients. Moreover, occupancy density heat maps, derived from computer vision algorithms, dynamically updated the internal load model. By employing fine mesh generation and a dynamic refresh mechanism, the model established a high-frequency real-time feedback loop, ensuring the state evolution of the digital twin remained highly consistent with the physical entity.

Simulation verification occurred in a standard classroom environment under diverse boundary conditions, such as abrupt changes in natural daylight, random occupant movements, and shifts in HVAC system operations. By numerically solving partial differential equations and performing time-series comparative analysis with multi-point sensor data from field deployments, the results demonstrated that the model accurately replicated the spatiotemporal distribution characteristics of key indicators like PMV (Predicted Mean Vote), PPD (Predicted Percentage of Dissatisfied), and illuminance uniformity. This high-fidelity simulation environment functioned as a digital testbed for subsequent reinforcement learning algorithm training.

3.2.2. Dynamic Data Acquisition and Synchronization

Developing an efficient and accurate dynamic data acquisition system is crucial for constructing a high-fidelity digital twin. This system continuously monitors and analyzes multidimensional environmental parameters, ensuring the virtual model can track real-time changes in the physical environment. The data acquisition layer uses a heterogeneous IoT sensor network and employs the MQTT protocol [36] to facilitate low-latency data transmission. This method strictly maintains synchronization errors between the virtual model and the physical entity within a defined threshold.

Specifically, data collection covers three dimensions:

(1)

Light Environment Dimension: High-sensitivity light sensors and spectrometers monitor and quantify the illuminance levels and color temperature characteristics of various classroom work surfaces in real-time. This data directly correlates with assessments of visual comfort.

(2)

Thermal Environment Dimension: Researchers constructed a three-dimensional distribution map of the indoor microclimate using a distributed array of thermocouples, humidity sensors, and hot-wire anemometers. This setup effectively captured local variations in temperature and humidity.

(3)

Air Quality Dimension: This study incorporated a non-dispersive infrared (NDIR) carbon dioxide sensor [37] to monitor air cleanliness in real-time. These measurements offered data support for regulating the fresh air system.

The spatiotemporal synchronization of multi-source heterogeneous data is essential for ensuring the reliability of digital twins. The system uses a unified Network Time Protocol (NTP) to assign microsecond-level timestamps to all nodes, effectively addressing clock drift in distributed sensors. To resolve data alignment challenges caused by inconsistent sampling frequencies among different sensors (e.g., temperature at 1 Hz, illuminance at 10 Hz) and communication delays, this study employed a high-order spline interpolation algorithm for temporal resampling. Additionally, Kalman Filter technology was used to eliminate measurement noise and outliers. The preprocessing module also incorporates a statistics-based anomaly detection algorithm, which intelligently identifies and corrects data loss due to equipment failures, thus providing robust millisecond-level perception inputs for upper-layer control algorithms.

3.2.3. Integration of Virtual and Real Environments

The seamless integration of virtual and physical environments serves as the neural hub of digital twin systems, enabling bidirectional closed-loop interactions between physical entities and digital models. Given the highly nonlinear and time-varying nature of classroom environments, it is essential to develop a real-time responsive virtual-physical mapping mechanism to achieve adaptive control.

The system interface design follows principles of modularity and standardization, employing industrial communication standards like RESTful API to seamlessly integrate software and hardware components. In the sensor uplink, preprocessed sensor data streams are mapped in real-time to the virtual model’s input boundary conditions. For the control downlink, optimized control commands—such as LED dimming ratios, air conditioning set temperatures, and fan speeds—are directly transmitted to physical actuators via a high-speed fieldbus. To ensure system security and stability, the interface layer includes a watchdog mechanism and heartbeat detection. Upon detecting communication interruptions or device malfunctions, the system automatically switches to a Fail-Safe Mode.

This research relies on high-quality sensor data, and the sensor drift or network latency can degrade control performance.

3.3. Design of Reinforcement Learning Controller

Leveraging the high-fidelity environmental interactions offered by the digital twin platform, this study developed an adaptive environmental control strategy utilizing deep reinforcement learning (DRL). As the decision-making core of the system, the reinforcement learning agent maps state perception to action execution through ongoing interaction with the virtual environment. It autonomously adjusts environmental parameters—such as temperature, humidity, and illumination—in response to real-time changes, including personnel movement and sudden weather shifts, with the goal of optimizing both comfort and energy consumption.

To effectively illustrate the closed-loop interaction mechanism between the agent and the digital twin environment, this study developed a system architecture, as shown in Figure 2. In this framework, the digital twin serves as a high-fidelity training platform, providing real-time state feedback and computing rewards to facilitate policy iteration within the Actor-Critic network.

Figure 2. The Adaptive Control Framework based on Digital Twin and PPO.

3.3.1. State-Space Design

The state space forms the perceptual foundation for reinforcement learning controllers, crucially transforming the multidimensional physical states of a digital twin classroom environment into mathematical vector representations for processing by an agent’s neural network. Designing an effective state space necessitates adherence to the principles of completeness and low redundancy. It must comprehensively reflect the environment’s dynamics while minimizing noise interference, precisely capturing the intrinsic causal relationships relevant to control objectives.

In this study, we selected state variables that strike a balance between clear physical significance and feasible data acquisition. This approach resulted in a composite state vector that integrates multidimensional features. Specifically, it includes:

(1)

Visual Environmental Characteristics: The real-time illuminance matrix, derived from a distributed light sensor array, directly reflects the level of visual comfort.

(2)

Thermal and Humidity Environmental Characteristics: Integrated high-precision temperature sensors and humidity-sensitive components measure key values, serving as core indicators for evaluating the thermal environment.

(3)

Characteristics of Air Quality: Carbon dioxide (CO₂) concentration data characterize indoor air quality and indirectly reflect ventilation conditions.

(4)

Load Disturbance Characteristics: The spatial and temporal variations in environmental thermal and moisture loads are indicated by the distribution of personnel density and activity intensity, as inferred from visual data or infrared sensors.

(5)

Environmental Context Features: By incorporating external meteorological parameters, including outdoor temperature, humidity, and solar radiation intensity, along with temporal variables such as specific times and dates, the agent gains the ability to anticipate future environmental conditions.

The state space design carefully accounts for the environment’s dynamic characteristics. It encompasses current temperature, humidity, and illumination, while also integrating historical state sequences to address the environment’s non-Markovian nature [38] (POMDP).

Formally, $\mathrm{t},$ the state vector, at time ${\mathrm{S}}_{\mathrm{t}}$ can be defined as

$${\mathrm{S}}_{\mathrm{t}}=[{\mathrm{T}}_{\mathrm{i}\mathrm{n}},{\mathrm{H}}_{\mathrm{i}\mathrm{n}},{\mathrm{L}}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}},{\mathrm{C}}_{\mathrm{c}\mathrm{o}2},{\mathrm{N}}_{\mathrm{o}\mathrm{c}\mathrm{c}},{\mathrm{T}}_{\mathrm{o}\mathrm{u}\mathrm{t}},{\mathrm{I}}_{\mathrm{s}\mathrm{o}\mathrm{l}\mathrm{a}\mathrm{r}},{\mathrm{\tau }}_{\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}}{]}^{\mathrm{T}}$$

The indoor temperature and humidity are denoted as ${\mathrm{T}}_{\mathrm{i}\mathrm{n}}$ and ${\mathrm{H}}_{\mathrm{i}\mathrm{n}}$, the illumination of the working surface as ${\mathrm{L}}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}}$, the carbon dioxide concentration as ${\mathrm{C}}_{\mathrm{c}\mathrm{o}2}$, the personnel density as ${\mathrm{N}}_{\mathrm{o}\mathrm{c}\mathrm{c}}$, and the external solar radiation intensity as ${\mathrm{I}}_{\mathrm{s}\mathrm{o}\mathrm{l}\mathrm{a}\mathrm{r}}$.

When constructing specific state vectors, balancing information richness with computational efficiency is crucial. The wide disparities in units and numerical ranges of various physical variables, such as CO₂ concentration in ppm and temperature in °C, can destabilize neural network gradient updates if input directly. To mitigate this, this study applied Z-Score normalization to preprocess all input data. Additionally, a Kalman filter algorithm was implemented to remove sensor noise and outliers, ensuring data quality and temporal consistency. To address the curse of dimensionality caused by strong coupling between variables and inconsistent dynamic response frequencies, Principal Component Analysis (PCA) was introduced to extract key features. A sliding time window technique was also employed to smooth high-frequency disturbances. This approach ensures that the state space, action space, and reward function operate in harmony, providing high signal-to-noise ratio environmental feedback for multi-objective optimization.

3.3.2. Design of the Action Space

The action space delineates the range of potential control commands an agent can execute, directly impacting the expressiveness of control strategies and the learning efficiency of algorithms. When addressing the intricate requirements for regulating lighting and thermal environments in digital twin classrooms, the design of the action space must balance control continuity, equipment’s physical constraints, and the feasibility of algorithm convergence.

This study developed a hybrid action space that integrated the key operational variables of both lighting and HVAC systems. For lighting adjustment, the system issues continuous brightness adjustment commands using either PWM duty cycles or 0–10 V voltage signals. This adjustment covers the entire physical range of the lighting fixture, from 0% to 100%. The minimum adjustment increment, typically between 3% and 5%, is determined based on human visual characteristics to ensure smooth transitions in the lighting environment. This threshold is selected based on the Weber-Fechner Law [39], which suggests that intensity changes within this range are smooth enough to prevent visual distraction for students while allowing fine-grained energy optimization.

For thermal comfort regulation, this process entails modifying the temperature setpoint and managing fan speed settings. The temperature setpoint remains within the human thermal comfort range, typically between 20 °C and 26 °C. Fan speed control is tailored to the equipment’s specifications, permitting either discrete settings or continuous frequency adjustments.

The action space comprises continuous dimming commands for the lighting system and setpoint adjustments for the air conditioning system. To reduce the frequent on-off cycling of equipment, we implemented a constraint mechanism within the action space [40].

The dimensions and granularity of the action space significantly affect the performance of reinforcement learning algorithms. A large action space can exponentially expand the search area, delaying policy learning. Conversely, overly coarse granularity may not provide the precision needed for effective control. To address these challenges, our study implemented a hierarchical action decision mechanism. This approach first determines the macro-control direction, such as heating/cooling or brightening/dimming, and then fine-tunes the micro-parameters. We also integrated physical constraint logic into the action output layer, which strictly limits the rate of change in control commands. This strategy effectively prevents frequent oscillations or sudden spikes in energy consumption due to the agent’s exploratory behavior, thereby protecting hardware and enhancing the stability of the user experience.

3.3.3. Design of Reward Function

In reinforcement learning, the reward function acts as the guiding force, and its design quality directly influences an agent’s ability to converge on an optimal strategy. Accurately quantifying the value relationship between environmental states and action choices is crucial during the design process. This quantification helps the agent balance multiple objectives in complex game tasks [41].

The reward function plays a crucial role in directing an agent’s learning process. In this study, we developed a composite reward function that integrates PMV thermal comfort, illuminance deviation, and energy consumption penalties.

To guide the RL agent toward a policy that balances multi-objective conflicts, we formulate a composite reward function ${\mathrm{R}}_{\mathrm{t}}$. The function integrates thermal comfort, visual satisfaction, energy efficiency, and control stability. The mathematical expression is defined as follows:

$${\mathrm{R}}_{\mathrm{t}}=\underset{\mathrm{Thermal} \mathrm{Comfort}}{\underbrace{{\mathrm{w}}_1\cdot (1-|\mathrm{P}\mathrm{M}{\mathrm{V}}_{\mathrm{t}}\left|\right)}}+\underset{\mathrm{Visual} \mathrm{Precision}}{\underbrace{{\mathrm{w}}_2\cdot \mathrm{e}\mathrm{x}\mathrm{p}(-\mathrm{\lambda }|{\mathrm{L}}_{\mathrm{o}\mathrm{b}\mathrm{s}}-{\mathrm{L}}_{\mathrm{s}\mathrm{e}\mathrm{t}}\left|\right)}}-\underset{\mathrm{Energy} \mathrm{Penalty}}{\underbrace{{\mathrm{w}}_3\cdot {\stackrel{\sim }{\mathrm{E}}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}}}-\underset{\mathrm{Action} \mathrm{Smoothness}}{\underbrace{{\mathrm{w}}_4\cdot \parallel \mathsf{\Delta }{\mathrm{A}}_{\mathrm{t}}{\parallel }_2}}$$

Thermal Comfort: The term $(1-|\mathrm{P}\mathrm{M}{\mathrm{V}}_{\mathrm{t}}\left|\right)$ serves as a soft constraint. It yields a positive reward when the predicted mean vote satisfies $\left|\mathrm{P}\mathrm{M}\mathrm{V}\right|<1$ (within the acceptable range defined by ISO 7730 [42]) and imposes a penalty when deviations exceed this threshold, encouraging the agent to maintain thermal neutrality. Visual Precision: An exponential kernel is employed, where $\mathrm{\lambda }$ is a sensitivity coefficient. This function provides a sharp gradient near the target illuminance (${\mathrm{L}}_{\mathrm{s}\mathrm{e}\mathrm{t}}$), incentivizing the agent to minimize the steady-state error of the lighting system precisely. Energy Penalty: ${\stackrel{\sim }{\mathrm{E}}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ represents the normalized total power consumption of HVAC and lighting systems at the current step. Minimizing this term drives the agent to seek the most energy-efficient valid state. Action Smoothness: $\parallel \mathsf{\Delta }{\mathrm{A}}_{\mathrm{t}}{\parallel }_2=\parallel {\mathrm{A}}_{\mathrm{t}}-{\mathrm{A}}_{\mathrm{t}-1}{\parallel }_2$ denotes the Euclidean norm of the action change vector. This penalty suppresses high-frequency oscillations (bang-bang control), thereby reducing mechanical wear on the actuators.

To ensure numerical stability, the weight coefficients ${\mathrm{w}}_1$ to ${\mathrm{w}}_4$ are calibrated via the Analytic Hierarchy Process (AHP) and empirically fine-tuned to balance the order of magnitude among the four terms.

To address the issue of sparse rewards, this study introduced a reward reshaping mechanism grounded in curriculum learning [43]. In multi-objective optimization, where conflicts like the trade-off between energy efficiency and comfort occur, a Pareto optimal weighted strategy was employed to achieve a dynamic balance [44]. Before integration, all sub-reward functions were normalized to ensure dimensional consistency. Furthermore, a dynamic weight adjustment mechanism was implemented to adaptively modify strategy preferences in various scenarios, such as prioritizing either energy efficiency or comfort.

3.3.4. Selection of Reinforcement Learning Algorithms

Selecting reinforcement learning algorithms necessitates a thorough evaluation of control task continuity, sample efficiency, and training stability. This study performed a comparative analysis of three prominent algorithms: Deep Deterministic Policy Gradient (DDPG) [45] Proximal Policy Optimization (PPO) [46] and Deep Q-Network (DQN) [47].

While DQN performs well in discrete action spaces, it encounters challenges with continuous variables that demand precise adjustments, such as the dimming ratios examined in this study. Discretizing the action space can result in the curse of dimensionality and inadequate control precision. Although DDPG is appropriate for continuous control, it is highly sensitive to hyperparameters, frequently leading to Q-value overestimation and unstable convergence during training.

This study utilized the Proximal Policy Optimization (PPO) algorithm, taking into account the continuity of control variables and sample efficiency. In the context of building control, the trust region mechanism of PPO is critical. It prevents the agent from making drastic updates to the policy that could lead to abrupt changes in actuator states (e.g., sudden fan speed spikes), thereby ensuring the physical safety of the equipment and the continuity of user comfort, which DDPG often fails to guarantee due to training instability. Unlike the Deep Deterministic Policy Gradient (DDPG) algorithm, PPO employs a clipping mechanism to limit the extent of policy updates. This approach led to improved convergence and robustness in building environment control tasks.

To ensure stable convergence in the continuous action space of the classroom environment, we adopt the PPO algorithm. The core innovation of PPO lies in its clipped surrogate objective function, which prevents destructive policy updates. The objective function ${\mathrm{L}}^{\mathrm{C}\mathrm{L}\mathrm{I}\mathrm{P}}(\mathsf{\theta })$ is mathematically formulated as follows:

$${\mathrm{L}}^{\mathrm{C}\mathrm{L}\mathrm{I}\mathrm{P}}(\mathsf{\theta })={\widehat{\mathbb{E}}}_{\mathrm{t}}\left[\mathrm{m}\mathrm{i}\mathrm{n}\left({\mathrm{r}}_{\mathrm{t}}(\mathsf{\theta }){\widehat{\mathrm{A}}}_{\mathrm{t}},\mathrm{clip}({\mathrm{r}}_{\mathrm{t}}(\mathsf{\theta }),1-\mathsf{ϵ},1+\mathsf{ϵ}){\widehat{\mathrm{A}}}_{\mathrm{t}}\right)\right]$$

where the components are defined as follows: $\mathsf{\theta }$ represents the parameters (weights) of the policy network. ${\mathrm{r}}_{\mathrm{t}}\left(\mathsf{\theta }\right)$ denotes the probability ratio between the new policy ${\mathrm{\pi }}_{\mathsf{\theta }}$ and the old policy ${\mathrm{\pi }}_{{\mathsf{\theta }}_{\mathrm{o}\mathrm{l}\mathrm{d}}}$, defined as ${\mathrm{r}}_{\mathrm{t}}(\mathsf{\theta })={\displaystyle \frac{{\mathrm{\pi }}_{\mathsf{\theta }}\left({\mathrm{a}}_{\mathrm{t}}\right|{\mathrm{s}}_{\mathrm{t}})}{{\mathrm{\pi }}_{{\mathsf{\theta }}_{\mathrm{o}\mathrm{l}\mathrm{d}}}\left({\mathrm{a}}_{\mathrm{t}}\right|{\mathrm{s}}_{\mathrm{t}})}}$. ${\widehat{\mathrm{A}}}_{\mathrm{t}}$ is the estimated Advantage function, indicating how much better a specific action is compared to the average action in that state. $\mathsf{ϵ}$: is the hyperparameter for the clipping range (set to 0.2 in this study).

This mechanism strictly constrains the probability ratio ${\mathrm{r}}_{\mathrm{t}}\left(\mathsf{\theta }\right)$ within the interval $[1-\mathsf{ϵ},1+\mathsf{ϵ}]$ when the advantage is positive, effectively establishing a trust region. This prevents the agent from making excessively large steps during optimization, thereby mitigating the risk of performance collapse, a critical requirement for maintaining safety in physical building control systems.

To validate the controller’s convergence performance, we conducted pre-training within a digital twin environment. Figure 3 presents the reward convergence curve of the agent during training, along with the Pareto frontier distribution, which illustrates the trade-off between energy efficiency and comfort.

Figure 3. Training performance and multi-objective optimization analysis. (a) The PPO training convergence process showing raw and smoothed rewards; (b) Energy vs. Comfort Pareto analysis, where the red dashed line indicates the Pareto Optimal Frontier. The color gradient of the dots (transitioning from dark purple to yellow) represents the increasing magnitude of energy consumption, corresponding to the shift from energy-saving strategies to comfort-oriented strategies.

3.4. System Integration

To rigorously validate the effectiveness and robustness of the proposed method, this study constructed a closed-loop experimental platform that integrates physical entities with virtual models. This phase aims to overcome the limitations of isolated simulation environments by combining high-fidelity digital twin models, deep reinforcement learning controllers, and physical hardware facilities. This integration bridges the gap between theoretical derivation and practical engineering applications for intelligent control strategies.

This study developed an integrated hardware-software experimental validation platform. The system architecture utilizes a cloud-edge collaborative model. Intensive training tasks were completed on the cloud-based digital twin platform, while real-time inference was deployed on the edge computing gateway. This approach aimed to minimize the impact of network latency on control stability.

To clearly demonstrate the operational logic of the system within the cloud-edge collaborative architecture, this study designed an interaction model for both data flow and control flow, as shown in Figure 4.

Figure 4. Data Flow and Control Logic in Cloud-Edge Collaborative Architecture.

3.4.1. Design of Data Flow and Control Flow

The essence of system architecture is the efficient coordination and orchestration of data and control flows. Their seamless integration constitutes the circulatory system of digital twin systems.

• Perceptual Data Stream: This component collects, transmits, and processes multidimensional environmental information in real time. Initially, data from various heterogeneous sensors—monitoring light, temperature, humidity, and air quality—are aggregated at an edge gateway. At this point, the data undergo cleaning, spatiotemporal alignment, and sensor fusion. The processed data support local inference and are uploaded to a cloud-based digital twin platform. This platform uses the data to calibrate virtual model parameters and update states, offering high-confidence state observations for reinforcement learning agents.
• Decision Control Flow: Dominating the Generation and Execution of Regulatory Commands. The reinforcement learning policy network, implemented at the edge, generates optimal control actions based on the current state. These commands are transmitted directly to physical actuators, such as smart lighting and HVAC systems, using low-latency industrial communication protocols like Modbus TCP or DALI. The actuators’ responses, combined with environmental feedback, generate new data inputs, effectively closing the control loop.

To guarantee real-time system responsiveness and maintain stable closed-loop control, the integration of data and control flows utilizes standardized RESTful API protocols alongside NTP time synchronization mechanisms. This architecture adopts a distributed processing strategy, delegating high-frequency, low-latency inference tasks to edge computing nodes situated near data sources. This method effectively minimizes network transmission delays and significantly improves the system’s fault tolerance during abnormal conditions, such as network disconnections.

3.4.2. Hardware and Software Co-Collaboration

The efficient functioning of a system depends on the physical support from hardware infrastructure and the logical guidance of software algorithms. Their close synergy dictates the overall system performance.

Hardware Layer (Physical Sensing and Actuation Layer): This layer functions as the system’s senses and actuators. It incorporates a high-precision environmental sensor array, featuring NDIR CO₂ sensors and thermocouple matrices. Additionally, it includes adjustable LED drivers, variable frequency air conditioning terminals, and high-performance edge computing gateways like the NVIDIA Jetson series. These devices are interconnected via IoT gateways, enabling the digital mapping of the physical world.

The Software Layer (Computational Logic and Decision-Making Layer): This layer integrates a digital twin simulation engine built on the EnergyPlus (v25.2.0) and Radiance (v1.12.4) core, a reinforcement learning algorithm module using the PyTorch framework (v2.9.1) framework, and middleware for processing heterogeneous data. This layer is essential for executing core functions, including data preprocessing, real-time model simulation, intelligent decision-making inference, and command distribution.

The interaction between hardware and software is facilitated through standardized MQTT (v5.0) message queues and Open API interfaces. To ensure efficient collaboration among heterogeneous components, the system employs a dynamic resource scheduling module. This module coordinates computing resources between the edge and the cloud, enabling real-time decision-making inference and balanced load distribution for backend model updates. This modular and standardized collaborative design effectively ensures the system’s long-term operational stability.

3.4.3. System Performance Evaluation

This study established a multidimensional performance indicator system to comprehensively quantify and evaluate the engineering effectiveness of the proposed system. The system included dynamic response, control accuracy, and energy efficiency levels.

Response time, denoted as (${\mathrm{\tau }}_{\mathrm{r}\mathrm{e}\mathrm{s}}$), refers to the interval between when the system detects an environmental disturbance, like a sudden person entering, and when the actuator completes its action. This metric directly indicates the system’s real-time performance.

Control accuracy indicators:

Thermal comfort compliance rate (${\mathrm{\eta }}_{\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}}$): This indicates the percentage of time the PMV index remains within the designated comfort zone ($[-0.5,+0.5]$).

The illuminance compliance rate, denoted as (${\mathrm{\eta }}_{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{u}\mathrm{a}\mathrm{l}}$), indicates the probability that the illumination on a work surface remains within the specified target range (${\mathrm{E}}_{\mathrm{s}\mathrm{e}\mathrm{t}}\pm 10\%$).

Energy Use Intensity (EUI): quantifies the total energy consumption per unit area over a specified time period. The formula is

$$\mathrm{E}\mathrm{U}\mathrm{I}={\displaystyle \frac{{\int }_0^{\mathrm{T}}({\mathrm{P}}_{\mathrm{h}\mathrm{v}\mathrm{a}\mathrm{c}}(\mathrm{t})+{\mathrm{P}}_{\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}(\mathrm{t}))\mathrm{d}\mathrm{t}}{\mathrm{A}\cdot \mathrm{T}}}$$

Among them, $\mathrm{P}\left(\mathrm{t}\right)$ represents the real-time power and $\mathrm{A}$ represents the classroom area.

System Robustness: This term describes a system’s capacity to sustain control performance without failure when encountering challenges such as sensor white noise interference, equipment communication delays, or external random disturbances, including abrupt changes in outdoor weather.

The evaluation method utilized a hybrid approach, combining large-scale simulations within a digital twin environment with field experiments in real classroom settings for mutual validation. Comparative experiments were conducted against traditional control strategies, including PID control, rule-based logic control, and fixed schedule control, to quantitatively assess the system’s advantages across various metrics. Preliminary experimental results showed that the integrated system achieved about 15% energy savings while maintaining a comprehensive comfort compliance rate exceeding 90%. Furthermore, it exhibited excellent response speed and robustness under high dynamic disturbances.

4. Experimental Design and Implementation

To scientifically and objectively evaluate the effectiveness and superiority of the proposed adaptive control strategy in real building environments, this study employed a rigorous simulation-field dual-validation experimental procedure. Initially, algorithm parameters were pre-trained and optimized within a high-fidelity digital twin environment. Subsequently, the algorithm’s generalization capability and robustness were validated in real-world conditions using a physical experimental platform.

4.1. Experimental Setup

The experimental platform was built using a standard teaching unit to simulate the common challenges of coupled light-thermal regulation found in university classrooms.

4.1.1. Classroom Model Construction

This study focused on a standard rectangular classroom measuring 10.8 m (L) × 8.4 m (W) × 3.5 m (H). To ensure high fidelity, the digital twin geometry strictly mirrored the physical site, including the window-to-wall ratio of 0.4 on the south facia and the specific thermal properties of the building envelope (e.g., wall U-value: 0.55 W/m²K).

To validate the algorithm’s generalization capability, we introduced a virtual data augmentation technique utilizing Generative Adversarial Networks (GANs) [48] to generate diverse scenarios of extreme weather and personnel distribution. Based on this augmented dataset, the model was calibrated using historical measured data. The mean absolute percentage error (MAPE) for key parameters, including illuminance, temperature, and relative humidity, remained within 5%, demonstrating the digital twin high fidelity and establishing a reliable foundation for subsequent training.

4.1.2. Installation of Sensors and Actuators

To synchronize data between the physical world and the digital twin, researchers implemented a high-density, heterogeneous Internet of Things (IoT) sensing and actuation network in the actual experimental classroom.

Figure 5 provides a schematic diagram illustrating the planar layout of sensor nodes and actuator devices, effectively showcasing the hardware configuration of the experimental setup. The images of sensors are shown in Figure 6. The technical specifications of the IoT sensor network can be find in Table 2.

Figure 5. Experimental Setup: Sensor and Actuator Layout Plan.

Figure 5. Experimental Setup: Sensor and Actuator Layout Plan.

Sensor subsystem:

(1)

Illumination Sensor Network: A network of illuminance sensors was evenly distributed at a working height of 0.75 m and on the ceiling to map the indoor light environment distribution in real-time.

(2)

Temperature and Humidity Monitoring Points: We deployed high-precision temperature and humidity sensors at the front, middle, and rear areas of the classroom, as well as in densely populated zones, to capture the spatial heterogeneity of the thermal environment.

(3)

Environmental Quality Monitoring: We installed NDIR carbon dioxide sensors in areas with inadequate air circulation to offer supplementary feedback for ventilation management.

(4)

Personnel Perception: Non-contact infrared thermal imaging sensors monitored the number of individuals and their spatial distribution density indoors in real time. This provided baseline data essential for predicting thermal load.

All sensors were set to a sampling frequency of 1 Hz, with microsecond-level time synchronization achieved through the Network Time Protocol (NTP).

Actuator subsystem:

(1)

The intelligent lighting system incorporates dimmable LED fixtures compatible with the DALI protocol, allowing for continuous brightness adjustment from 0% to 100%. These fixtures operate flicker-free and exhibit a response delay of less than 200 ms.

(2)

HVAC Control Interface: The system connects to the central air conditioning terminal using the Modbus protocol. This integration allows for both discrete and continuous control of temperature setpoints and fan coil unit (FCU) fan speed levels.

All sensing and execution devices integrate into the edge computing gateway using industrial Ethernet. This integration forms the foundational layer for physical perception and control within the system.

Figure 6. IoT Sensors: (a) Temp & Humidity Sensor; (b) Illuminance Sensor; (c) Ambient Gas Sensor.

Figure 6. IoT Sensors: (a) Temp & Humidity Sensor; (b) Illuminance Sensor; (c) Ambient Gas Sensor.

Table 2. Technical Specifications of the IoT Sensor Network.

Device Type	Key Parameter	Specification/Performance
1. Temp & Humidity Sensor	Accuracy	Temp: ±0.5 °C (25 °C); Hum: ±3%RH
	Resolution	Temp: 0.1 °C; Hum: 0.1%RH
	Response Time	Temp: ≤25 s; Hum: ≤8 s (1 m/s wind)
	Communication	ModBus-RTU (RS485)
2. Illuminance Sensor	Range	0–65,535 Lux (High Precision Mode)
	Accuracy	±4% (25 °C)
	Response Time	≤2 s
	Stability	≤1%/year
3. Ambient Gas Sensor	Parameters	CO2, TVOC, PM2.5 (Optional)
	Operating Range	−10 °C ~ 55 °C; 0~95%RH
	Output	RS485 (ModBus-RTU)/LoRa Wireless
	Features	Auto-calibration, OLED Display

Table 2. Technical Specifications of the IoT Sensor Network.

Device Type	Key Parameter	Specification/Performance
1. Temp & Humidity Sensor	Accuracy	Temp: ±0.5 °C (25 °C); Hum: ±3%RH
	Resolution	Temp: 0.1 °C; Hum: 0.1%RH
	Response Time	Temp: ≤25 s; Hum: ≤8 s (1 m/s wind)
	Communication	ModBus-RTU (RS485)
2. Illuminance Sensor	Range	0–65,535 Lux (High Precision Mode)
	Accuracy	±4% (25 °C)
	Response Time	≤2 s
	Stability	≤1%/year
3. Ambient Gas Sensor	Parameters	CO2, TVOC, PM2.5 (Optional)
	Operating Range	−10 °C ~ 55 °C; 0~95%RH
	Output	RS485 (ModBus-RTU)/LoRa Wireless
	Features	Auto-calibration, OLED Display

4.1.3. Agent Training

The training of reinforcement learning agents primarily depends on digital twin simulation environments, which offer safety, efficiency, and repeatability.

(1)

Training Environment: A calibrated high-fidelity digital twin model serves as the interactive environment.

(2)

Algorithm Configuration: The Proximal Policy Optimization (PPO) algorithm is utilized, as it demonstrates remarkable stability in controlling tasks within continuous action spaces.

(3)

Training Process: The agent iteratively interacts over millions of time-steps within a simulated environment. To improve sample efficiency, an Experience Replay Buffer is used [49]. Furthermore, a dynamic exploration noise decay strategy, specifically Ornstein-Uhlenbeck Noise [50], balances Exploration and Exploitation [51]. Throughout the training, the cumulative reward curve, PMV comfort convergence, and energy consumption levels were monitored in real-time to ensure the strategy consistently converged to the Pareto optimal frontier.

4.1.4. Training and Testing Environments

This study employed a virtual-physical integration experimental validation paradigm:

(1)

Simulation Training (Offline Training): By leveraging the parallel computing power of the cloud-based digital twin platform, we conducted extensive accelerated training under predefined scenarios. These scenarios encompassed different seasonal weather conditions, varied course schedules, and fluctuating personnel densities. This method effectively mitigated potential disturbances and risks linked to trial-and-error in real-world settings.

(2)

Online Validation: After maturing through simulation training, the strategy model was transferred and deployed to the edge gateway of a real classroom. During the field testing phase, the system operated independently over an extended period. The focus was on evaluating its inference latency, interference resistance, and long-term stability in a real-world physical environment.

The simulation training plus field validation methodology guarantees both high data throughput for algorithm training and the practical feasibility of engineering applications.

4.2. Comparison Benchmarks

To objectively evaluate the performance of the proposed deep reinforcement learning (DRL) control strategy, this study selected three classical strategies, commonly used in building environmental control, as baselines for comparative analysis.

4.2.1. Traditional PID Control

Proportional-Integral-Derivative (PID) controllers represent the most established feedback control technology and find widespread use in industrial applications.

(1)

Control Logic: $\mathrm{u}(\mathrm{t})={\mathrm{K}}_{\mathrm{p}}\mathrm{e}(\mathrm{t})+{\mathrm{K}}_{\mathrm{i}}\int \mathrm{e}(\mathrm{t})\mathrm{d}\mathrm{t}+{\mathrm{K}}_{\mathrm{d}}{\displaystyle \frac{\mathrm{d}\mathrm{e}}{\mathrm{d}\mathrm{t}}}$, denoting the deviation between the set value and the measured value.

(2)

Application Method: In the classroom, fixed target values are established for the lighting and air conditioning systems, specifically an illuminance of 500 lux and a temperature of 24 °C. Two PID control loops are independently operated to maintain these conditions.

(3)

Limitations Analysis: PID parameters are usually tuned with static methods, which find it difficult to adjust to the significant nonlinear characteristics and time-varying gains in classroom environments, such as sudden thermal load changes due to direct sunlight. Additionally, as a single-input single-output (SISO) system, the PID controller encounters challenges in managing the multivariable synergistic optimization of light and thermal coupling, frequently resulting in control oscillations between energy consumption and comfort.

4.2.2. Rule Control

Current building automation systems commonly use control strategies based on preset logic thresholds.

(1)

Control Logic: Utilizes Boolean logic in an if-then format.

(2)

Limitations Analysis: This strategy predominantly depends on thresholds determined by expert experience, lacking self-learning and adaptive capabilities. The inflexible rules are insufficient for handling complex environmental coupling relationships, often leading to frequent equipment start-stop cycles (short-cycling) in dynamic environments. Consequently, this approach generally maintains only a basic level of comfort while compromising energy-saving potential.

4.2.3. Schedule Control

Open-Loop Control Strategy Based on Fixed Time Series.

(1)

Application Method: The equipment’s on/off protocol is strictly adhered to as per the class schedule. For example, the air conditioning and lighting systems are activated 15 min prior to the start of class and turned off immediately after class concludes.

(2)

Limitations Analysis: This strategy fails to account for real-time environmental conditions, such as cloudy or sunny weather, as well as actual occupancy scenarios, like rescheduled classes or self-study sessions. Consequently, it results in energy waste, exemplified by lights on in empty rooms, or discomfort, such as overcrowded high temperatures, due to its lack of fundamental dynamic responsiveness.

4.2.4. Design of Comparative Experiments

This study established a rigorous A/B testing protocol to systematically compare the proposed RL strategy with the three previously mentioned benchmark strategies under identical environmental conditions.

(1)

Performance Metrics Collection: During the experiment, we simultaneously recorded thermal comfort (PMV), visual comfort (PPD), integrated energy consumption per unit time, and the system’s response time to environmental changes.

(2)

Robustness Stress Testing: In the experiment, we intentionally introduced external disturbances, including step changes in outdoor temperature and sudden increases in personnel density, to assess the interference resistance of various control systems. The evaluation centered on recovery time after disturbances and steady-state error.

(3)

Statistical Evaluation: To ensure the scientific validity of the experimental conclusions, this study uses t-tests and Analysis of Variance (ANOVA) to analyze the statistical significance of the collected data. These tests determine whether the reinforcement learning strategy provides a statistically significant benefit in improving comfort and reducing energy consumption.

5. Results and Discussion

To validate the proposed framework, the evaluation was conducted through simulation and field experiments. For simulation, we utilized the digital twin environment to train the RL agent and analyzed algorithmic convergence without physical risks. As illustrated in Figure 2, the PPO agent interacted with the virtual environment for 2000 episodes (approx. 1.5 million steps). For field experiment validation, a series of rigorous comparison experiments were conducted over 10 working days. The following sections provide an in-depth analysis of the experimental data from three dimensions: thermal comfort stability, energy efficiency levels, and multi-objective trade-off mechanisms, using statistical analysis and visual charts.

5.1. Comparison Between Thermal Comfort and Visual Comfort

Initially, we evaluated the effectiveness of different control strategies in maintaining indoor environmental quality (IEQ). The experiment took place on a typical school day from 08:00 to 18:00, during which we introduced random fluctuations in occupancy, as well as outdoor light and thermal disturbances.

In examining the thermal comfort index (PMV), PID control effectively maintained the basic setpoint under steady-state conditions but showed considerable lag in response to dynamic disturbances. Statistical tests demonstrated that the mean PMV difference between the two strategies was not significant overall ($p=0.78$). However, an analysis of variance (ANOVA) indicated that PMV fluctuations were significantly less variable under RL control compared to PID control ($\mathrm{F}=12.4,p<0.01$). This finding suggests that the RL strategy offers greater resistance to disturbances, enabling it to maintain the thermal environment more consistently within the comfort zone ($\mathrm{P}\mathrm{M}\mathrm{V}\in [-0.5,+0.5]$).

Regarding visual comfort, RL control successfully reduced instances of both excessive and insufficient illumination. It achieved this by dynamically adjusting the interaction between artificial lighting and natural daylight. As shown in Figure 7, during significant personnel increases at 10:00 and 14:00, the PID control (indicated by the red dashed line) experienced notable fluctuations in PMV values, with maximum deviations exceeding 0.8, which fell outside the comfort range. In contrast, the RL control (represented by the blue solid line) effectively anticipated load changes and used a pre-adjustment mechanism to maintain PMV fluctuations within 0.3. Similarly, when comparing visual comfort indicators (PPD), the RL control consistently kept dissatisfaction rates below 10%, significantly outperforming the fluctuation levels of PID control. This demonstrates that the RL agent successfully learned the environment’s dynamic characteristics, ensuring consistency in both light and thermal conditions while significantly reducing the system’s recovery time.

Figure 7. Temporal Performance Comparison: Thermal & Visual Comfort.

5.2. Energy Efficiency Optimization Analysis

Operational energy efficiency is a critical metric for assessing the effectiveness of control strategies. This study provided a detailed breakdown and statistical analysis of cumulative energy consumption during the experimental period.

Experimental data indicate that under identical comfort constraints ($\left|\mathrm{P}\mathrm{M}\mathrm{V}\right|\le 0.5, \mathrm{Lux}\in \left[\mathrm{300,500}\right]$), the RL control strategy demonstrates significant energy-saving potential. Independent sample t-tests revealed that the daily average energy consumption of the RL strategy was significantly lower than that of the PID strategy ($\mathrm{t}=15.6, p<0.0001$), achieving an energy-saving rate of up to 28.4%. Furthermore, to benchmark against advanced control benchmarks, we compared the proposed method with Model Predictive Control (MPC) mechanisms cited in the recent literature [52,53].While MPC strategies typically achieve comparable energy savings (approximately 25–30%) by solving optimization problems at each time step, they incur a high computational burden unsuitable for low-cost edge devices. In contrast, our deployed RL agent operates via pre-trained neural network inference. Quantitative analysis reveals that the computational latency of our RL method is reduced by approximately 98% compared to standard MPC solvers (milliseconds vs. seconds), validating its distinct superiority for real-time deployment on resource-constrained edge gateways.

Figure 8 shows that PID control (gray area) often follows a boom and bust regulation pattern, leading to frequent overshooting and unnecessary energy waste. In contrast, RL control (green area) exhibits a more proactive regulation strategy. For example, it leverages outdoor natural cooling sources or reduces load in anticipation of occupants leaving, thereby decreasing peak power usage. Notably, during transitional seasons, the RL agent effectively uses natural light to offset indoor illumination, cutting lighting energy consumption by about 40%. This contrast underscores the substantial potential of reinforcement learning in enhancing energy efficiency for nonlinear, time-varying systems. It illustrates that the agent has progressed from basic feedback control to an energy management expert with perception-prediction-decision capabilities.

Figure 8. Energy Efficiency Analysis: Power Profile and Cumulative Consumption.

5.3. The Pareto Trade-Off Between Comfort and Energy Consumption

This study developed a multi-objective optimization approach utilizing Pareto Frontier analysis to investigate the inherent contradiction and balance mechanism between comfort and energy consumption.

In practical applications, focusing exclusively on comfort typically results in increased energy consumption, and the reverse is also true. The primary benefit of the RL controller is its multi-objective weighting mechanism within the reward function, which allows it to identify optimal solutions within the feasible domain. Figure 9 shows the distribution of different control strategies on the energy consumption-comfort deviation two-dimensional plane.

Figure 9. Multi-objective Optimization Analysis: Pareto Frontier.

As shown in Figure 9, the PID control data points (red scatter points) are widely dispersed, mainly clustering in the upper right quadrant. This suggests that high energy consumption has not effectively improved thermal comfort. In contrast, the reinforcement learning control data points (blue scatter points) are tightly clustered in the lower left quadrant, nearing the theoretical Pareto optimal boundary. By adjusting the weight factor $\mathrm{\lambda }$ in the reward function, the reinforcement learning agent can dynamically switch between energy-saving priority and comfort priority modes. In all modes, its overall utility score significantly exceeded that of the baseline control strategy.

In conclusion, the experimental results robustly validated the adaptive control strategy based on digital twins and reinforcement learning. This strategy not only surpassed traditional methods in steady-state performance but also demonstrated significant advantages in robustness under dynamic disturbances and multi-objective collaborative optimization. These findings offer solid empirical evidence supporting the low-carbon operation of future smart classrooms.

5.4. Analysis of Transient Response and System Resilience

The dynamic recovery capability of a system, which indicates its ability to quickly return to a steady state following an external step disturbance, is an essential metric for evaluating the effectiveness ${\mathrm{\tau }}_{\mathrm{r}\mathrm{e}\mathrm{c}}$ of adaptive control strategies. To quantify this performance, this study simulated a typical cold load pickup scenario, such as when a class break ends and 40 students simultaneously enter a classroom and turn on electronic devices. We introduced the recovery time constant, denoted as ${\mathrm{\tau }}_{\mathrm{r}\mathrm{e}\mathrm{c}}$, as a measure of evaluation. This constant is defined as the time needed $\mathrm{x}\left(\mathrm{t}\right)$ for the system state variable to first return to and stabilize within the target range after $\pm \mathrm{\delta }$ the disturbance occurs at time ${\mathrm{t}}_0$.

$${\mathrm{\tau }}_{\mathrm{r}\mathrm{e}\mathrm{c}}=\mathrm{m}\mathrm{i}\mathrm{n}\{\mathrm{t}>{\mathrm{t}}_0:|\mathrm{x}\left(\mathrm{t}\right)-{\mathrm{x}}_{\mathrm{s}\mathrm{e}\mathrm{t}}|\le \mathrm{\delta },\forall \mathrm{\tau }>\mathrm{t}\}$$

The experimental results showed that when exposed to environmental thermal shocks of equal intensity, the control strategy based on reinforcement learning (RL) demonstrated exceptionally rapid response characteristics. In comparison to traditional PID control, the RL controller decreased the average recovery time by about 31.5%.

Figure 10 clearly illustrates the dynamic response curve of the PMV following a sudden increase in personnel disturbance. When a step disturbance occurs at time $\mathrm{t}=100 \mathrm{s}$, the PID controller (represented by the red dashed line) experiences significant overshoot and prolonged oscillatory decay due to the cumulative effect of the integral component. This causes the PMV value to deviate from the comfort zone for an extended period ($\mathrm{P}\mathrm{M}\mathrm{V}>0.5$). In contrast, the RL controller (depicted by the blue solid line) benefits from its implicit modeling of environmental dynamics, which allows it to anticipate changes in state trends and output compensatory actions in advance. As a result, it quickly converges to the target value along a nearly critically damped trajectory. This predictive-feedforward mechanism enables the RL strategy to maintain environmental stability far more effectively than the PID strategy, which relies solely on error feedback.

Figure 10. Transient Response Analysis: Comparison of System Recovery Time.

Beyond transient disturbances, regarding long-term system sustainability, we addressed the challenge of model maintenance under layout changes. To minimize operational disruption, the framework employs a Transfer Learning strategy where feature extraction layers are frozen, reducing the data requirement for retraining to approximately two weeks of historical logs. During this model update window, the system is designed to automatically revert to a robust rule-based fallback mode to maintain standard ISO setpoints, ensuring continuous and safe environmental control without requiring manual intervention.

Furthermore, regarding operational reliability under abnormal scenarios, we simulated instances of sensor data packet loss and communication interruptions. Results verified that the system’s fail-safe mechanism successfully identifies signal loss within 10 s and automatically switches to the aforementioned rule-based fallback mode, preventing control instability during hardware malfunctions.

5.5. Analysis of Environmental Robustness

Robustness analysis assesses the ability of control systems to maintain stability in the face of multi-source stochastic disturbances and parameter uncertainties. In this study, we constructed a complex test scenario incorporating strong optothermal coupling interference. This scenario simulated high-frequency fluctuations in solar radiation under cloudy weather conditions, with illuminance drastically varying between 200 and 800 lux, and was superimposed with random drifts in outdoor ambient temperature.

The experimental results showed that the reinforcement learning agent displayed remarkable generalization abilities and resilience to interference. When external environmental parameters underwent significant fluctuations, the reinforcement learning controller promptly reconfigured its control strategy. It effectively minimized the impact of these disturbances on indoor environmental parameters by collaboratively adjusting the angles of blinds, the intensity of artificial lighting, and the power of the HVAC system.

Figure 11 illustrates the error distribution and stability performance of different control strategies under composite disturbance conditions. When managing high-frequency light fluctuations, PID control frequently results in over-bright or over-dark indoor lighting due to actuator lag. This leads to significant jagged fluctuations in the thermal comfort PMV curve, indicating the system’s approach to instability. In contrast, the RL control strategy exhibits exceptional robustness, consistently keeping the PMV and illumination curves within the Comfort Band. This stability is due to the domain randomization technique used during RL training, which allows the agent to practice various extreme conditions in a digital twin environment. As a result, the system demonstrates composed adaptability during actual deployment. Furthermore, quantitative sensitivity analysis indicates that the control policy maintains operational stability, with PMV violation rates remaining below 5%, even when key internal physical parameters—specifically thermal transmittance coefficients and solar heat gain coefficients—deviate by up to ±15% from the actual values. This resilience validates that the error boundaries established during the domain randomization training phase effectively mitigate the risk of policy drift caused by simulation-to-reality mismatch.

Figure 11. Robustness Analysis: System Response to Stochastic Multi-source Disturbances.

5.6. Statistical Significance Verification

To ensure the reliability of the experimental results, this study utilized a two-sample t-test and an F-test for variance on the key performance indicators. The statistical outcomes are detailed in Table 3. The t-test results for energy consumption data revealed a significant effect ($p<0.0001$), with a Cohen’s d of 2.58, which greatly surpassed the commonly defined large effect threshold (d = 0.8). This finding demonstrated that the reinforcement learning control strategy provided a substantial and significant advantage in energy savings, rather than being due to random error. The deviation of thermal comfort was significantly reduced. The t-test results for the average absolute PMV value ($p<0.0001$) showed a highly significant difference. The RL strategy decreased the mean from 0.38, as observed with PID control, to 0.21. This finding demonstrates that RL control more accurately maintains the environment near the ideal comfort point (PMV = 0), effectively reducing periods of discomfort caused by overcooling or overheating. The F-test for the variance of PMV (volatility) indicated a significant difference ($p<0.05$). The variance under PID control was three times greater than that under reinforcement learning control (0.12 vs. 0.04). This finding confirmed that the reinforcement learning strategy exhibited greater convergence and robustness when encountering random disturbances, thus offering a more stable indoor environment experience.

Table 3. Overview of Statistical Test Results for Performance Variations Across Control Strategies, where * represents p ≤ 0.05, ** represents p ≤ 0.01, *** represents p ≤ 0.001.

Metric Category	Performance Indicator (Unit)	PID Baseline (Mean ± SD)	RL Strategy (Mean ± SD)	Improvement (%)	t-Statistic	p-Value
Energy Efficiency	Daily Energy Consumption (kWh)	$18.42\pm 2.15$	$13.19\pm 1.88$	$\downarrow 28.4\%$	$15.62$	<0.0001 ***
	Average Power (W)	$768.5\pm 89.4$	$549.6\pm 78.2$	$\downarrow 28.5\%$	$14.85$	<0.0001 ***
Thermal Comfort	Mean Absolute PMV	$0.38\pm 0.15$	$0.21\pm 0.08$	$\downarrow 44.7\%$	$8.45$	<0.0001 ***
	PMV Variance (${\mathsf{\sigma }}^2$)	$0.12$	$0.04$	$\downarrow 66.7\%$	$\mathrm{F}=3.00$	<0.05 *
System Resilience	Recovery Time (${\mathrm{\tau }}_{\mathrm{r}\mathrm{e}\mathrm{c}}$, min)	$24.5\pm 5.2$	$16.8\pm 3.1$	$\downarrow 31.4\%$	$9.82$	<0.0001 ***
Visual Comfort	Illuminance Deviation (Lux)	$85.6\pm 32.4$	$24.3\pm 10.5$	$\downarrow 71.6\%$	$13.20$	<0.0001 ***

5.7. Discussion and Limitations of this Work

The digital twin-driven reinforcement learning control framework proposed in this study successfully addressed the longstanding energy consumption-comfort trade-off challenge in conventional building environmental control.

Unlike existing single-objective optimization studies, such as model predictive control methods that concentrate solely on energy savings in HVAC systems, this research introduces a novel approach to multi-objective collaborative optimization within a photothermal coupled environment. Traditional proportional-integral-derivative (PID) control is constrained by its linear feedback mechanism, which struggles to manage the nonlinear and lag-prone nature of building thermodynamic systems. Consequently, this often results in control inaccuracies under dynamically changing conditions. In this study, a digital twin platform offers an expansive trial-and-error environment for reinforcement learning agents, allowing them to develop robust control strategies that effectively address complex dynamic changes. The objectives of energy saving and comfort are inherently conflicting (Pareto-conflicting). A simple summation without weights would likely lead the agent to a trivial solution (e.g., turning off all systems to save max energy, disregarding comfort). We utilized a weighted sum approach where weights (w₁, w₂, w₃) were tunable hyperparameters. By setting higher weights for comfort violation penalties (w₁, w₂), we enforced a soft constraint: the agent learned that maintaining ISO-compliant conditions is the prerequisite for exploring energy-saving actions. This guides the policy convergence toward the useful segment of the Pareto frontier.

Regarding the robustness of these hyperparameters, a sensitivity analysis was conducted. Results indicate that varying the weights by ±10% leads to less than a 4.2% shift in total energy consumption, suggesting that the system is stable around the chosen operating point and not overly reliant on precise tuning. Although the current weights are static, derived from AHP consensus for standard teaching modes, future iterations could incorporate dynamic weight adjustment to adapt to extreme seasonal shifts.

Experimental data confirmed that this strategy increased the system’s response speed by over 30% and significantly reduced energy consumption without compromising comfort. This virtual-real integration and virtual control of reality regulation paradigm provided crucial theoretical validation and a technical pathway for the future development of intelligent buildings towards cognitive and self-evolving models. The experimental results further demonstrated the superiority of the DT-RL framework in dynamic environments. Compared to recent similar studies [54,55], which focused solely on single-dimensional thermal comfort control, our research overcame the limitations of local optimization. To rigorously validate the advancement of the proposed DT-RL framework, we benchmarked our results against recent pivotal studies in the field. First, regarding energy efficiency, our system achieved a 28.4% reduction in total energy consumption. This performance significantly outperforms standard PID and rule-based control strategies, which, as highlighted in the comprehensive review by Wang and Hong [55] typically struggle to exceed 10–15% savings due to their inability to handle thermal inertia and delayed system responses. Furthermore, compared to specific Deep Reinforcement Learning implementations, such as the Deep Q-Network (DQN) approach presented by Fang et al. [56], which achieved approximately 16.6% to 22% energy savings in simulated office environments, our PPO-based method demonstrates superior performance. The advantage lies in PPO’s ability to handle continuous action spaces (e.g., precise dimming of lights from 0–100% rather than discrete steps), allowing for finer granularity in control and avoiding the bang-bang control oscillations common in discrete DQN approaches. Consequently, the DT-RL framework not only exceeds the energy baselines established in the recent literature but also provides a more stable thermal environment by leveraging the predictive capabilities of the digital twin.

The comparison between recent works and our results demonstrates that our multi-objective approach achieves a superior balance. For facility managers, this study demonstrates that deploying edge-based AI control does not require replacing existing HVAC infrastructure but rather upgrading the sensing layer. For policymakers, the results support the inclusion of adaptive smart control credits in green building certification standards.

However, we frankly acknowledge that there are limitations of this research. The model is trained for a specific classroom geometry and may require Transfer Learning for new layouts. The current approach optimizes for average PMV, neglecting individual metabolic differences. Nevertheless, monitoring data from the distributed sensor network indicated that the spatial temperature variance across different seating zones was maintained within 0.5 °C. This suggests that while physiological personalization is a future goal, the current system effectively minimizes environmental asymmetry, ensuring a high baseline of uniform comfort for all occupants.

6. Conclusions

This study investigated the regulation of coupled light-thermal environments in smart classrooms by introducing a closed-loop adaptive control framework. Based on the simulation studies and field experiments, this study presents a Digital Twin-driven PPO framework for adaptive classroom control. Field experiments show that the system achieves substantial energy savings (28.4%) and enhances thermal stability (PMV variance -66.7%) compared to traditional methods.

Future work will also focus on developing advanced fault-tolerant reinforcement learning algorithms to further enhance system robustness against heterogeneous device failures in large-scale deployments.