Deep Reinforcement Learning-Based Joint Optimization Control of Indoor Temperature and Relative Humidity in Office Buildings

Abstract

Indoor temperature and relative humidity control in office buildings is crucial, which can affect thermal comfort, work efficiency, and even health of the occupants. In China, fan coil units (FCUs) are widely used as air-conditioning equipment in office buildings. Currently, conventional FCU control methods often ignore the impact of indoor relative humidity on building occupants by focusing only on indoor temperature as a single control object. This study used FCUs with a fresh-air system in an office building in Beijing as the research object and proposed a deep reinforcement learning (RL) control algorithm to adjust the air supply volume for the FCUs. To improve the joint control satisfaction rate of indoor temperature and relative humidity, the proposed RL algorithm adopted the deep Q-network algorithm. To train the RL algorithm, a detailed simulation environment model was established in the Transient System Simulation Tool (TRNSYS), including a building model and FCUs with a fresh-air system model. The simulation environment model can interact with the RL agent in real time through a self-developed TRNSYS–Python co-simulation platform. The RL algorithm was trained, tested, and evaluated based on the simulation environment model. The results indicate that compared with the traditional on/off and rule-based controllers, the RL algorithm proposed in this study can increase the joint control satisfaction rate of indoor temperature and relative humidity by 12.66% and 9.5%, respectively. This study provides preliminary direction for a deep reinforcement learning control strategy for indoor temperature and relative humidity in office building heating, ventilation, and air-conditioning (HVAC) systems.

1. Introduction

With a rapidly developing economy, the number of office buildings in China has gradually increased. One study reported that the area of office buildings in China has grown from 1.6 billion to 4.8 billion ${\mathrm{m}}^2$ in the past two decades [1]. An increase in building area has driven the growth of building energy consumption, and a study reported that the entire lifecycle energy consumption of buildings accounts for approximately 46.5% of the total energy consumption, and the total lifecycle carbon emissions of buildings account for 51.2% of total carbon emissions in China [2]. The total energy consumption and carbon emissions from office buildings are relatively high compared to other building types [3,4,5]. This indicates that office buildings are essential for economic development. Furthermore, office buildings are a kind of building where modern people spend much time. Their indoor environmental quality significantly affects the health and work efficiency of users [6]. Mechanical and electrical equipment in office buildings also provides various convenient services for humans, such as air-conditioning equipment, elevators, and lighting. In general, office buildings are among the most important buildings that support economic growth, stimulate investments, and facilitate services in any country. Therefore, research on various aspects of office buildings is crucial for social and economic development, human health, and air quality.

As the location of human activities gradually shifts from outdoors to indoors, humans spend on average 80–90% of their time inside buildings, especially office buildings [7], which increases requirements for the environment and indoor thermal comfort of office buildings. Studies [8,9] have indicated that the office building environment and thermal comfort affect occupant health and work efficiency. With a suitable office environment and thermal comfort, occupants will have a greater sense of well-being, and increase their work efficiency by approximately 15–20%. Therefore, controlling the indoor-air status of office buildings within an appropriate range and improving the indoor environment quality of office buildings have received considerable attention from various scholars worldwide.

The indoor-air environment is mainly affected by outdoor weather, occupant behavior, and energy-using equipment [10]. Creation of the indoor-air environment mainly depends on various air-conditioning equipment. Currently, the number of office buildings in Chinese major cities is increasing, and fan coil units, as a type of air-conditioning equipment, have been widely used in heating, ventilation, and air-conditioning (HVAC) systems of office buildings because of their small size, high flexibility in arrangement, and individual control [11]. Existing research on the control of fan coil units has mainly focused on reducing the fluctuation of indoor temperature to obtain satisfactory indoor thermal comfort. This method of using only indoor temperature as a control object ignores the influence of indoor relative humidity on the human body and thermal comfort evaluation of the indoor environment by different groups of people in different relative humidities. Relative humidity affects human thermal comfort mainly by influencing heat and water–salt metabolism in the human body [12], and different people have different sensitivities to indoor relative humidity. For most people, fluctuations in indoor relative humidity within an appropriate range at the same indoor temperature do not significantly affect their thermal comfort evaluation of indoor environments, whereas, for people with respiratory diseases, differences in relative humidity significantly increase their discomfort and affect their actual thermal comfort evaluation of indoor environment [13,14]. Therefore, it is crucial to develop a control method for fan coil units to jointly control the indoor temperature and relative humidity of office buildings within an appropriate range for the health status of occupants, improvement in occupant work efficiency, and thermal comfort evaluation.

Currently, commonly used control methods for fan coil units usually consider only indoor temperature as the control object, such as on/off control, rule-based control (RBC), and proportional–integral–derivative (PID) control. These control methods are widely used in actual projects owing to their simple deployment. For example, a PID controller was proposed in [15] and its control effect on indoor temperature and relative humidity was tested. Lifei Xu of the Harbin Institute of Technology [16] designed a cascade control system for indoor temperature and relative humidity, and optimized the performance of the controller by self-tuning the parameters of the PID controller using artificial neural networks (ANNs). However, HVAC systems, as a class of highly nonlinear time-varying systems, often have difficulty achieving the desired control effect using conventional control methods [17]. Recently, the application of model predictive control (MPC) to HVAC systems has received considerable attention. The MPC as a supervisory control has better stability and multiobjective rolling optimization, but the operation effect of MPC depends on accurate mathematical models, and requires data information that can accurately reflect changes in indoor and outdoor building parameters [18]. If the difference between the mathematical model and the actual HVAC system is significant, the control effect of the MPC is difficult to ensure.

With the development of big data technology and artificial intelligence (AI), a machine learning method that is model-free and self-learning has emerged in recent years called reinforcement learning (RL) [19,20,21]. Some scholars have conducted research on the optimal control of RL algorithms in HVAC systems.

Table 1. Summary of the RL algorithm for optimal control of HVAC system.

Reference	Building	HVAC System	Control Action	Optimization Objective	RL Algorithm
Junwei Yan et al. [22]	Office building	Water-cooled central air-conditioning system	Chilled water outlet temperature and chilled water flow	Energy consumption and indoor air temperature	Double DQN
Xi Fang et al. [23]	Office building	VAV	Air supply temperature setpoint and chilled supply water temperature setpoint	Energy consumption and thermal comfort	DQN
Ruihua Ding et al. [24]	Data center	Water-cooled central air-conditioning system	Chilled water outlet temperature and the pressure difference of chilled water pump	Energy consumption and temperature of air inlet area of cabinet	DQN based on expert knowledge
Yan Du et al. [25,26]	Residential building	Split air conditioner	Zone temperature setpoint	Energy consumption cost and thermal comfort	DDPG
Zhiang Zhang et al. [27]	Office building	Water-based radiant heating system	Supply water temperature setpoint	Energy consumption and thermal comfort	A3C
Marco Biemann et al. [28]	Data center	VAV	Zone temperature setpoint and zone fan mass flow rate	Energy consumption and indoor temperature	Actor–critic
Guanyu Gao et al. [29]	Laboratory	Split air conditioner	Air temperature setpoint and humidity setpoint	Energy consumption and thermal comfort	DDPG
Yiqun Pan et al. [30]	Office building	VAV	Air supply volume	Energy consumption and indoor temperature	DQN

summarizes the related RL studies for the optimal control of building HVAC systems. Junwei Yan et al. [22] applied the double deep Q network (DQN) algorithm to the energy-saving optimization operation of a central air-conditioning system in an office building in Guangzhou. In the premise of meeting indoor thermal comfort requirements, compared with PID control, this algorithm reduces the total energy consumption of the system by approximately 5.36%. Guangcai Gong et al. [23] applied the DQN algorithm to a variable air volume (VAV) system to save the total system energy and satisfy indoor thermal comfort. They verified that the control effect of the DQN algorithm was superior to RBC in most cases by controlling the setpoint of the air supply temperature and that of the chiller water supply temperature. Ruihua Ding et al. [24] proposed a deep reinforcement learning optimal control method based on expert knowledge to study a water-cooled air-conditioning system in a data center, and compared the method with traditional RBC and PID control to demonstrate that the method can reduce the total system energy consumption, while retaining the cabinet outlet air temperature within a safe range. Yan Du et al. [25,26] applied the deep deterministic policy gradient (DDPG) algorithm to a multizone residential HVAC system to minimize energy consumption costs while maintaining indoor environment thermal comfort. Zhiang Zhang et al. [27] proposed a control method based on the asynchronous advantage actor–critic (A3C) algorithm, which was then deployed in an actual radiant heating system for testing. The results demonstrated that the control method had over 95% probability of saving 16.6% of the heating demand during the deployment period. Marco Biemann et al. [28] evaluated four actor–critic algorithms in a simulated data center environment and demonstrated that all four algorithms could achieve zone temperature maintenance within the desired range while reducing energy consumption by 10% compared to a model-based controller. Guanyu Gao et al. [29] used the DDPG algorithm to regulate a HVAC system to reduce energy consumption while meeting the thermal comfort requirements of occupants. Yiqun Pan et al. [30] used a VAV air-conditioning system for an office building as a case study to validate the optimization performance of an RL controller-based DQN algorithm. They demonstrated that the RL controller is more energy efficient than RBC and PID controllers in meeting indoor temperature requirements. Currently, research on the application of reinforcement learning algorithms in HVAC systems mainly focuses on reducing the total system energy consumption while meeting indoor temperature requirements, and its control objects are mostly various setpoints. This ignores the potential risk of not meeting indoor relative humidity and deviation between setpoints and actual operating conditions.

From the literature review, in the control research of fan coil units, the current common control method considers temperature as a single control object and disregards the effect of indoor relative humidity. On the one hand, owing to the coupling relationship between indoor temperature and relative humidity, it is difficult to regulate fan coil units to maintain both the indoor temperature and relative humidity of office buildings within an appropriate range, and there are still relatively few related studies. On the other hand, as machine learning control methods, RL algorithms have appeared in recent years, and there have been preliminary studies on their application in HVAC systems, but most of these studies focus on reducing system energy consumption in the premise of meeting only indoor temperature requirements, and control objects are often various setpoints. There are few studies on joint control of both indoor temperature and relative humidity in office buildings using RL algorithms. Therefore, it is significant to study the RL algorithm for the joint control of indoor temperature and relative humidity. To solve these problems, this study considers fan coil units with a fresh-air system in an office building in Beijing as the study object, develops a TRNSYS–Python co-simulation platform, and proposes a reinforcement learning algorithm based on action intervention to regulate the air supply volume for the fan coil units. The study objective is to improve the joint control satisfaction rate of indoor temperature and relative humidity. In summary, this study provides preliminary direction for a deep reinforcement learning control strategy for indoor temperature and relative humidity in office building HVAC systems.

The remaining parts of this article is organized as follows: Section 2 introduces methodology, including overall technical approach, establishment of simulation environment, algorithm principle and design, co-simulation platform operating principle, and algorithm evaluation; Section 3 shows the optimization control results of the controller proposed in this study, which are compared with other controllers. Additionally, the sensitivity analysis results of the DQN algorithm are also shown in this section; lastly, the conclusions and limitations are summarized in Section 4.

2. Methodology

2.1. Overall Technical Approach

In this study, an RL algorithm based on action intervention was proposed to regulate the air supply volume for the fan coil units, commonly used in office buildings in China. This study used a fan coil unit with a fresh-air system in an office building as a case study to validate the optimization control performance of the RL controller proposed in this study. The objective was to improve the joint control satisfaction rate of indoor temperature and relative humidity. The overall technical approach of this study is illustrated in Figure 1, and is divided into four parts:

• Establish a building virtual simulation environment. The building and its energy system are modeled in TRNSYS software, which provides an interactive environment for subsequent agent training.
• Design and deployment of a reinforcement learning algorithm. To improve the joint control satisfaction rate of indoor temperature and relative humidity, this study designed an RL algorithm with advanced applicability for regulating the air supply volume for the fan coil units. The algorithm was deployed in TensorFlow.
• Development of the TRNSYS–Python co-simulation platform. In this study, real-time interactions between TRNSYS and Python were realized using a data transfer method. This method is based on files. A co-simulation platform was developed for RL algorithm testing and evaluation.
• Algorithm evaluation. For the joint control effect on indoor temperature and relative humidity, the RL control algorithm proposed in this study was compared with the traditional control method. Subsequently, sensitivity of the RL algorithm was analyzed.

2.2. Establishment of Simulation Environment

In this study, the Transient System Simulation Tool (TRNSYS) was used to build a simulation environment. TRNSYS is an extremely flexible graphically based software environment used to simulate the behavior of transient systems. It is a modular system with a large component library, and users can create their own models. The TRNSYS software has been widely used to study the performance simulation of HVAC components and systems. The software is also verified by comparing with the experimental setup. For example, Martinez et al. [31] modeled an air system with a desiccant wheel in TRNSYS, and then designed a test facility to verify the effectiveness of the model.

This study established a simulation environment in TRNSYS based on weather data, building information, and HVAC equipment information collected on-site. The simulation environment is used for subsequent algorithm training, testing, and evaluation.

2.3. Reinforcement Learning Algorithm Design and Deployment

2.3.1. Reinforcement Learning Introduction

In this study, an RL control algorithm based on action intervention was proposed to regulate the air supply volume for the fan coil units to improve the joint control satisfaction rate of indoor temperature and relative humidity in office buildings.

Reinforcement learning is the third basic learning method in machine learning, in addition to supervised and unsupervised learning. Its inspiration comes from behaviorism theory in psychology, which focuses on the idea that organisms constantly interact with the environment to obtain rewards or punishments given by the environment, and then gradually form expectations of rewards and punishments to produce actions that can obtain maximum benefits [32]. Figure 2 shows a schematic of the reinforcement learning algorithm.

In Figure 2, $\mathit{S}$ stands for the state and observation of the agent, $\mathit{A}$ stands for an action taken by the agent, and $\mathit{R}$ stands for the reward given to the agent by the environment. The specific interaction process follows: at each decision moment $\mathit{t}$, the agent executes the action ${\mathit{a}}_{\mathit{t}}$, and after a time step $\Delta \mathit{t}$, the environment is at moment $\mathit{t}+1$, and the state changes from ${\mathit{s}}_{\mathit{t}}$ to ${\mathit{s}}_{\mathit{t}+1}$. The agent observes ${\mathit{s}}_{\mathit{t}+1}$ and realizes the reward $\mathit{R}\left({\mathit{s}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}}\right)$ in this time step, which is fed back by the environment.

The iterative object of the RL algorithm is the maximum expected reward value function $Q$ based on the state–action pair, represented by $Q\left(s_t,a_t\right)$, which is the cumulative reward value that the system will obtain when the action $a_t$ is executed in state $s_t$. Through the continuous interaction between the agent and environment, the $Q$ value is updated by Equation (1).

$$Q^{new}\left(s_t,a_t\right)\leftarrow \left(1-\alpha \right)·Q^{old}\left(s_t,a_t\right)+\alpha ·\left(R\left(s_t,a_t\right)+\gamma ·maxQ\left(s_{t+1},a\right)\right)$$

(1)

where $\alpha$ is the learning rate, $\alpha \in \left(0,1\right]$. When the learning rate approaches one, the algorithm converges faster, but the risk of oscillation is higher; when the learning rate approaches zero, the algorithm converges slower but the risk of oscillation is lower. Let $\gamma$ denote the discount factor $\gamma \in \left[0,1\right]$, which means the effect of the current action on future long-term rewards. The larger $\gamma$ is, the more the agent values long-term rewards obtained in the future; conversely, the smaller $\gamma$ is, the more myopic the agent is regarding the rewards.

In an actual HVAC system, there are various devices and sensors, the dimensions of the state are large, and many states are continuous rather than discrete. Calculating each $Q\left(s_t,a_t\right)$ is complicated and inefficient. To solve this problem, a method for estimating the Q value using artificial neural networks (ANNs) was proposed. The input of the ANNs is the state, and its output is Q value for each action. Such RL algorithms equipped with ANNs are called deep reinforcement learning (DRL) algorithms. The deep Q network (DQN) algorithm is a DRL algorithm with two ANNs (i.e., Q-network and target Q-network) and an experience memory. The Q-network must be trained to output the maximum Q-value. The target Q-network does not require training but only serves as a label for the Q-network when it is trained, and its parameters are updated from replicating the Q-network parameters over a fixed time step. The experience memory holds experience generated by the agent interacting with the environment, which is extracted and input into the Q-network as training data when the Q-network is being trained. The specific flow of the DQN algorithm is presented in Algorithm 1.

Algorithm 1: Deep Q Network Algorithm Flow

1: Initialize memory M = [empty set] 2: Initialize Q network with parameters ω 3: Copy Q network and store as $\widehat{Q}(·|\widehat{\omega })$ 4: Initialize control action a and state $s_{pre} \mathrm{and} s_{cur}$ 5: for m: = 1 to N do 6:          Reset the environment to the initial state 7:          for ts: = 0 to L do 8:                   if ts mod k == 0 then 9:                       $s_{cur}\leftarrow current observation$ 10:                       $r=reward\left(s_{pre},a,s_{cur}\right)$ 11:                       $M\leftarrow \left(s_{pre},a,r,s_{cur}\right)$ 12:                       $Draw mini-batch\left(s,a,r,s^{\prime }\right)\leftarrow M$ 13:                       $Target vectors v\leftarrow target\left(s\right)$ 14:                       $Train Q\left(·|\omega \right) with s,v$ 15:                       $Every d\Delta t_c steps,\widehat{Q}\left(·|\widehat{\omega }\right)\leftarrow Q\left(·|\omega \right)$ 16:                       $\epsilon =\max \left(\epsilon -·\epsilon ,min\epsilon \right)$ 17:                       $a=\left\{\begin{array}{c}{a}_i\in A | i=random\left(n\right), probality \epsilon \\ argmaxQ\left(s_{cur},a\right), otherwise\end{array}\right.$ 18:                       $s_{pre}\leftarrow s_{cur}$ 19:                 end if 20:                 Execute action a in the environment 21:       end for 22: end for

Considering that the data generated by the operation of HVAC system are considerably large and complex, the indoor-air state is a continuous variable rather than discrete. The DQN algorithm was used in this study to solve the optimal control problem of the air supply volume for the fan coil units.

2.3.2. Design of the DQN Algorithm

• Selection of input parameters for the DQN algorithm.

In optimal control strategies based on DRL algorithms, selecting state $S$ is important. The more influencing factors the state contains, the more comprehensive the information about the environment the agent receives, and the closer the final learned strategy is to the optimal control strategy. However, an increase in the state dimension leads to a longer training time and a more extensive space for the agent to explore, which increases the risk of failure in agent learning. Therefore, in this study, after numerous experiments, indoor temperature $tem$ and indoor relative humidity $RH$ are simultaneously selected as inputs for the DQN algorithm after conversion. These experiments mainly consider different combinations of input parameters and whether these input parameters need to be converted. Detailed experiment settings are shown in

Table 2. Detailed experiment settings.

Potential Input Parameters	Conversion
Indoor temperature	Yes or No
Indoor relative humidity	Yes or No
Indoor humidity ratio	Yes or No
Outdoor temperature	Yes or No
Outdoor relative humidity	Yes or No
Outdoor humidity ratio	Yes or No
Wind velocity	Yes or No
Occupancy	No
Total horizontal radiation	Yes

, and the conversion formulas are shown in Equations (2) and (3).

$$te{m}^{\prime }=\left\{\begin{array}{c}-1+\frac{1-\left(-1\right)}{T_{upper bound}-T_{lower bound}}·\left(tem-T_{lower bound}\right), if T_{lower bound}\le tem\le T_{upper bound}\\ 1+\left(tem-T_{upper bound}\right), if tem>T_{upper bound}\\ -1-\left(T_{lower bound}-tem\right), if tem<T_{lower bound}\end{array}\right.$$

(2)

$$R{H}^{\prime }=\left\{\begin{array}{c}-1+\frac{1-\left(-1\right)}{R{H}_{upper bound}-R{H}_{lower bound}}·\left(RH-R{H}_{lower bound}\right), if R{H}_{lower bound}\le RH\le R{H}_{upper bound}\\ 1+\left(RH-R{H}_{upper bound}\right)/10, if RH>R{H}_{upper bound}\\ -1-\left(R{H}_{lower bound}-RH\right)/10, if RH<R{H}_{lower bound}\end{array}\right.$$

(3)

where $tem$ and $RH$ denote the temperature and relative humidity before conversion, and $te{m}^{\prime }$ and $R{H}^{\prime }$ are the temperature and relative humidity after conversion, respectively. The purpose of Equation (2) is to distribute $te{m}^{\prime }$ between −1 and 1 when $tem$ is between $T_{lower bound}$ and $T_{upper bound}$. If $tem$ is greater than $T_{upper bound}$ or less than $T_{lower bound}$, $te{m}^{\prime }$ increases or decreases when the value of $tem$ linearly exceeds the boundary. Similarly, the purpose of Equation (3) is to distribute $R{H}^{\prime }$ between −1 and 1 when $RH$ is between $R{H}_{lower bound}$ and $R{H}_{upper bound}$. If $RH$ exceeds the upper or lower boundary, $R{H}^{\prime }$ increases or decreases with the values of $RH$ that exceeded the boundary, based on the scale of one tenth. This conversion keeps the scale of $R{H}^{\prime }$ close to that of $te{m}^{\prime }$.

• Output setting for the DQN algorithm.

The output of the DQN algorithm can be considered a controllable variable in a HVAC system. Based on the purpose of this study, the air supply volume for the fan coil units was selected as the output. The fan coil units used in this study have four levels of air supply volume: off, low, medium, and high, corresponding to 0%, 50%, 75%, and 100% of the rated air volume, respectively. Therefore, action space $A=\left[a_0,a_1,a_2,a_3\right]=\left[0, 50\%, 75\%, 100\%\right]$.

• Design of the reward function for the DQN algorithm.

In theory, an agent is trained to maximize the cumulative reward value. The design of the reward function determines the time an agent takes to train and whether the training is effective. According to the purpose of this study, the reward function is represented by the negative form of the temperature penalty and relative humidity penalty terms, as shown in Equations (4)–(6).

$$Reward=-k_1·penalt{y}_{tem}-k_2·penalt{y}_{RH}$$

(4)

$$penalt{y}_{tem}=\left\{\begin{array}{c}0, if -1\le te{m}^{\prime }\le 1\\ abs\left(te{m}^{\prime }\right)-1, else\end{array}\right.$$

(5)

$$penalt{y}_{RH}=\left\{\begin{array}{c}0, if -1\le R{H}^{\prime }\le 1\\ abs\left(R{H}^{\prime }\right)-1, else\end{array}\right.$$

(6)

where $k_1$ denotes the temperature penalty term coefficient and $k_2$ denotes the relative humidity penalty term coefficient.

• Exploration and exploitation of the DQN algorithm and hyperparameter setting.

In this study, we selected the ε-greedy exploration strategy to explore more state–action pairs, and the specific process is that in the training phase, a random number is generated at each time step, and if the random number is smaller than ${\epsilon }_i$ at this time, the agent randomly selects an action; otherwise, the agent selects an action based on the prediction of the Q-network. The formula for ${\epsilon }_i$ is shown in Equation (7).

$${\epsilon }_i={\epsilon }_0-{\epsilon }_{decay}·ste{p}_i$$

(7)

where ${\epsilon }_{decay}$ is the decay coefficient of ε and $ste{p}_i$ is the i-th time step.

In this study, we intervened in the actions of the agent to avoid meaningless exploration and enhance the utility of the RL controller. Specifically, during the training phase, if the indoor temperature was higher than $T_{upper bound}+2 °\mathrm{C}$, the air supply volume for the fan coil units was 100% of the rated air supply volume, and if the indoor temperature is lower than $T_{lower bound}-2 °\mathrm{C}$, fan coil units were turned off. Such a setting enables the agent to avoid meaningless exploration and reduce the computation cost of learning. During the testing phase, if the indoor temperature was higher than $T_{upper bound}$, the fan coil units were turned on to high airflow volume (i.e., 100% of the rated air supply volume), and if the indoor temperature was lower than $T_{lower bound}$, the fan coil units were turned off. On the one hand, this setting can prevent the agent from ignoring indoor temperature to obtain appropriate indoor relative humidity; on the other hand, it can also avoid damage to HVAC equipment.

The settings for the other hyperparameters in the DQN algorithm used in this study are listed in

Table 3. Hyperparameter settings.

Hyperparameter	Candidate Value	Selected Value
$k_1$	{1, 2, 5, 10}	5
$k_2$	-	1
$\alpha$	{0.001, 0.01, 0.1, 0.3, 0.5, 0.9}	0.01
$\gamma$	{0.01, 0.1, 0.3, 0.5, 0.9, 0.99}	0.1
${\epsilon }_{decay}$	{0.0001, 0.001, 0.01}	0.01
${\epsilon }_0$	{0.1, 0.3, 0.5, 0.8, 1}	0.3
Memory size	{32, 64, 128, 256}	256
Batch Size	{32, 64, 128, 256}	256
Target network update	{1 day, 3 days, 5 days, 7 days}	1 day
Number of hidden layers	{1, 2}	1
Number of neural units in hidden layer	{32, 64, 128, 256}	128
Activation function of hidden layer	-	Sigmoid
Activation function of output layer	-	Linear
Number of inputs	-	2
Number of outputs	-	4
Optimizer	-	Adam

.

2.4. TRNSYS–Python Co-Simulation Platform Development

The agent must be trained to learn the control strategy. During training, the agent must continuously receive information regarding the environment and output an action to be executed. If an untrained DQN algorithm is deployed in an actual building HVAC system, there is a risk of equipment damage and serious deviation of the indoor air from the comfort range. Therefore, in this study, a virtual simulation environment was built in the TRNSYS software for training the agent, testing, and evaluating the DQN algorithm. An RL controller based on the DQN algorithm was implemented in Python, and the artificial neural networks were built and trained in the free and open-source deep learning library TensorFlow. TensorFlow is an open-source software library developed by Google for various machine learning tasks in perception and language understanding.

To achieve real-time interaction between TRNSYS and the RL controller, we used a file-based data transfer method. Specifically, the RL controller writes a control action to the .in file, TRNSYS reads the file and executes the corresponding action, and after reaching the next simulation time step, TRNSYS writes information about the environment to the .out file, which is read by the RL controller. The data transfer principle is illustrated in Figure 3.

Based on the design of the DQN algorithm and the real-time interaction between the TRNSYS software and the RL controller, the overall architecture of the TRNSYS–Python co-simulation platform proposed in this study is shown in Figure 4.

2.5. Algorithm Evaluation

2.5.1. Metric for Training Convergence

The agent training must be stopped at the appropriate time. If the training time is overly short, the learning of the agent may be incomplete, and the reliability of the experience learned by the agent may be insufficient. If the training time is overly long, the artificial neural network may fall into the predicament of overfitting. Therefore, it is necessary to set an appropriate metric to determine whether the training of the agent should end. After repeated experiments, we select $Stepwise Average Reward$ as the metric for training convergence, as shown in Equation (8).

$$Stepwise Average Reward=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{r}_i$$

(8)

where $r_i$ denotes the value of the reward in the i-th time step and N denotes the number of time steps performed.

2.5.2. Comparison and Evaluation of Control Effects

To verify the effectiveness of the proposed RL controller for the joint control of indoor temperature and relative humidity, we selected on/off and rule-based controllers commonly used in various projects. The specific settings for these controllers are presented in

Table 4. Controller settings.

Controller	Indoor Temperature	Air Supply Volume for Fan Coil Units
On/Off controller	tem ≥ 27 °C	I	50% of the rated air volume
		II	75% of the rated air volume
		III	100% of the rated air volume
	tem ≤ 25 °C	0
	Elsewise	Keep the air supply volume constant
Rule-based controller	tem ≥ 26.7 °C	100% of the rated air volume
	tem ≤ 25.3 °C	0
	Elsewise	75% of the rated air volume

.

In this study, we selected the temperature satisfaction rate, relative humidity satisfaction rate, and joint control satisfaction rate of temperature and relative humidity as evaluation indices, which are calculated as shown in Equations (9)–(11).

$${\varphi }_{tem}=\frac{n_{tem}}{N}·100\%$$

(9)

$${\varphi }_{RH}=\frac{n_{RH}}{N}·100\%$$

(10)

$${\varphi }_{tem\&RH}=\frac{n_{tem\&RH}}{N}·100\%$$

(11)

where $n_{tem}$ is the number of indoor temperature points within the upper and lower limits; $n_{RH}$ is the number of indoor relative humidity points within the upper and lower limits, $n_{tem\&RH}$ is the number of both indoor temperature points and relative humidity points within the upper and lower limits, respectively; and N is the total number of points.

2.5.3. Sensitivity Analysis

To evaluate the sensitivity of the DQN algorithm, we analyzed the sensitivity of the key parameters (i.e., learning rate α and discount factor γ) in Equation (1). First, the discount factor γ was fixed, and joint control effects on temperature and relative humidity based on the RL controller with different learning rates were compared. Subsequently, we fixed the learning rate α and compared joint control effects on temperature and relative humidity based on the RL controller with varying discount factors.

3. Case Study

3.1. Case Introduction

The building for the case study is a trade union activity room in an office building in the Haidian District, Beijing, with an area of 116 m². Its air-conditioning system comprises fan coil units with a fresh-air system. The geometry of the building was modeled using SketchUp software, as shown in Figure 5. A schematic diagram of the HVAC system operation is shown in Figure 6. The virtual simulation environment of the entire building’s HVAC system was built in TRNSYS software, as shown in Figure 7. The RL controller regulates the air supply volume for the fan coil units to improve the joint control satisfaction rate of indoor temperature and relative humidity. The thermodynamic parameters of the office building envelope are listed in

Table 5. Building envelopes.

Building Envelope			Heat Transfer Coefficient $\mathbf{W}/\left({\mathbf{m}}^2\mathbf{·}\mathbf{K}\right)$
External wall	Cement slag mortar	20 mm	2.266
	Steel reinforced concrete	370 mm
	Dali granite basalt	20 mm
Roof	Cement mortar	20 mm	0.804
	Cellular concrete	200 mm
	Steel reinforced concrete	130 mm
	Cement mortar	15 mm
External window	Glass	6 mm	1.46 (SHGC = 0.52)

. The setting for the building envelopes is based on actual engineering design drawings. The settings for the environmental thermal disturbances in the office are listed in

Table 6. Thermal disturbances.

Thermal Disturbance		Value
Human body heat generation		66 $\mathrm{W}/\mathrm{p}$
Occupant density		0.1 $\mathrm{p}/{\mathrm{m}}^2$
Human body moisture generation		0.109 $\mathrm{kg}/\left(\mathrm{h}·\mathrm{p}\right)$
Light and equipment heat generation		2.586 $\mathrm{W}/{\mathrm{m}}^2$
Occupancy	8:00–20:00	1
	Elsewise	0

. The other HVAC system settings are listed in

Table 7. Settings for the HVAC system.

Setting Items		Value
Fresh-air volume		10% of total air volume
Indoor environmental control objectives	Upper limit of indoor temperature	27 °C
	Lower limit of indoor temperature	25 °C
	Upper limit of indoor relative humidity	60%
	Lower limit of indoor relative humidity	40%
Air conditioning on or off	7:00–20:00	On
	Elsewise	Off
Simulation time step		12 min

. These settings refer to the Design Standards for Energy Efficiency of Public Buildings and on-site investigation. It should be noted that the air conditioner is set to be turned on one hour earlier than occupancy. This setting is to ensure that the indoor temperature is within the appropriate range when the staff enters the room, improving the thermal comfort of the staff.

3.2. Simulation Results Analysis of the Reinforcement Learning Controller

In this study, we selected 0:00 on July 1 to 0:00 on July 15 as the training period, and the stepwise average reward curve of the training process is shown in Figure 8.

As shown in Figure 8, the stepwise average reward climbs rapidly during the first 300 steps when the agent constantly interacts with the environment and learning experiences. After 300 steps, the agent initially completed learning and continued to interact with the environment and learn more experiences, and the stepwise average reward curve fluctuated within a small range.

The trained model was tested from 0:00 on 1 August to 0:00 on 31 August, and the simulation results during the test period were counted; the results are shown in Figure 9.

In this study, we selected indoor-air state on a typical day (5 August) for drawing, and the results are shown in Figure 10.

As shown in Figure 10, when the indoor temperature initially deviates from the comfort range and the relative humidity is outside the comfort range, the RL controller proposed in this study takes action to maintain the indoor temperature near the comfort range and to avoid further deviation of the indoor temperature from the comfort range. This ensures the normal operation of the HVAC equipment and avoids damaging the equipment. When the indoor temperature is within the comfortable range, the RL controller can select the air supply volume for the fan coil units to achieve better joint control satisfaction rate of indoor temperature and relative humidity.

3.3. Comparison and Analysis of Simulation Results for Different Controllers

To further verify control the effect of the reinforcement learning controller on the indoor temperature and relative humidity, we selected the on/off and rule-based controllers for simulation comparison. The simulation results of the indoor temperature and relative humidity in different controllers are shown in Figure 11 and Figure 12.

From the temperature distribution shown in Figure 11, the center of distribution of the indoor temperature is more biased toward 25 °C in the RL controller, rule-based controller, and on/off controller Ⅰ, while the center of distribution of indoor temperature is more concentrated at 26 °C in the on/off controller Ⅱ and on/off controller Ⅲ. From the distribution of relative humidity in Figure 11, the distribution of indoor relative humidity deviates to a higher relative humidity in all five controllers. By analyzing the weather data, it was found that there were more cloudy and rainy days during the test period, and the relative humidity of the outdoor atmosphere was higher during cloudy and rainy days, thus increasing the indoor relative humidity.

As shown in Figure 12, for the indoor temperature satisfaction rate, the on/off controller Ⅰ has the best control effect, which is 85.78%, and the rule-based controller has the worst effect, which is 70.17%. For the satisfaction rate of indoor relative humidity, the control effect of the RL controller is the best, at 54.78%, and the effect of the on/off controller Ⅲ is the worst, at 34.50%. For the joint control satisfaction rate of indoor temperature and relative humidity, the control effect of the RL controller is the best, at 48.94%, 9.5% higher than that of the rule-based controller, and 12.66% higher than that of the on/off controller Ⅰ.

3.4. Sensitivity Analysis

To evaluate the sensitivity of the DQN algorithm, the key parameters (i.e., learning rate α and discount factor γ) in Equation (1) were quantitatively analyzed in this study.

Maintaining the discount factor $\gamma =0.1$ constant, a result of comparing the joint control effect on indoor temperature and relative humidity by the RL controller at different learning rates is shown in Figure 13.

As shown in Figure 13, the joint control effect on indoor temperature and relative humidity by the proposed RL controller is relatively robust in the range of learning rate $\alpha \le 0.01$. When the learning rate $\alpha \ge 0.01$, the control effect of the controller is reduced and oscillation occurs because with an increase in the learning rate α, the training of the agent oscillates and converges with difficulty.

When the learning rate $\alpha =0.01$ is constant, the result of comparing the joint control effect on indoor temperature and relative humidity by the RL controller at different discount factors is shown in Figure 14.

As shown in Figure 14, the sensitivity of the proposed RL controller to the discount factor $\gamma$ is weaker than that of the learning rate $\alpha$. The overall control effect of the controller is more robust at different discount factors. However, for smaller discount factors $\gamma \le 0.5$, the controller has a better joint control effect on indoor temperature and relative humidity. This is because the input parameters of the DQN algorithm are the indoor temperature and relative humidity at the same time, and no outdoor weather parameters are introduced. If we aim to achieve a better control effect, we need the agent to prefer immediate rewards; that is, we need a relatively “short-sighted” agent, so the selection of the discount factor is more suitable for a smaller value.

4. Conclusions

In this study, an RL control method based on action intervention was proposed, and its input parameters, reward function, and agent exploration and exploitation mechanism were designed. Subsequently, this study considered fan coil units with a fresh-air system in an office building in Beijing as the research object and developed a TRNSYS–Python co-simulation platform to verify the control effect of the proposed method, and the following conclusions were obtained:

(1) Using file-based data transfer, this study developed a TRNSYS–Python co-simulation platform, which makes it easier to train the agent and test and evaluate the performance of comprehensive RL algorithms in a simulation environment.

(2) The DQN algorithm based on action intervention can reduce the training time computation cost in the training phase and increase the security of algorithm deployment in the testing phase. From the simulation results of this study, the algorithm can achieve a better joint control effect on indoor temperature and relative humidity in office buildings. Specifically, the method can improve the joint control satisfaction rate of indoor temperature and relative humidity by 9.5% and 12.66%, respectively, compared with the traditional rule-based controller and on/off controller Ⅰ.

(3) The setting of hyperparameters has a relatively significant impact on the control performance of the algorithm, which is robust when the hyperparameters are in an appropriate range. Otherwise, the control effect of the algorithm is reduced, and there is a risk of oscillation.

Therefore, the control method proposed in this study can achieve a better joint control effect on indoor temperature and relative humidity in office buildings. This study provides a new direction for indoor thermal comfort and environment control in office buildings, and has engineering application value.

Deep reinforcement learning for optimization control of an HVAC system is a complicated problem, and some limitations need to be improved and studied in the future. Firstly, the HVAC system and control action in this study are relatively simple, not involving heat and humidity transfer between multizones or excessive control actions. The stronger the coupling and nonlinear relationship between control actions of the HVAC system, the more potent the RL will be. Applying RL to complicated HVAC systems is challenging work. Furthermore, the RL controller proposed in this study is not deployed in an actual HVAC system. In the future, it will be significant to deploy the RL controller in an actual building and evaluate the practical control effect. Finally, the building in this case study is located in Beijing, China. It is desirable to test the control effect of the proposed RL controller on buildings in different climate zones.