A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems

Abstract

Energy optimization in buildings by controlling the Heating Ventilation and Air Conditioning (HVAC) system is being researched extensively. In this paper, a model-free actor-critic Reinforcement Learning (RL) controller is designed using a variant of artificial recurrent neural networks called Long-Short-Term Memory (LSTM) networks. Optimization of thermal comfort alongside energy consumption is the goal in tuning this RL controller. The test platform, our office space, is designed using SketchUp. Using OpenStudio, the HVAC system is installed in the office. The control schemes (ideal thermal comfort, a traditional control and the RL control) are implemented in MATLAB. Using the Building Control Virtual Test Bed (BCVTB), the control of the thermostat schedule during each sample time is implemented for the office in EnergyPlus alongside local weather data. Results from training and validation indicate that the RL controller improves thermal comfort by an average of 15% and energy efficiency by an average of 2.5% as compared to other strategies mentioned.

1. Introduction

1.1. Motivation and Background

According to the U.S. Energy Information Administration, the average energy consumed in the buildings sector for residential and commercial users accounts for 20.1% of global energy consumption worldwide. This energy demand is projected to increase by 1.5–2.1% annually between 2014 and 2040 [1]. In the United States, energy consumption from residential and commercial buildings was close to 41% of the total U.S. energy consumption in 2015, or around 39 quadrillion BTU [2]. Of this amount, the buildings’ Heating, Ventilation and Air Conditioning (HVAC) systems can account for up to 50% of total building energy demand [2,3]. In the hopes of moving toward a greener, more energy-efficient future, a significant improvement in energy efficiency is needed to achieve this goal. Control of building HVAC systems can lead us one step closer to that goal.

Despite the advances in research on HVAC control algorithms, most field equipment is controlled using classical methods that include hysteresis/on/off and Proportional Integral and Derivative (PID) controllers. These classical methods have been investigated extensively in HVAC lower-loop controls [4,5,6], such as room temperature control [7,8,9] and supply air temperature control [10,11,12]. Despite their popularity, these classical methods do not perform optimally. The high thermal inertia of buildings induces large time delays in the building dynamics, which cannot be handled efficiently by the simple on/off controllers. PID controllers offer an improvement in this respect. However, due to the high non-linearity in building dynamics coupled with uncertainties such as weather, energy pricing, etc., these PID controllers require extensive re-tuning or auto-tuning capabilities [13], which increases the difficulty and complexity of the control problem.

Due to these challenging aspects of HVAC control, various advanced control methods have been investigated, ranging from gain-scheduling [14], non-linear/robust model predictive control [15,16,17,18] to optimal control [19,20,21,22,23]. Between these methods, the quality of control performance relies heavily on accurate process identification and modelling. However, large variations exist with building design, zone layout, long-term dynamics and wide ranging operating conditions. In addition, large disturbance effects from external weather, occupancy schedule changes and varying energy prices make process identification a very challenging problem. This complexity requires one to have expert knowledge in both the underlying systems and controls.

In this work, we introduce novel control algorithms from a branch of machine learning called reinforcement learning. From a controls perspective, reinforcement learning algorithms can be considered as direct adaptive optimal control [24]. Like optimal control, reinforcement learning algorithms minimize the cumulative sum of costs over a time horizon. Unlike traditional optimal control, reinforcement learning algorithms can learn optimal control actions directly through trial-and-error interactions with the plant, without explicit identification of the plant model. By parametrizing the policy and value functions of reinforcement learning algorithms with artificial neural networks, the reinforcement learning controllers can learn to adapt to time-varying and non-linear system dynamics.

In our current approach, the impetus is thermostat control. Instead of traditional on/off heating/ cooling control, reinforcement learning is utilized to set this schedule to obtain improved Predicted Mean Vote (PMV)-based thermal comfort at an optimal energy expenditure. Hence, a thermostat schedule is computed using an RL controller.

1.2. Previous Work

The various advantages of reinforcement learning methods have prompted some research directions [25,26,27] in applying it to HVAC control. For example, Gregor. P et al. [28,29] investigated the use of Q-learning for optimizing thermal energy storage systems in a building. They developed a building simulation platform utilizing MATLAB and EnergyPlus and investigated the use of the tabular Q-learning algorithm. They found encouraging signs of the Q-learning controller learning a desirable load distribution strategy to minimize energy and cost. However, they were concerned with the sample efficiency of the method as compared to a Model-Predictive Controller (MPC). Artificial neural network-based Q-learning was also investigated, but they did not observe improved performance. However, it did improve the sample efficiency compared to the tabular case.

Despite the promising results seen in their work, there were several disadvantages in using the tabular Q-learning method for the HVAC system control problem. Firstly, the Q-learning algorithm requires that the control action space be discretized, and this loses the precise control advantage that a continuous control output algorithm can achieve. Secondly, since tabular Q-learning merely updates Q-value estimates within a finite-sized look-up table, the set of state space representation is limited by the size of the table and is again also discrete. This also reduces the algorithm’s ability to generalize to new, unseen state spaces. Thirdly, the Q-learning algorithm implemented assumed state observations to have Markov properties. However, this Markov assumption of full state observation is not applicable in the HVAC control problem, as state observations are both inherently noisy, as well as having high temporal correlations (for example, the indoor zone temperature and outdoor temperature). Lastly, the Q-learning algorithm is deterministic and cannot consider any inherent stochasticity introduced by the external variables, such as weather, occupancy and energy prices.

In our work, we show an improved algorithm and approach that attempt to alleviate all of the issues listed above. Specifically, we introduce two core novelties in our approach:

• The use of a model-free, actor-critic method that allows for a stochastic policy and continuous controller output. The use of a critic as a variance-reducing baseline also improves upon the sample efficiency.
• The use of state-of-the-art recurrent neural networks [30,31] for both policy and value representation, which allows for inferring hidden states on noisy observations and incorporating temporal information in state observations

For simulation, we utilize the Building Controls Virtual Test Bed (BCVTB) co-simulation, with the controller implementation in MATLAB and EnergyPlus for building simulation. We simulated an isolated single thermal zone with a simple Variable Air Volume (VAV) HVAC loop, consisting of heating/ cooling elements and air blower fans. Low level control is implemented internally within EnergyPlus.

In Section 2, we will provide an overview of the building co-simulation platform developed. In Section 3, we will provide an overview of reinforcement learning, as well as an overview of the algorithms utilized. In Section 4, we will provide an overview of recurrent neural networks and specifically the Long-Short-Term-Memory (LSTM) variant. In Section 5, we will provide a detailed overview of the experimental setup. Lastly, Section 6 present results and discussions on the experimental simulation runs utilizing the platform.

2. Platform Setup

Figure 1 shows a conceptual flow diagram of our building simulation and control platform.

Figure 2 shows the control platform developed. A custom Building Controls Virtual Test Bed (BCVTB) co-simulation allows for communication between building simulation in EnergyPlus and the controller module in MATLAB. An optimization variable module is added to receive raw data from EnergyPlus and computes the optimization cost. In the figure, blue bars are for splitting output vector streams into individual multi-line vector streams, which can then be plotted in the various plotters implemented.

The optimization variable module implemented in MATLAB 2016 uses the Predicted Mean Vote (PMV) [32], which is the thermal comfort measure, and the energy consumed during the sample time to calculate cost. Both PMV and energy values are provided by EnergyPlus to MATLAB via BCVTB.

Additionally, run-time plotters are included in the platform to allow real-time monitoring of the co-simulation ecosystem. The plotters visualize the optimization cost, EnergyPlus outputs and thermostat schedule, as well as controller output. This enabled easier troubleshooting of the platform during initial stages of implementation.

2.1. EnergyPlus

The building simulation is run in EnergyPlus 8.5, using an IDF format files. The process for modelling the building into an IDF starts with the detailed 3D modelling of the building or zone of interest. In this platform, the office space within the Donadeo Innovation Centre for Engineering (ICE) at the University of Alberta has been considered. The thermal zone with attributes representing neighbouring thermal zones is shown in Figure 3.

During the building design process, using OpenStudio 2.1.0, we also utilized SketchUp 2016 with OpenStudio to specify the space’s internal HVAC loop, an isolated single thermal zone with a simple Variable Air Volume (VAV) HVAC loop, consisting of heating/cooling elements, air blower fans and a generic thermostat. See Figure 4.

2.2. BCVTB

The BCVTB 1.60 co-simulation environment is a Java-based interface to allow for the exchange of data between MATLAB and EnergyPlus. Using the components specifying data flow, plotters, counters and timers, the handshake between EnergyPlus and MATLAB on a sample time basis is implemented. It also allows for initialization and time-out conditions to be specified for the experiment. It allows through the external variable interface in EnergyPlus both higher level (thermostat schedule control) and also access to low level control loops (actuator controls).

3. Deep Reinforcement Learning

3.1. Introduction

Reinforcement Learning (RL) is a computation approach for sequential decision making under uncertainty. Figure 5 shows the interaction between the agent and an environment: at each time step, the agent takes an action and receives a state observation and scalar reward signal from the environment, which is unknown. An RL algorithm tries to maximize the agent’s total accumulated reward, given a previously unknown environment, through a trial-and-error learning process.

The description of the general reinforcement learning setup is inherently very general and thus can be applied to a wide range of applications, ranging from robotics [33], game playing [34] to inventory management [35].

3.2. Markov Decision Processes and Value Functions

The Markov Decision Process (MDP) mathematically formalizes the agent-environment interactions. It has the following core components:

• $s\in \mathbb{S}$: state space, the set of all possible states of the environment.
• $a\in \mathbb{A}$: action space, the set of all possible actions, from which the agent selects one at every time step.
• $r\in \mathbb{R}$: reward, a scalar reward emitted by the environment at every time step.
• $\pi (a\mid s)$: policy, which maps from state observation s to action a. Typically, stochastic policies are used; however, deterministic policies can also be specified.
• $\mathbb{P}(s^{\prime },r|s,a)$: transition probability distribution, which specifies the one-step dynamics of the environment, the probability that the environment will emit reward r and transition to subsequent state $s^{\prime }$ from being in state s and having taken action a.

The goal of the RL agent is to learn an optimal policy that maximizes the expected sum of rewards, also called returns, $G_t$, shown in Equation (1).

$$G_t\doteq r_{t+1}+r_{t+2}+r_{t+3}+\dots +r_{\infty }$$

(1)

In general, it is common to apply a discounting factor to the returns, called the discount rate $\gamma$, $0\le \gamma \le 1$. It acts to determine the present value of the rewards. The smaller the $\gamma$, the less future rewards matter, and vice versa. In addition, it acts to threshold the returns to a finite, bounded value. With discounting, the total returns are shown in Equation (2).

$$G_t\doteq r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\dots =\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

(2)

Value functions are important parts of many RL algorithms. In essence, they estimate the expected total returns starting from some state. Two core value functions are present in RL, the first one being the state-value function, $v^{\pi }\left(s\right)$. This function Equation (3) estimates the expected total returns $G_t$, starting from a certain state s, and following policy $\pi$.

$$.v^{\pi }\left(s\right)\doteq {\mathbb{E}}_{\pi }\left[G_t\mid s\right]={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}\mid s\right]$$

(3)

The other value function of interest is called the action-value function, $q^{\pi }(s,a)$ Equation (4). Similar to the state-value function above, the action-value function estimates the expected total returns $G_t$ starting from some state s and taking an action a following some policy $\pi$.

$$q^{\pi }(s,a)\doteq {\mathbb{E}}_{\pi }\left[G_t|s,a\right]={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}|s,a\right]$$

(4)

Many popular RL algorithms rely heavily on the above value functions, which are estimated from experience. Their usefulness lies in that once the optimal value functions, $v_{\pi }^{*}$ or $q_{\pi }^{*}$, have been obtained from sufficient experience, the optimal policy, ${\pi }^{*}$, can be found by repeatedly applying the value function for every state and taking the greedy action that leads to the state with the highest value estimate. One example of such an algorithm is Q-learning, which utilizes a parameterized action-value function $q_{\pi }(s,a)$ to learn the optimal action-value function.

3.3. Policy Gradient

One class of reinforcement learning methods deals exclusively within the parameterized policy space. Specifically, a differentiable parameterized model, such as a neural network model with parameters $\theta \in {\mathbb{R}}^d$, is used to represent a stochastic policy. This direct policy parameterization allows us to convert the RL problem into that of an optimization problem; namely, maximizing the objective function $J\left(\theta \right)$, the total returns, under the policy ${\pi }_{\theta }$ given in Equation (5).

$$\underset{\theta }{max}J\left(\theta \right)=\mathbb{E}\left(G_t|{\pi }_{\theta }\right)$$

(5)

The policy gradient method offers several advantages over value-based methods, improving in areas such as stability, direct optimization, etc. There exist multiple ways to estimate the policy gradient, from finite difference methods [36] to likelihood ratio methods [37]. We focus on the latter method, specifically the REINFORCE policy gradient [38]. The likelihood ratio method’s advantages are faster convergence rates, and this nullified the need for generating random perturbations in policy parameters, which can lead to instability.

The REINFORCE [38] policy gradient provides an unbiased estimate of the gradient of the policy with respect to the objective function $J\left(\theta \right)$ given in Equation (6).

$${▽}_{\theta }J\left(\theta \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{T-1}{▽}_{\theta }log\pi \left(a_t\right|s_t,\theta )\sum _{t^{\prime }=t}^{T-1}{r}_{t^{\prime }}\right]$$

(6)

One point with the REINFORCE policy gradient is that it has high variance, as it is using the actual returns $r_{t^{\prime }}$ as a component of the gradient estimate. It can be shown that subtracting the returns $r_{t^{\prime }}$ by an arbitrary baseline term, $b\left(s_t\right)\in \mathbb{R}$, can significantly reduce the variance in gradient estimates [39]. It can also be shown that the baseline introduces no bias in the gradient estimate [39], i.e., ${\mathbb{E}}_{\pi }[{▽}_{\theta }log\left(a_{t^{\prime }}\right|s_{t^{\prime }},\theta )b\left(s_t\right)]=0$. Thus, by choosing our baseline carefully, one can significantly reduce the variance and, thus, the sample complexity in our gradient estimates. Thus, the general policy gradient estimate with an arbitrary baseline function is provided in Equation (7).

$${▽}_{\theta }J\left(\theta \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{T-1}{▽}_{\theta }log\pi \left(a_t\right|s_t,\theta )\left(\sum _{t^{\prime }=t}^{T-1}{r}_{t^{\prime }}-b\left(s_t\right)\right)\right]$$

(7)

3.4. Actor-Critic Methods

Actor-critic methods are based mainly on the policy gradient with variance-reducing baseline derivation obtained in Equation (7). A more general representation is to call the term on the right-hand side of Equation (7) the advantage estimator, $A_t$. The various different actor-critic methods thus differ in how they approximate the advantage estimator. Thus, in actor-critic methods, the actor term refers to a parameterized policy, and the critic term refers to some parameterized advantage estimate.

In our work, we introduce a Monte Carlo actor-critic algorithm that utilizes a state-value function, $v^{\pi }\left(s\right)$, as the critic. The policy gradient estimate is shown in Equation (8).

$${▽}_{\theta }J\left(\theta \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{T-1}{▽}_{\theta }log\pi \left(a_t\right|s_t,\theta )\left(\sum _{t^{\prime }=t}^{T-1}{r}_{t^{\prime }}-v^{\pi }\left(s_t\right)\right)\right]$$

(8)

The algorithm is updated off-line using the collection of experiences obtained on-line with fixed parameters. Due to the inherent noise and high sequential nature of the HVAC thermostat-schedule problem, we opt to utilize recurrent neural networks for parametrizing both the policy and critic. We utilize recurrent neural networks termed Long-Short-Term-Memory networks to address the issue of noise and the partial observability of states. Since LSTMs maintain a hidden state during computation, this hidden state is able to incorporate historical data to infer the true state of the system, from observing a history of noisy, the partially-observed state observation data.

The pseudo-code for our algorithm is given below:

Algorithm 1 Monte Carlo on policy actor-critic.

Require: Initialize policy $\pi$ with parameters ${\theta }_{\pi }$ and value critic $v_{\pi }$ with parameters ${\theta }_v$ 1: for each episode do 2:       Get initial state s 3:       Initialize storage buffer $S,A,R,S^{\prime }$ 4:       for $i=1,2,3...N$ steps do 5:             Sample action with policy: $a\sim {\pi }_{\theta }\left(s\right)$ 6:             Run action through environment, obtain reward and post state: $r,s^{\prime }\leftarrow ENV(s,a)$ 7:             Collect and store: $S,A,R,S^{\prime }\leftarrow s,a,r,s^{\prime }$ 8:             $s\leftarrow s^{\prime }$ 9:       end for 10:       Compute discount returns: $\widehat{V}={\sum }_{l=0}^{N-1}{\gamma }^l{r}_{t+l}$ 11:       Update ${\theta }_v$ to minimize ${\sum }_{n=1}^N{\parallel v_{\pi }\left(s_n\right)-{\widehat{V}}_n\parallel }^2$ 12:       With learning rate $\alpha$, update policy: ${\theta }_{\pi }\leftarrow {\theta }_{\pi }+\alpha {\nabla }_{\theta }log\pi \left(A\right|S)v_{\pi }\left(S\right)$ 13: end for

4. Recurrent Neural Networks

Artificial neural networks, specifically the recent branch termed as deep learning, have gained an unprecedented revived interest and attention in the past decade from both academia and industry. They have achieved the state-of-the-art in many fields, ranging from computer vision [40], natural language processing [41] to niche applications, such as self-driving cars [42]. For datasets with a sequential nature, such as for natural language processing, Recurrent Neural Networks (RNN) have been applied successfully due to their capacity to model highly non-linear data, increasing the availability of datasets and power computers. Despite traditional recurrent neural network’s inability to handle long sequences of data, a new class of network architectures with learnable gates has been shown to effectively alleviate this problem. The most popular of these variant is the long-short-term-memory network architecture.

4.1. Vanilla Recurrent Neural Network

Artificial neural networks are a class of non-linear function approximations. They are composed of nested affine transforms followed by a non-linear activation transform. Given an input $\mathit{x}$ and layer input weight matrix ${\mathit{W}}_{\mathit{in}}$, we first apply an affine transform and then compute $g\left(x\right)$, a non-linear activation transform, over the affine transform output. The computation is shown in Equation (9).

$$f(\mathit{x};\mathit{w},b)=g({\mathit{W}}_{\mathit{in}}\mathit{x}+\mathit{b})$$

(9)

Since RNN works on sequential data, it has an explicit hidden state, $h_t$, that is recursively carried forward and updated as each element of the sequential data is passed through the recurrent neural network. To facilitate this, an additional recurrent weight, ${\mathit{W}}_{\mathit{rec}}$, is need for this computation step. For a given sequential data from $t=1$ to $t=T$, the update Equation (10) is applied for a hidden layer recurrent neural network.

$$\begin{array}{c}\hfill {\mathit{a}}_t=\mathit{W}{\mathit{x}}_t+\mathit{U}{\mathit{h}}_{t-1}+\mathit{b}\end{array}$$

(10)

$$\begin{array}{c}\hfill {\mathit{h}}_t=g\left({\mathit{a}}_t\right)\end{array}$$

(11)

Two points of importance are noted from the computation steps above:

• The same layer weights $\mathit{W}$ and $\mathit{U}$ are re-used for each step of computation throughout the sequential data
• An RNN is similar to a deep feedforward neural network when unrolled through time. The depth will be of the length of the sequential data, and for each layer, the weights are the same

4.2. Vanishing and Exploding Gradient Problem

The back-propagation of error signals through the RNN is similar to that of a feed-forward neural network. Starting from the error output, we go backwards and apply the chain rule to compute the gradient of each layer’s parameters using gradients coming from the next layer. For RNN, this is called Back-Propagation Through Time (BPTT). Due to the fact that the same layer weights are re-used for each item in the sequential data, when the computation is unrolled through time, the network can be viewed as a very deep feed-forward network in the time dimension, with shared weights at all layers. Figure 6 shows a diagram of an unrolled RNN over three computation steps.

The BPTT algorithm computes the gradient of the total loss, $\mathcal{E}$ across the entire sequence. Since the RNN’s parameters are re-used for each element in the input sequential data, the total loss gradient is the sum of the loss gradient for each sequence of time, given in Equation (12).

$$\frac{\partial \mathcal{E}}{\partial \theta }=\sum _{1\le t\le T}\frac{\partial {\mathcal{E}}_t}{\partial \theta }$$

(12)

We can further expand the right-hand side to include two gradient components, first the gradient from the loss at each time step, as well as the recurrent gradient flow back, provided in Equation (13).

$$\frac{\partial {\mathcal{E}}_t}{\partial \theta }=\sum _{1\le t\le T}\left(\frac{\partial {\mathcal{E}}_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial \theta }\right)$$

(13)

In particular, when we focus on the temporal gradient flow-back component, $\frac{\partial h_t}{\partial h_k}$, we can see that it is a product of the same gradient starting from time t and extending until the end of the sequence k, as shown in Equation (14).

$$\frac{\partial h_t}{h_k}=\prod _{t\ge i>k}\frac{\partial h_i}{\partial h_{i-1}}=\prod _{t\ge i>k}{\mathit{U}}^T\mathit{diag}\left(g^{\prime }\left(h_{i-1}\right)\right)$$

(14)

4.3. Long-Short-Term-Memory Recurrent Neural Network

One popular method to alleviate the above-mentioned vanishing gradient problem is LSTM [30,31] networks. LSTM is a special variant of recurrent neural networks that improves upon the traditional RNN in the following ways:

• Learnable gates that modulate the flow of information.
• A persistent cell state that has minimal interactions, providing an easy path for gradient flow during back-propagation.

A total of three gates is utilized within an LSTM; these are the input gate $i_t$, Equation (16), forget gate $f_t$, Equation (15) and output gate, $o_t$ Equation (17). To allow for the gates to modulate the flow of information, as well as be differentiable, sigmoid/logistic function $\sigma$ is used to threshold the gate outputs to between [0,1].

$$\begin{array}{c}\hfill f_t=\sigma (W_f{x}_t+U_f{h}_{t-1}+b_f)\end{array}$$

(15)

$$\begin{array}{c}\hfill i_t=\sigma (W_i{x}_t+U_i{h}_{t-1}+b_i)\end{array}$$

(16)

$$\begin{array}{c}\hfill o_t=\sigma (W_o{x}_t+U_o{h}_{t-1}+b_o)\end{array}$$

(17)

$$\begin{array}{c}\hfill c_t=f_t\odot c_{t-1}+i_t\odot tanh(W_c{x}_t+U_c{h}_{t-1}+b_c)\end{array}$$

(18)

$$\begin{array}{c}\hfill h_t=o_t\odot \sigma \left(c_t\right)\end{array}$$

(19)

Once the gates are computed, the cell state $c_t$ is updated by Equation (18) in two parts. The first component is a dot product between the previous cell state $c_{t-1}$ and the forget gate $f_t$. This modulates how much of the cell state information from the previous time step should be kept. The second part is a dot product between the current cell state computation and the input gate $i_t$. This modulates how much of the current information should be allowed into $c_t$. Once the cell state has been updated, the hidden output $h_t$ is modulated by the output gate $o_t$ via another dot product Equation (19). Figure 7 provides a visualization of the LSTM equations.

5. Setup

For comparative study, we implemented an actor-critic-based RL controller applied to the building HVAC control and compared its performance against three baselines.

5.1. Simulation Setup and Parameters

We utilized our simulation platform setup for the experimental case study. The simulation was conducted over a period of seven days, two days for training the RL controller and five days for validation testing. We utilized our simulation platform setup for the experimental case study. The simulation was conducted over a period of seven days, two days for training the RL controller and five days for validation testing. We selected this simulation window to ensure no large variations (example: $\pm 10\%$ of $26.53{}^{\circ }$C for ambient temperature) in external variables, such as ambient temperature, cloud cover and solar irradiance, between the training set and validation set. A single room zone (Zone 1) with window access is investigated in our simulation. This single zone has thermal contributions from surrounding ideal air zones (Zones 2–5). The simulation sampling time is 5 min. The occupancy considers three people in an office setting, measured as a percentage.

Table 1. This table provides the daily occupancy schedule for a typical work day on an hourly-basis. Values reported in fractions. This occupancy schedule was used in our simulation.

Hours	Office Occupancy	Equipment Active Schedule
01:00 a.m.	0	0
02:00 a.m.	0	0
03:00 a.m.	0	0
04:00 a.m.	0	0
05:00 a.m.	0	0
06:00 a.m.	0	0
07:00 a.m.	0.5	1
08:00 a.m.	1.0	1
09:00 a.m.	1.0	1
10:00 a.m.	1.0	1
11:00 a.m.	1.0	1
12:00 p.m.	0.5	1
01:00 p.m.	1.0	1
02:00 p.m.	1.0	1
03:00 p.m.	1.0	1
04:00 p.m.	1.0	1
05:00 p.m.	0.5	1
06:00 p.m.	0.1	1
07:00 p.m.	0	0
08:00 p.m.	0	0
09:00 p.m.	0	0
10:00 p.m.	0	0
11:00 p.m.	0	0
12:00 a.m.	0	0

provides detailed information.

For training of the reinforcement learning controller, we utilized two days of simulation data to train the policy and critic LSTM networks in our RL controller. After the training, we tested the performance of the controller against benchmarks in the five-day simulation.

5.2. Reinforcement Learning Controller Design

The reinforcement learning controller is composed of two LSTM networks, one for the actor and one for the critic.

The state observations, which are provided at every control time step to the RL agent, are chosen to be both informative and easy to access. Specifically, the state observations are composed of the outdoor temperature $T_a$ (${}^{\circ }$C), indoor temperature $T_r$ (${}^{\circ }$C), the energy demand from the last time step $Q_{cool}$ (W/h) and solar irradiance $Q_{sol}$ (W/m${}^2$). The controller outputs over a discrete control action set of size 26, from 20 ${}^{\circ }$C to 25 ${}^{\circ }$C with 0.2 ${}^{\circ }$C resolution. This control output is then interpreted as the room cooling temperature set point $T_c$, which will be passed onto the simulation platform.

Figure 8 shows the RL controller architecture setup. We see that the discrete control action output is computed by the actor, given the state observation. The critic, a state-value function $v^{\pi }$, computes a single scalar value representing the state-value of a particular state. This scalar value is used as the baseline in the critic’s policy gradient update.

5.3. Optimization Cost Function Structure

The optimization cost function (also referred to as the reward function) design is of great importance in the reinforcement learning problem setup, as an ill-defined cost function can lead to unexpected and undesirable controller performance.

In our study, we would like the controller to achieve thermal comfort while maintaining a certain level of energy efficiency. It is hoped that by relaxing the thermal comfort threshold ($\tau$), the RL controller can achieve energy savings while still ensuring a tolerable level of thermal comfort. To obtain a numerical value for thermal comfort, the Predicted Mean Vote (PMV) developed by P.O.Fanger [32] was used. The PMV calculation takes many factors into account, ranging from air temperature, relative humidity, clothing insulation and metabolic rate. The PMV calculation represents the thermal comfort within a scale from hot[+3] to cold[−3]. Usually, thermal comfort values of between ±0.5 and ±1.0 are considered acceptable. In our setup, the simulation platform computes the PMV values internally in EnergyPlus.

The total cost, C, which is computed at every control step, is composed of a weighted sum of the two objectives, PMV (thermal comfort) and energy demand, given in Equation (20).

$$C=\sum _{t=0}^{t=T}(\alpha ·PM{V}_{cost}+(1-\alpha )·Q_{cool})$$

(20)

The $\alpha$ in the equation is a weighting that balances the contribution from the two component costs, PMV cost ($PM{V}_{cost}$) in Equation (21) and energy ($Q_{cool}$). In our experiment, we set $\alpha$ at 0.5. In addition, $PM{V}_{cost}$ and $Q_{cool}$ are normalized to be between [0,1].

$$PM{V}_{cost}=\left\{\begin{array}{ccc}\hfill 0& \mathrm{for}& \left|PMV\right|\le \tau \hfill \\ \hfill \frac{\left|PMV\right|}{3}& \mathrm{for}& \tau \le \left|PMV\right|\le 3\hfill \\ \hfill \left|PMV\right|-\tau & \mathrm{for}& 3<\left|PMV\right|\hfill \end{array}\right.$$

(21)

The $PM{V}_{cost}$ is computed to have zero effect on the total cost if it is below the PMV threshold, $\tau$. Note that since PMV values typically range between $\pm 5$, we take the PMV’s absolute value when computing its cost contribution.

The energy demand $Q_{cool}$ is obtained raw from the our simulation tool and is also normalized.

6. Results and Discussion

6.1. Baseline Setup

For comprehensive comparison, two different control baselines were setup to evaluate the effectiveness of the reinforcement learning controller. These baselines vary in the various aspects of the controller performance.

The first baseline is called the ideal PMV baseline, $B_{ideal}$. This baseline maintains the ideal PMV without regard for the energy usage required to maintain this precise PMV requirement. This is given in Equation (22).

$$B_{ideal}=\left\{\begin{array}{c}\hfill T_h=21\\ \hfill T_c=23\end{array}\right.$$

(22)

where $T_c$ is the cooling set point (${}^{\circ }$C) and $T_h$ is the heating set point (${}^{\circ }$C).

Secondly, a variable control $B_{variable}$ was implemented. This baseline contains some basic temperature-tracking capabilities. Specifically, it tracks the external temperature and sets $T_c$ according to the following equation if it is within 20 and 25 ${}^{\circ }$C. Since the simulation runs are conducted in the summer month of July, room heating is never required as external temperatures range well above 20 ${}^{\circ }$C. Due to this factor, heating set point $T_h$ is set constant at 16 ${}^{\circ }$C for our baseline setup. Note that $T_h$ can be set up in a similar fashion to $T_c$ for simulation through colder months when room heating is needed.

$$B_{variable}=\left\{\begin{array}{c}\hfill T_h=16\\ \hfill T_c=20& \mathrm{for}& T_r<20\hfill \\ \hfill T_c=T_r& \mathrm{for}& 20\le T_r\le 25\hfill \\ \hfill T_c=25& \mathrm{for}& T_r>25\hfill \end{array}\right.$$

(23)

where $T_r$ is the room temperature (${}^{\circ }$C).

6.2. RL Training Phase

The RL controller was trained using two days of simulation data with a sampling time of 5 min. Figure 9 shows the RL controller’s trajectories across each iteration of training. It can be observed that initially, the controller’s output contains large variation, as clearly observed in the cooling rate $Q_{cool}$. With each training iteration, the controller outputs stabilize, until they stabilize to the optimal controller output with low overall cost.

Table 2. RL training results.

Iteration Number	PMV Total (Unitless)	Energy Total (W)	Standard Deviation (${\mathit{Q}}_{\mathit{c}\mathit{o}\mathit{o}\mathit{l}}$ W/h)
1	216.70	16,717	3580
2	113.50	18,079	3480
3	105.60	18,251	3500
4	262.00	15,398	3070
5	132.12	17,698	3450
6	211.30	16,300	3240
7	211.50	16,310	3240
8	211.30	16,298	3228

provides the metrics associated with the tuning exercise. As described in the optimization cost function in Equation (20), the metrics involving both thermal comfort Equation (21) and energy consumption are provided. With a blind start initially, the iterations continue to improve both measures, limited by the inertia of the building profile. On obtaining a negligible gradient in metrics between consecutive iterations, the procedure is halted.

Between the iterations, the aggression of the RL controller is tuned based on PMV and energy consumption metrics. This is evident from the steady decline in the standard deviation measure of cooling rate $Q_{cool}$, observed in Table 2. However, owing to the stochastic nature in which the neural networks are trained, there is a possible situation where the minimization goal is not met. In such a scenario, as observed in

Table 3. Training comparison.

Control Type	PMV (Unitless)	Energy Total (W)
Ideal PMV	190.00	16,676
Variable Control	250.00	15,467
RL Control	211.3	16,298

, Row 4, though an improved energy efficiency is evident, it greatly increases the PMV measure, indicating very poor thermal comfort. In such a scenario, the anomaly is discarded, and the next iteration is initialized with the previous valid iteration’s outcome.

A benchmarking of the performance of controllers under uniform environmental conditions is carried out. The experiments are performed using the $B_{ideal}$ and $B_{variable}$ control structures.

$B_{ideal}$ is a control that can offer an improved thermal comfort resulting from higher energy consumption. With the threshold provided in Equation (21), ($\tau$ = 0.75), the average value of the PMV, calculated in Equation (24), for the $B_{ideal}$ case (0.66) can be relaxed further for improving energy efficiency without sacrificing the thermal comfort. A visualization is provided in Figure 10.

$$\tau =\frac{PM{V}_{Total}}{No.ofDay\times No.ofSamples/day}$$

(24)

This measure for the $B_{variable}$ case is 0.87. However, the RL controller achieves an average PMV of 0.73. This also improves the energy utilization by 2.27.

The iterative tuning of the RL controller is performed on identical datasets. In order to evaluate the effectiveness of the RL controller, a five-day window, with weather data pertaining to five contiguous days, is utilized. These five days are not contiguous with respect to the training days; however, they are from the same season as the training data (summer), with slow-varying cloud cover and ambient temperature. The weather pattern provides cloud cover on Days 4 and 5, as observed in Figure 11 from the low solar irradiance and ambient temperature conditions. It is observed that the RL controller positions the cooling setpoint ($T_c$) slightly above the $B_{ideal}$ controller. As observed from the cooling rate in Figure 11, it is evident that this enables a slightly lower cooling of the office thermal zone while leading to improved energy efficiency at the cost of acceptable thermal comfort loss. The average thermal comfort of the $B_{ideal}$ is 0.708 and that of variable control is 0.917. The RL controller has an average thermal comfort of 0.775, improving the thermal comfort by 15.5% over the $B_{variable}$ and driving down energy by 5.03% as compared to the $B_{ideal}$ controller.

Table 4. Validation comparison over 5 simulation days.

Control Type	PMV Total (Unitless)	Energy Total (W)
Ideal PMV	510	42,094
Variable Control	660	38,536
RL Control	558	39,978

shows the numerical comparison between the RL controller and the two baselines. Due to the large variation in weather across the five-day validation phase, we can observe a more pronounced improvement from the RL controller. Compared to the ideal PMV baseline, the RL controller was able to achieve 2.12 kW of energy rate savings. Compared to the variable control baseline, the RL controller was able to maintain a 102 lower PMV total, which consumes only 1.44 kW more power. Similar to what is observed with the results from the training phase, the RL controller is able to sit at the sweet spot, where some slight thermal comfort is sacrificed for improvements in overall less HVAC power consumption.

7. Conclusions

A thermal zone of the office space has been designed. A platform for closed-loop simulation of HVAC systems has been created. Based on the influence of weather data on the building dynamics, uncertainties were introduced into the building model. A model-free reinforcement learning-based thermostat schedule controller has been developed using the novel long-short-term memory recurrent neural network, by closed-loop control of the HVAC system for two days. Using the simulation platform, we experiment on the $B_{ideal}$ and $B_{variable}$ strategies. A five-day validation experiment is carried out with weather data from outside the training window. It is observed that during both training and validation, the RL controller was able to maintain the average PMV within an admissible range while saving energy compared to the $B_{ideal}$ case. It provides the possibility to implement customized control for building HVAC control with minimal human intervention. Using the definition of the optimization cost, it will be possible to manipulate the goals of the RL controller towards improving the energy efficiency or thermal comfort.

To expand further upon the current simulation setup of a single zone, future research will be on developing RL thermostat schedule control across multiple non-ideal zones interacting with each other. In addition, integration with real building management systems (via Modbus or Lontalk) on the real-time building thermostat schedule will also be investigated.