Economic Dispatch for Smart Buildings with Load Demand of High Volatility Based on Quasi-Quadratic Online Adaptive Dynamic Programming

Abstract

In this paper, a quasi-quadratic online adaptive dynamic programming (QOADP) algorithm is proposed to realize optimal economic dispatch for smart buildings. Load demand of high volatility is considered, which is modeled by an uncontrollable state. To reduce residual errors of the approximation structure, a quasi-quadratic-form parametric structure was designed elaborately with a bias term to counteract effects of uncertainties. Based on action-dependent heuristic dynamic programming (ADHDP), an implementation of the QOADP algorithm is presented that involved obtaining optimal economic dispatch for smart buildings. Finally, hardware-in-loop (HIL) experiments were conducted, and the performance of the proposed QOADP algorithm is superior to that of two other typical algorithms.

1. Introduction

The world is seeing tremendous growth in independent smart buildings due to the development of generators and battery techniques. When distributed generators and storage equipment are connected to smart buildings, there are new challenges for traditional energy management mechanisms due to uncertainties of user behaviors. These uncertainties are likely to result in a load demand sequence with high volatility. Therefore, an intelligent energy management system with the ability to confront with the uncertainties is necessary for smart buildings to realize optimal and economic dispatch.

1.1. Literature Review

Optimal economic dispatch is generally formulated as an optimization problem [1,2]. Therefore, traditional optimization techniques are widely adopted to obtain optimal economic dispatch solutions. The convex optimization algorithm is a fundamental and well-developed method for dispatch tasks. In [3], the authors developed a Lyaponov-based optimization method for a transformed problem of the original non-convex problem. In [4], a robust convex optimization algorithm was designed for single-phase or balanced three-phase microgrids with adverse conditions of random demand and renewable energy resources. In [5], a chance-constraint-based stochastic optimization technique was applied to design an optimal energy scheduling policy for multi-service battery energy storage. However, it is still very difficult for some nonlinear and strongly coupled problems to be transformed into a convex formulation. Motivated by the development mechanism in nature, heuristic algorithms were developed to search for optimal solutions in the feasible domain. In [6], the authors developed a particle swarm optimization-based algorithm to form a reliable hybrid microgrid with optimal economic dispatch for non-sensitive loads and energy storage systems. In [7], a genetic-algorithm-based, online, real-time optimized energy management strategy was proposed for plug-in hybrid electric vehicles. Most of the literature above focused on static optimization problems whose solutions were in a space with finite dimensions. For optimization problems of dynamical systems, dynamic programming is a promising technique, since the optimality principle simplifies the numerical computation. In [8], the authors developed a two-scale dynamic-programming-based optimal energy management method for wind–battery-power systems. A multidimensional dynamic-programming-based energy control policy was proposed for global composite operating cost minimization in [9]. However, dynamic programming suffers from the so-called “curse of dimensionality” as the number of time steps increases. The solution space evolves with a exponential rate, which hinders the application of dynamic programming.

To avoid the “curse of dimensionality”, adaptive dynamic programming (ADP) was developed to solve the optimal economic dispatch problem with unknown system dynamics [10]. One can refer to [11,12] for the pioneering work and an overview of ADP. Due to its self-learning ability and adaptivity, it is popular in optimal economic dispatch problems of smart buildings. An optimal battery-control policy was obtained by designing a time-based ADP algorithm for residential buildings in [13]. In [14], the authors designed action-dependent, heuristic dynamic programming-based algorithms for optimal economic dispatch of multiple buildings. In [15], the authors proposed a temporal difference learning-based ADP algorithm for fast computation. In a recent work [16], a novel, model-free, real-time RT-ADP algorithm was verified in a hardware-in-the-loop experimental platform. However, models studied in the above literature are deterministic. In engineering practice, uncertainties are inevitable in most circumstances. Therefore, ADP was developed for optimal economic dispatch of smart buildings with uncertainties in the past few years [17,18,19,20]. To make the ADP algorithms more convincing, stability analysis was provided for ADP algorithms of energy systems in [21,22,23,24,25]. In [21], the authors proposed a dual, iterative Q-learning algorithm to solve the optimal battery-control law for a smart microgrid. Extensions of [21] were created in [23] for cases with renewable generation and in [22] for cases with actuator constraints. For application in appliance-level economic dispatch, a data-based ADP algorithm was studied for ice-storage air conditioning systems in [24]. The above works commonly assumed that the load demand of users is periodic. However, the load demand of users is random with high volatility in practice; thus, the periodic assumption cannot be satisfied in most real cases.

In the implementation of an ADP algorithm, the design of a function approximator shows significant effects on the performance of the overall algorithm. The feedforward neural network is the most commonly used as a function approximator. Leaving aside feedforward neural networks, the authors of [26] employed an echo-state networks as a parametric structure. The authors of [27] approximated the iterative value function by a fuzzy structure. However, these general-function approximators were employed without considering features of the considered systems, which can result in large residual errors.

1.2. Motivation and Contributions

According to the above discussion, the existing algorithms are mainly concerned with optimal economic dispatch for smart buildings with periodic load demands. In practice, load demands are not periodic due to uncertain user behaviors, and even with high volatility, which inspires the study of our work. In this paper, a QOADP algorithm is proposed to obtain the optimal economic dispatch for smart buildings with high volatility load demand. A quasi-quadratic form parametric structure is designed elaborately to approximate the value function precisely. Compared with the existing literature, the main contributions of this paper are listed as follows:

• A kind of online ADP is proposed to iteratively obtain optimal economic dispatch for smart buildings with high volatility of load demand. The online algorithm allows parameters of controller to achieve optimal control with the changing of load demand.
• A quasi-quadratic form parametric structure is designed for the implementation of QOADP with a bias term to counteract the effects of uncertainties. To simplify the function approximation structure in the proposed algorithm, the feedforward information of the uncontrollable state is taken into account in the iterative value function and the iterative controller.

The rest of this paper is organized as follows. In Section 2, the mathematical model for the EMS is established. In Section 3, the QOADP is proposed for the EMS, and the optimality is analyzed. Then, parametric structures are designed for implementation. In Section 4, the data-driven approach is proposed. Section 5 provides hardware-in-loop (HIL) experimental results of the proposed method, verifying the effectiveness of QOADP. In Section 6, conclusions are drawn.

2. EMS of Smart Buildings

A smart building is composed of an utility grid; an energy storage system, which is connected to the point of common coupling (PCC) with the power electronic converter; and the building load. The diagram of the EMS of a smart building is shown in Figure 1.

Power balance at the PCC is

$$P_{\mathrm{G}}\left(t\right)=P_{\mathrm{L}}\left(t\right)-P_{\mathrm{B}}\left(t\right)$$

(1)

The battery model is

$$\frac{d\left[SOC\right(t\left)\right]}{dt}=\left\{\begin{array}{c}\hfill \frac{\eta }{E_{\mathrm{B}},\mathrm{rate}}{P}_{\mathrm{B}}\left(t\right),P_{\mathrm{B}}\left(t\right)\ge 0\\ \hfill \frac{1}{\eta E_{\mathrm{B}},\mathrm{rate}}{P}_{\mathrm{B}}\left(t\right),P_{\mathrm{B}}\left(t\right)<0\end{array}\right.$$

(2)

where $P_{\mathrm{B}}\left(t\right)\ge 0$ denotes battery charging and $P_{\mathrm{B}}\left(t\right)<0$ denotes battery discharging. In reality, dispatch of the storage system must satisfy the following constraints:

$$SO{C}_{\min }\le SOC\left(t\right)\le SO{C}_{\max }$$

(3)

$$|P_{\mathrm{B}}\left(t\right)|\le P_{\mathrm{B},\mathrm{rate}}$$

(4)

Constraint (3) makes sure the battery energy is bounded within an allowable range. Constraint (4) limits the charging/discharging power of the battery.

The transition function of the load demand in continuous form can be defined as

$${\dot{P}}_{\mathrm{L}}\left(t\right)=g\left(P_{\mathrm{L}}\left(t\right)\right)$$

(5)

The optimal energy management problem is formulated as a discrete-time control problem with $\tau =1$ h, which is consistent with the real-time price (RTP). Using the Euler method with $\tau =1$ h leads to

$$\begin{array}{cc}\hfill SOC(k\tau +\tau )& =SOC\left(k\tau \right)+(sgn\left(P_{\mathrm{B}}\left(k\tau \right)\right)·\frac{\eta }{E_{\mathrm{B}},\mathrm{rate}}\hfill \\ \hfill & +sgn\left(P_{\mathrm{B}}\left(k\tau \right)\right)·\frac{1}{\eta E_{\mathrm{B}},\mathrm{rate}})P_{\mathrm{B}}\left(k\tau \right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill P_{\mathrm{L}}(k\tau +\tau )& =P_{\mathrm{L}}\left(k\tau \right)+\tau g\left(P_{\mathrm{L}}\left(k\tau \right)\right)\hfill \\ \hfill & =\widehat{g}\left(P_{\mathrm{L}}\left(k\tau \right)\right)\hfill \end{array}$$

(7)

$$P_{\mathrm{G}}\left(k\tau \right)=P_{\mathrm{L}}\left(k\tau \right)-P_{\mathrm{B}}\left(k\tau \right)$$

(8)

According to (6)–(8), the state of the system is defined as ${\mathit{x}}_k={[P_{\mathrm{G}}\left(k\tau \right),SOC\left(k\tau \right)-SO{C}_{\mathrm{ref}},P_{\mathrm{L}}\left(k\tau \right)]}^T$, i.e., $x_{1,k}=P_{\mathrm{G}}\left(k\tau \right)$, $x_{2,k}=SOC\left(k\tau \right)-SO{C}_{\mathrm{ref}}$, $x_{3,k}=P_{\mathrm{L}}\left(k\tau \right)$. Let ${\mathit{u}}_k=P_{\mathrm{B}}\left(k\tau \right)$ be the control vector.

For convenience of analysis, delays are introduced in $P_{\mathrm{B}}\left(t\right)$ and $P_{\mathrm{L}}\left(t\right)$. Therefore, the discrete-time building management system is

$$\begin{array}{cc}\hfill {\overline{\mathit{x}}}_{k+1}& =\mathit{A}{\overline{\mathit{x}}}_k+\mathit{B}{\mathit{u}}_k+\mathit{D}{\widehat{\mathit{x}}}_k\hfill \\ \hfill {\widehat{\mathit{x}}}_{k+1}& =g^{\prime }\left({\widehat{\mathit{x}}}_k\right)\hfill \end{array}$$

(9)

where ${\overline{\mathit{x}}}_k={\left[x_{1,k}{x}_{2,k}\right]}^T$ is controllable and ${\widehat{\mathit{x}}}_k=x_{3,k}$ is uncontrollable. The drift, input and disturbance dynamics of controllable subsystem are $\mathit{A}=\mathrm{diag}(0,1)$, $\mathit{B}={[-1sign\left({\mathit{u}}_k\right)·\frac{\eta }{E_{\mathrm{B}},\mathrm{rate}}+sign\left({\mathit{u}}_k\right)·\frac{1}{\eta E_{\mathrm{B}},\mathrm{rate}}]}^T$ and $\mathit{D}={\left[10\right]}^T$, respectively.

It should be noted that the building energy system (9) is partly controllable, since it is assumed that users in the building are not sensitive to the electricity price. Differently from completely controllable systems, the system state will deviate from the equilibrium point. There is not a continuous control law $\mu \left({\mathit{x}}_k\right)$ asymptotically stabilizing the system. Hence, no fixed optimal control law ${\mu }^{*}\left({\mathit{x}}_k\right)$ is able to maintain the optimality from step k to $+\infty$.

It is essential for smart building users to find appropriate discharging/charging power of battery to reduce the electricity cost under the RTP scheme. Hence, EMS is designed to maximize economic benefits of users and extend the battery life. Therefore, a utility function is defined as

$$\begin{array}{cc}\hfill U({\mathit{x}}_k,{\mathit{u}}_k,k)& =m_1{\left({\lambda }_k{x}_{1,k}\right)}^2+m_2{x}_{2,k}^2+m_3{x}_{3,k}^2+m_4{\mathit{u}}_k^T{\mathit{u}}_k\hfill \\ \hfill & ={\mathit{z}}_k^T{\mathit{M}}_k{\mathit{z}}_k\hfill \end{array}$$

(10)

where ${\mathit{M}}_k=diag(m_1{\lambda }_k^2,m_2,m_3,m_4)$ is a time-varying matrix due to dynamic price. The first term denotes the electricity cost, the second term represents the depth of discharge and the fourth term is to shorten the charging/discharging cycle. The utility function is continuous positive definite, which ensures that the performance index function is convex. Then, the performance index function can be defined as the sum of utility functions from time step k to ∞—i.e.,

$$J({\mathit{x}}_k,{\underline{\mathit{u}}}_k,k)=\sum _{i=k}^{\infty }{\gamma }^{i}U({\mathit{x}}_i,{\mathit{u}}_i,i)$$

(11)

Now, our goal is to design an optimal control sequence ${\underline{\mathit{u}}}_k^{*}=\{{\mathit{u}}_k^{*},{\mathit{u}}_{k+1}^{*},\cdots \}$ which minimizes the performance index function (11). For $\forall k$, the optimal control is

$$u^{*}\left({\mathit{x}}_k\right)=arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,{\mathit{u}}_k,k)+\gamma J^{*}({\mathit{x}}_{k+1},k+1)\}$$

(12)

where

$$J^{*}({\mathit{x}}_k,k)=\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,{\mathit{u}}_k,k)+\gamma J^{*}({\mathit{x}}_{k+1},k+1)\}$$

(13)

The discount factor $\gamma$ should be less than one to ensure convergence of the series.

3. Building Energy Management Strategy

In this section, a QOADP-based building energy management strategy is proposed to solve the optimal control problem formulated in Section 2. Moreover, the iterative QOADP algorithm and the corresponding parametric structure are designed.

3.1. Iterative QOADP Algorithm

ADP is an effective tool for solving the nonlinear optimal control problem based on the principle of optimality [11,12]. Generally, the iterative ADP algorithm can be classified into value iteration (VI) and policy iteration (PI). The main idea of ADP is the solution of the Hamilton–Jacobi–Bellman (HJB) equation can be iteratively approximated. We focus on the value iteration in this study.

The building energy system in Section 2 is time-varying, since the utility function varies with time. Hence, the performance index function $J^{*}$ is time-varying; i.e., $J^{*}=J^{*}(x_k,k)$.

An initial value function $V_0(\mathit{x},0)$ is used to approximate $J^{*}(\mathit{x},0)$. Then, the initial iterative control policy ${\mu }_0(\mathit{x},0)$ can be solved by

$$\begin{array}{cc}\hfill {\mu }_0({\mathit{x}}_k,0)& =arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,\mathit{u},0)+\gamma V_0({\mathit{x}}_{k+1},0)\}\hfill \\ \hfill & =arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,\mathit{u},0)+\gamma V_0(F({\mathit{x}}_k,\mathit{u},0),0)\}\hfill \end{array}$$

(14)

Next, the iterative value function at time index $k=0$ can be improved by

$$V_1({\mathit{x}}_k,0)=U({\mathit{x}}_k,{\mu }_0({\mathit{x}}_k,0),0)+\gamma V_0({\mathit{x}}_{k+1},0)$$

(15)

For $k=0$, $i=1,2,\cdots$, the iterative control policy and iterative value function can be updated by

$${\mu }_i({\mathit{x}}_k,0)=arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,\mathit{u},0)+\gamma V_0({\mathit{x}}_{k+1},0)\}$$

(16)

and

$$V_{i+1}({\mathit{x}}_k,0)=U({\mathit{x}}_k,{\mu }_i({\mathit{x}}_k,0),0)+\gamma V_i({\mathit{x}}_{k+1},0)$$

(17)

For $k=1,2,\cdots$, the real-time iterative control policy and the updating iterative value function will be obtained by

$${\mu }_i({\mathit{x}}_k,k)=arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,\mathit{u},k)+\gamma V_i({\mathit{x}}_{k+1},k)\}$$

(18)

$$V_{i+1}({\mathit{x}}_k,k)=U({\mathit{x}}_k,{\mu }_i({\mathit{x}}_k,k),k)+\gamma V_i({\mathit{x}}_{k+1},k)$$

(19)

The above description explains the offline iterative algorithm. However, it is nearly impossible to be applied in the real engineering tasks. The reasons are listed as follows:

(1) The number of iterations is infinite between time step k and time step $k+1$, which cannot be carried out.

(2) The iterative control policy obtained by finite iterations ${\mu }_i({\mathit{x}}_k,k)$ may not be stable. Hence, it is likely to lead to abnormal operations of the system.

To overcome the difficulties, it is meaningful to develop an online algorithm, in which the time index k is equal to iteration index i. Hence, the online value iteration can be summarized as

$${\mu }_k({\mathit{x}}_k,k)=arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,\mathit{u},k)+\gamma V_k({\mathit{x}}_{k+1},k)\}$$

(20)

and

$$V_{k+1}({\mathit{x}}_k,k)=U({\mathit{x}}_k,{\mu }_k({\mathit{x}}_k,k),k)+\gamma V_k({\mathit{x}}_{k+1},k)$$

(21)

with a positive semidefinite initial function $V_0(x_k,0)$. The whole structure diagram is shown in Figure 2.

Note that the online value iteration cannot directly be applied to the EMS, as it is not completely controllable, which differs from most nonlinear systems in the literature [22,28,29]. For controllable systems, there is an admissible control law [30] to bring states of systems to the equilibrium point. Hence, the discount factor should not be omitted in the EMS control. Moreover, the dynamics of load demand is very complex and stochastic. However, it is not desirable to use a complex controller in the real application.

To solve this problem, load forecasting module is separated from the controller, and the controller ${\mu }_k(x_k,k)$ is modified by ${\mu }_k(x_k,{\overline{\widehat{\mathit{x}}}}_{k+1},k)$, where ${\overline{\widehat{\mathit{x}}}}_{k+1}$ is the predictive value of load demand at time step k. The predictive module can be implemented by an exponential smoothing model [31], neural network model, etc. Therefore, the complexity of the controller is reduced, which is beneficial for the real application.

3.2. Parametric Structure Design for Value Function

Function approximation structures should be specified to approximate $V_k$ and ${\mu }_k$. The most commonly used parametric structures are the feedforward neural network [32] and quadratic form neural network [33]. However, they are not the most fitting function approximators to be used in EMS. Here, the iterative value function and iterative control policy are defined as

$$V_k({\mathit{x}}_k,k)={\mathit{x}}_k^T{\mathit{P}}_{0,k}{\mathit{x}}_k+{\mathit{P}}_{1,k}{\mathit{x}}_k+{\mathit{P}}_{2,k}$$

(22)

and

$${\mu }_k(x_k,k)={\mathit{K}}_k{\mathit{x}}_k+{\mathit{b}}_k.$$

(23)

where the coefficients of (22) and (23) can be expressed as

$$\begin{array}{cc}\hfill {\mathit{K}}_k=& -{(2m_4+2\gamma {\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}\mathit{B})}^{-1}\times \hfill \\ \hfill & \left[2\gamma {\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}^T\mathit{A}2\gamma {\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}^T\mathit{D}\right]\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathit{b}}_k=& -{(2m_4+2\gamma {\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}\mathit{B})}^{-1}\times \hfill \\ \hfill & (2\gamma {\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{21},k}^T{\widehat{\mathit{x}}}_{k+1}+\gamma {\tilde{\mathit{P}}}_{1_{11},k}\mathit{B})\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathit{P}}_{0,k+1}& ={\mathit{M}}_{\mathrm{s}}+{\mathit{K}}_k^T{\mathit{M}}_{\mathrm{u}}{\mathit{K}}_k{+\gamma (\left[\mathit{A}\mathit{0}\right]}^T{\tilde{\mathit{P}}}_{0_{11},k}\left[\mathit{A}\mathit{0}\right]\hfill \\ \hfill & +{\left[\mathit{A}\mathit{0}\right]}^T{\tilde{\mathit{P}}}_{0_{11},k}\mathit{B}{\mathit{K}}_k+{\mathit{K}}_k^T{\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}\left[\mathit{A}\mathit{0}\right]\hfill \\ \hfill & +{\mathit{K}}_k^T{\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{11},k}\mathit{B}{\mathit{K}}_k)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill {\mathit{P}}_{1,k+1}& =2{\mathit{b}}_k^T{\mathit{M}}_{\mathrm{u}}{\mathit{K}}_k+\gamma \{\left[{\tilde{\mathit{P}}}_{1_{11},k}\mathit{A}\mathit{0}\right]+{\tilde{\mathit{P}}}_{1_{11},k}\mathit{B}{\mathit{K}}_k+\hfill \\ \hfill & [{\left({\mathit{A}}^T{\tilde{\mathit{P}}}_{0_{12},k}{\widehat{\mathit{x}}}_{k+1}\right)}^T+{\widehat{\mathit{x}}}_{k+1}{\tilde{\mathit{P}}}_{0_{21},k}\mathit{A}\mathit{0}]+\hfill \\ & {\left({\mathit{K}}_k^T{\mathit{B}}^T{\tilde{\mathit{P}}}_{0_{12},k}{\widehat{\mathit{x}}}_{k+1}\right)}^T+{\widehat{\mathit{x}}}_{k+1}^T{\tilde{\mathit{P}}}_{0_{21},k}\mathit{B}{\mathit{K}}_k+\hfill \\ \hfill & {(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)}^T{\tilde{\mathit{P}}}_{0_{11},k}(\left[\mathit{A}\mathit{0}\right]+\mathit{B}{\mathit{K}}_k)+\hfill \\ \hfill & {(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)}^T{\tilde{\mathit{P}}}_{0_{11},k}^T(\left[\mathit{A}\mathit{0}\right]+\mathit{B}{\mathit{K}}_k)\}\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\mathit{P}}_{2,k+1}& ={\mathit{b}}_k^T{\mathit{M}}_{\mathrm{u}}{\mathit{b}}_k+\gamma [{\mathit{P}}_{2,k}+{\tilde{\mathit{P}}}_{1_{11},k}(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)\hfill \\ \hfill & +{\tilde{\mathit{P}}}_{1_{12},k}{\widehat{\mathit{x}}}_{k+1}+({(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)}^T{\tilde{\mathit{P}}}_{0_{12},k}{\widehat{\mathit{x}}}_{k+1}\hfill \\ \hfill & +{\widehat{\mathit{x}}}_{k+1}^T{\tilde{\mathit{P}}}_{0_{21},k}(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)+{\widehat{\mathit{x}}}_{k+1}^T{\tilde{\mathit{P}}}_{0_{22},k}{\widehat{\mathit{x}}}_{k+1})\hfill \\ \hfill & +{(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{x}}_k)}^T{\tilde{\mathit{P}}}_{0_{11},k}(\mathit{B}{\mathit{b}}_k+\mathit{D}{\widehat{\mathit{x}}}_k)\hfill \end{array}$$

(28)

The iterative control law is composed of a proportional control term and a bias vector. If the load demand ${\widehat{\mathit{x}}}_k=0$ for $\forall k=0,1,\cdots$, it becomes a standard LQR problem. The challenge of this paper comes from the dynamic load demand ${\widehat{\mathit{x}}}_k$. To solve this problem, the bias term ${\mathit{b}}_k$ is designed to confront with effects of the uncontrollable state ${\widehat{\mathit{x}}}_k$. It is should be pointed out that the bias term at time k is a function of ${\widehat{\mathit{x}}}_{k+1}$. Moreover, (25) shows that the bias term is a function of ${\tilde{x}}_{k+1}$, which means that the optimal controller contains feedback and feedforward signals.

4. Implementation of the Proposed Algorithm

The recursive formula for parameters of the iterative value function is given in Section 3, but one still can hardly identify parameters of the EMS. Therefore, an online data-driven implementation is necessary.

To carry out the above online iterative ADP algorithm, it is required to design a parametric structure to approximate ${\mu }_k\left({\mathit{x}}_k\right)$ and $V_k\left({\mathit{x}}_k\right)$. The output of the parametric structure is formulated as

$$\tilde{F}(\mathit{X};\mathsf{\omega })={\mathsf{\omega }}^T\mathsf{\sigma }\left(\mathit{X}\right)$$

(29)

where $\mathsf{\sigma }(·)\in {\mathbb{R}}^l$, and l is the dimension of vector $\mathsf{\sigma }(·)$.

The ADHDP technique is adopted to implement the QOADP algorithm, which combines the system dynamics and iterative value [34]. The Q function for ADHDP is defined as

$$\begin{array}{cc}\hfill Q_k({\mathit{x}}_k,{\mathit{u}}_k)& ={\mathsf{\omega }}_{\mathrm{c}}^T{\mathsf{\sigma }}_{\mathrm{c}}\left({\mathit{z}}_k\right)\hfill \\ \hfill & =V_k\left(F({\mathit{x}}_k,{\mathit{u}}_k)\right)\hfill \end{array}$$

(30)

Inputs of the parametric structure of iterative Q function are ${\mathit{x}}_k$ and ${\mathit{u}}_k$, and the activation function ${\mathsf{\sigma }}_c\left({\mathit{z}}_k\right)$ is defined as

$$\begin{array}{cc}\hfill {\mathsf{\sigma }}_c\left({\mathit{z}}_k\right)=& [x_{1,k}^2{x}_{1,k}{x}_{2,k}{x}_{1,k}{x}_{3,k}{x}_{1,k}{u}_k{x}_{2,k}^2{x}_{2,k}{x}_{3,k}\hfill \\ & x_{2,k}{u}_k{x}_{3,k}^2{x}_{3,k}{u}_k{u}_k^2{x}_{1,k}{x}_{2,k}{x}_{3,k}{u}_k{1]}^T\hfill \end{array}$$

(31)

The input of the iterative control policy is ${\mathit{x}}_k$, and the activation function ${\mathsf{\sigma }}_a\left({\mathit{x}}_k\right)$ is defined as

$$\begin{array}{cc}\hfill {\mathsf{\sigma }}_a\left({\mathit{x}}_k\right)=& {\left[x_{1,k}{x}_{2,k}{x}_{3,k}1\right]}^T\hfill \end{array}$$

(32)

The target of function Q is

$$\begin{array}{cc}\hfill Q_{k+1}({\mathit{x}}_k,{\mathit{u}}_k)=& {\mathit{x}}_k^T{Q}_k{\mathit{x}}_k+{\mathit{u}}_k^{T}R{\mathit{u}}_k\hfill \\ \hfill & +{\tilde{Q}}_k({\tilde{\mathit{x}}}_{k+1},{\mu }_k\left({\tilde{\mathit{x}}}_{k+1}\right))\hfill \end{array}$$

(33)

The error of Q function can be defined as

$$E_{\mathrm{c},k}\left({\mathsf{\omega }}_{\mathrm{c},k}\right)={({\tilde{Q}}_k({\mathit{x}}_k,{\mathit{u}}_k;{\mathsf{\omega }}_{\mathrm{c}})-Q_{k+1}({\mathit{x}}_k,{\mathit{u}}_k))}^2$$

(34)

and the parameter-update rule for ${\mathsf{\omega }}_{\mathrm{c},k}$ is a gradient-based iteration given by

$${\mathsf{\omega }}_{\mathrm{c},k}{{|}_{m+1}={\mathsf{\omega }}_{\mathrm{c},k}|}_m-{\alpha }_c\frac{\partial E_{\mathrm{c},k}{|}_m}{\partial {\mathsf{\omega }}_{\mathrm{c},k}{|}_m}$$

(35)

where ${\alpha }_c$ is the learning rate of the iterative Q function and m is the iterative step for updating ${\mathsf{\omega }}_{\mathrm{c},k}$. Define the maximum number of iteration steps as $\mathcal{M}$; then,

$${\mathsf{\omega }}_{\mathrm{c},k+1}={\mathsf{\omega }}_{\mathrm{c},k}{|}_{\mathcal{M}}$$

(36)

The target of the iterative control policy is given by

$${\mu }_k\left({\mathit{x}}_k\right)=arg\underset{\mathit{u}}{min}\{U({\mathit{x}}_k,{\mathit{u}}_k,k)+{\tilde{Q}}_k({\mathit{x}}_{k+1},\mathit{u})\}.$$

(37)

Additionally, error of the iterative control policy is

$$E_{\mathrm{a},k}\left({\mathsf{\omega }}_{\mathrm{a},k}\right)={({\tilde{\mu }}_k({\mathit{x}}_k;{\mathsf{\omega }}_{\mathrm{a}})-{\mu }_k\left({\mathit{x}}_k\right))}^2$$

(38)

Similarly, the weight update is

$${\mathsf{\omega }}_{\mathrm{a},k}{{|}_{m+1}={\mathsf{\omega }}_{\mathrm{a},k}|}_m-{\alpha }_a\frac{\partial E_{\mathrm{a},k-1}{|}_m}{\partial {\mathsf{\omega }}_{\mathrm{a},k-1}{|}_m}$$

(39)

where ${\alpha }_a$ is the learning rate of the iterative control policy

$${\mathsf{\omega }}_{\mathrm{a},k}={\mathsf{\omega }}_{\mathrm{c},k-1}{|}_{\mathcal{M}}$$

(40)

In summary, the data-driven QOADP algorithm implemented by parametric structures is shown in Algorithm 1.

Algorithm 1: Data-driven QOADP algorithm

Initialization: 1: Collect data of an EMS 2: Choose an initial array of ${\omega }_{\mathrm{c},0}$, which ensures initial $Q_0(·)$ to be positive semi-definite. 3: Choose the maximum time step $T_{\mathrm{final}}$. Iteration (Online): 4: Let $k=0$, and let $Q_0({\mathit{x}}_k,{\mathit{u}}_k)=Q_0({\mathit{x}}_k,{\mathit{u}}_k;{\omega }_{\mathrm{c},0})$. 5: Calculate ${\mu }_0\left({\mathit{x}}_k\right)$ and $Q_1({\mathit{x}}_k,{\mathit{u}}_k)$ by (37) and (33) ($k=0$). 6: Let $k=k+1$. 7: Calculate ${\mu }_k\left({\mathit{x}}_k\right)$ and $Q_{k+1}({\mathit{x}}_k,{\mathit{u}}_k)$ by (37) and (33) ($k\ne 0$). 8: If $k>T_{\mathrm{final}}$, go to next step. Otherwise, go to Step 6. 9: return.

5. Hardware-in-Loop Experimental Verification

In this section, two cases of numerical experiments were performed to examine the proposed QOADP algorithm. Comparisons will also be given to prove the superiority of QOADP. In case 1, QOADP was carried out under the assumption that the building load demand and electricity rate are periodic. In case 2, the dual iterative Q-learning (DIQL) algorithm [21], the fuzzy periodic value iteration (FPVI) algorithm [27] and the QOADP algorithm were, respectively, applied to the load demand of high volatility. The advantages of QOADP were verified by comparing the systems states and electricity cost of the three EMSs. For the purpose of comparison, the dual iterative Q-learning (DIQL) algorithm [21] and the fuzzy periodic value iteration (FPVI) algorithm [27] were also applied to the same load demand. Some important parameters for experiment can be found in

Table 1. Parameters for experiment.

Parameters	Value
Capacity of battery in kWh	25
Upper bound of SOC	0.9
Lower bound of SOC	0.15
Rated power output of the battery in kW	3.5
Initial energy of battery in kWh	12.5
Discount factor	0.996

.

5.1. HIL Platform

The proposed QOADP algorithm was verified by an HIL experimental platform, as shown in Figure 3. The HIL platform was composed of an OPAL-RT OP5600 digital simulator as the controlled plant emulator and a dSPACE DS1103 as the controller. DS1103 is a controller board for digital control, which enables the connection between hardware and MATLAB/Simulink simulation environment. By Simulink, algorithms were implemented, compiled and converted to C-code, which was then loaded to the real-time dSPACE processor. The RT-LAB rapid control prototype (RCP) drive system loaded the smart building model, including the utility grid, battery and loads onto a real-time simulation platform configured with the I/O interface required for the study. MATLAB R2021a was the platform, and CPU was an 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz.

5.2. Dataset

The load demand data were taken from the IEEE Power-Energy Society (PES) Open Dataset. The load demand contained the active power of a single household in a family housing from 25 December 2011 to 15 March 2013. The sampling time was 1 h. Figure 4a shows the load demand of a smart building in a week, which has high volatility. In Figure 4b, the load demand in every single day is plotted. The bold line is the average daily load demand. The bold line shows the average load demand in a 24 h time horizon. The electricity price is consistent with the one in [13]. Figure 4c shows the electricity price. The electricity price is quasi-periodic, since it is determined by the aggregated load demand, which has a smaller fluctuation than a single building. Similarly, Figure 4d shows seven daily electricity prices ranging from 1 to 24, and the average electricity price is plotted by a bold line. It is assumed that users are not aware of the electricity price, which means that the load is not scheduled, and the target is to design a control strategy for the battery system.

5.3. Case 1: Periodic Load Demand

In this case, the load demand and electricity price are periodic. Without rescheduling of the energy storage system, the power cost over one week was 1999.2 cents.

The iterative weights of value function and controller are depicted in Figure 5. The learning process shows that weights converge to a periodic sequence. Since the iterative algorithm has an exponential convergence rate, the initial weights have little influence on the performance.

The grid power is plotted in Figure 6a. Each of them works in a safe zone. The battery power is plotted in Figure 6b. Figure 6c shows the SOC of three EMSs. The battery charges when the electricity price is low at midnight and discharges in the daytime. Compared with QOADP, the maximum battery power levels of DIQL- and FPVI-based EMS are larger, which is not beneficial to battery life. Moreover, the total electricity cost of QOADP is less than those of DIQL and FPVI. Hence, QOADP has advantages in both economic costs and battery life degradation. QQADP-based EMS has higher grid power than DIQL and FPVI from 1 am to 6 am only when the electricity price is low.

5.4. Case 2: Load Demand of High Volatility

Consider the load demand with high volatility shown in Figure 4a,c, which depicts the electricity price.

Figure 7 shows ${\omega }_{\mathrm{c},12}$ and ${\omega }_{\mathrm{a},4}$ of the iterative value function and iterative controller. Differently from the periodic case, weights will not converge to an array of parameters, but iterate towards the optimal. In Figure 7, optimal parameters are plotted under load demand of high volatility.

The state and control variables of three EMSs are plotted in Figure 8. Similar to the first case, batteries charge when price is low and discharge when price is high. In QOADP-based EMS, the control variable, i.e., $P_{\mathrm{B}}$, can adapt to the real-time change in load demand. Hence, the battery power compensates the load demand in a good way. Furthermore, the grid power of QOADP is more stable than those of DIQL and FPVI, which is a desirable feature in the utility grid.

The electricity costs of three methods are listed in Figure 9. The proposed algorithm is not only able to adapt to the change of load demand, but also saves more electricity cost for residential users.

6. Conclusions

This paper proposes a QOADP algorithm to realize optimal economic dispatch in smart buildings. The load demand is allowed to be uncertain with high volatility, which is modeled by an uncontrollable state in the EMS. Firstly, the framework of iterative QOADP algorithm is designed to obtain optimal economic dispatch control law. Then, a quasi-quadratic-based function approximation structure with a bias term is utilized to approximate the value function precisely. Moreover, the implementation of the proposed algorithm is given out with detailed steps. The design of QOADP has paved the way for applying ADP technique to EMS of smart buildings and some other systems with uncertainties. Finally, experimental results reveal that QOADP not only can be applied to periodic load demand but also can be employed to load demand of high volatility with superior performance. Our future work will be concerned with expanding the proposed algorithm to more practical engineering systems and formulating a general way to apply ADP.