# Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

**(a) ERWS**, the weighted sum ${\beta}_{1}\mathbf{E}\left[R\right]+{\beta}_{2}\mathbf{E}\left[E\right]$ of the mean response time $\mathbf{E}\left[R\right]$ and the mean power cost $\mathbf{E}\left[E\right]$, where ${\beta}_{1},{\beta}_{2}\ge 0$ are weighted coefficients;

**(b) ERP**, the product $\mathbf{E}\left[R\right]\mathbf{E}\left[E\right]$ of the mean response time and the mean power cost. See Gandhi et al. [12] and Gandhi [10] for systematical research. Maccio and Down [14] generalized the

**ERP**to a more general performance cost function that considers the expected cycle rate. Gebrehiwot et al. [18] made some interesting generalizations of the

**ERP**and

**ERWS**by introducing multiple intermediate sleep states. Additionally, Gebrehiwot et al. [19,20] generalized the FCFS queue of the data center with multiple intermediate sleep states to the processor-sharing discipline and to the shortest remaining processing time (SRPT) discipline, respectively. In addition, Mitrani [21,22] provided another interesting method to discuss the data center of N identical servers that contain m reserves, while the idle or work of the servers is controlled by an up threshold U and a down threshold D. He established a new performance cost $C={c}_{1}\mathbf{E}\left[L\right]+{c}_{2}\mathbf{E}\left[S\right]$ and provided expressions for the average numbers $\mathbf{E}\left[L\right]$ and $\mathbf{E}\left[S\right]$, so that the performance cost C can be optimized with respect to the three key parameters m, U, and D.

## 2. Model Description

**Server groups:**The data center contains two server groups: Groups 1 and 2, each of which is also one interactive subsystem of the data center. Groups 1 and 2 have ${m}_{1}$ and ${m}_{2}$ servers, respectively. Servers in the same group are homogeneous, while those in different groups are heterogeneous. Note that Group 1 is viewed as a base-line group whose servers are always at the work state even if some of them are idle, the purpose of which is to guarantee a necessary service capacity in the data center. Hence, each server in Group 1 always works regardless of whether it has a job or not, so that it must consume an amount of energy at any time. In contrast, Group 2 is regarded as a reserved group whose servers may either work or sleep so that each of the ${m}_{2}$ servers can switch its state between work and sleep. If one server in Group 2 is at the sleep state, then it consumes a smaller amount of energy than the work state, as maintaining only the sleep state requires very little energy.

**A finite buffer:**The data center has a finite buffer of size ${m}_{3}$. Jobs must first enter the buffer, and then they are assigned to the groups (Group 1 is prior to Group 2) and subsequently to the servers. To guarantee that the float service capacity of Group 2 can be fully utilized when some jobs are taken from the buffer to Group 2, we assume that ${m}_{3}\ge {m}_{2}$, i.e., the capacity of the buffer must be no less than the server number of Group 2. Otherwise, if there are more jobs waiting in the buffer, the jobs transferred from Group 2 to the buffer will be lost.

**Arrival processes:**The arrivals of jobs at the data center are a Poisson process with arrival rate $\lambda $. If the buffer is full, then any arriving job has to be lost immediately. This leads to an opportunity cost ${C}_{5}$ per unit of time for each lost job due to the full buffer.

**Service processes:**The service times provided by each server in Groups 1 and 2 are i.i.d. and exponential with service rates ${\mu}_{1}$ and ${\mu}_{2}$, respectively. We assume that ${\mu}_{1}\ge {\mu}_{2},$ which makes the prior use of servers in Group 1. The service discipline of each server in the data center is First Come First Serve (FCFS). If a job finishes its service at a server, then it immediately leaves the system. At the same time, the data center can obtain a fixed service reward (or service price) $R$ from the served job.

**Switching between work and sleep:**To save energy, the servers in Group 2 can switch between the work and sleep states. On the one hand, if there are more jobs waiting in the buffer, then Group 2 sets up and turns on some sleeping servers. This process usually involves a setup cost ${C}_{3}^{\left(1\right)}.$ However, the setup time is very short as it directly begins from the sleep state and it can be ignored. On the other hand, if the number of jobs in Group 2 is smaller, then the working servers are switched to the sleep state, while the incomplete-service jobs are transferred to the buffer and served as the arriving ones.

**Transfer rules:**(1) To Group 1. Based on the prior use of servers in Group 1, if a server in Group 1 becomes idle and there is no job in the buffer, then an incomplete-service job (if it exists) in Group 2 must immediately be transferred to the idle server in Group 1. Additionally, the data center needs to pay a transferred cost ${C}_{4}$ to the transferred job.

**Power Consumption:**The power consumption rates ${P}_{1,W}$ and ${P}_{2,W}$ are for the work states of servers in Groups 1 and 2, respectively, while ${P}_{2,S}$ is only for the sleep state of a server in Group 2. Note that each server in Group 1 does not have the sleep state and it is clear that ${P}_{1,S}=0$. We assume that $0<{P}_{2,S}<$ ${P}_{2,W}$. There is no power consumption for keeping the jobs in the buffer. ${C}_{1}$ is the power consumption price per unit of the power consumption rate and per unit of time.

**Independence:**We assume that all the random variables in the data center defined above are independent.

**Remark**

**1.**

**Remark**

**2.**

## 3. Optimization Model Formulation

#### 3.1. A Policy-Based Block-Structured Continuous-Time Markov Process

**Remark**

**3.**

**Theorem**

**1.**

**Proof**

**of**

**Theorem**

**1.**

**Remark**

**4.**

#### 3.2. The Reward Function

**Case (a):**For ${n}_{1}=0,1,\dots ,{m}_{1}$ and ${n}_{2}={n}_{3}=0$, the profit rate is not affected by any policy, and we have

**Case (b):**For ${n}_{1}={m}_{1}$, ${n}_{2}=0$ and ${n}_{3}=1,2,\dots ,{m}_{3},$ the profit rate is affected by the setup policy ${\mathit{d}}^{W}$, we have

**Case (c):**For ${n}_{1}={m}_{1}$, ${n}_{2}=1,2,\dots ,{m}_{2}$ and ${n}_{3}=0,1,\dots ,{m}_{3}-{n}_{2}$, the profit rate is affected by the sleep policy ${\mathit{d}}^{S}$, we have

**Case (d):**For ${n}_{1}={m}_{1}$, ${n}_{2}={m}_{2}$ and ${n}_{3}={m}_{3}-{m}_{2}+1,{m}_{3}-{m}_{2}+2,\dots ,{m}_{3}$, the profit rate is not affected by any policy, we have

## 4. The Block-Structured Poisson Equation

**Theorem**

**2.**

**Proof**

**of**

**Theorem**

**2.**

## 5. Impact of the Service Price

#### 5.1. The Setup Policy

**Proposition**

**1.**

**Proof**

**of**

**Proposition**

**1.**

#### 5.2. The Sleep Policy

**Proposition**

**2.**

**Theorem**

**3.**

## 6. Monotonicity and Optimality

**Lemma**

**1.**

#### 6.1. The Service Price $R\ge {R}_{H}$

#### 6.1.1. The Setup Policy with $R\ge {R}_{H}^{W}$

**Theorem**

**4.**

**Proof**

**of**

**Theorem**

**4.**

**Theorem**

**5.**

**Proof**

**of**

**Theorem**

**5.**

#### 6.1.2. The Sleep Policy with $R\ge {R}_{H}^{S}$

**Theorem**

**6.**

**Proof**

**of**

**Theorem**

**6.**

**Theorem**

**7.**

#### 6.2. The Service Price $0\le R\le {R}_{L}$

#### 6.2.1. The Setup Policy with $0\le R\le {R}_{L}^{W}$

**Theorem**

**8.**

**Proof**

**of**

**Theorem**

**8.**

#### 6.2.2. The Sleep Policy with $0\le R\le {R}_{L}^{S}$

**Theorem**

**9.**

**Proof**

**of**

**Theorem**

**9.**

**Theorem**

**10.**

#### 6.3. The Service Price ${R}_{L}<R<{R}_{H}$

#### 6.3.1. The Setup Policy with ${R}_{L}^{W}<R<{R}_{H}^{W}$

**Theorem**

**11.**

**Proof**

**of**

**Theorem**

**11.**

**Corollary**

**1.**

#### 6.3.2. The Sleep Policy with ${R}_{L}^{S}<R<{R}_{H}^{S}$

**Theorem**

**12.**

**Proof**

**of**

**Theorem**

**12.**

**Corollary**

**2.**

## 7. The Maximal Long-Run Average Profit

**Case**

**1.**

**Case**

**2.**

**Remark**

**5.**

**Case**

**3.**

**Remark**

**6.**

**Remark**

**7.**

## 8. Conclusions

- Analyzing non-Poisson inputs such as Markovian arrival processes (MAPs) and/or non-exponential service times, e.g., the PH distributions;
- Developing effective algorithms for finding the optimal dynamic policies of the policy-based block-structured Markov process (i.e., block-structured MDPs);
- Discussing the fact that the long-run performance is influenced by the concave or convex reward (or cost) function;
- Studying individual optimization for the energy-efficient management of data centers from the perspective of game theory.

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A. Special Cases

**Case**

**A1.**

**Case**

**A2.**

**Case 1**, for the state transitions in which the transfer rates and the policies are simultaneously involved in the Markov chain with sync jumps of multi-events, the transfer rate is equal to the total entering rate at a certain state.

## Appendix B. State Transition Relations

**(a) The arrival and service rates:**The first type is ordinary for the arrival and service rates without any policy (see Figure A3a).

**(b) The setup policy:**For ${k}_{2}={k}_{1},$ we write

**(c) The sleep policy:**For ${k}_{3}=1,2,\dots ,{m}_{2},$${k}_{4}=0,1,\dots ,{m}_{3}-{k}_{3},$ and ${k}_{5}=0,1,\dots ,{m}_{2},$ we write

**Remark**

**A1.**

**Remark**

**A2.**

## Appendix C. Block Elements in Q^{(d)}

**(a)**For level 0, it is easy to see that

**(b)**For level 1, the setup policy affects the infinitesimal generator, and ${Q}_{1,0}$ is given by

**(c)**For level 2, i.e., ${k}_{3}=1$,

**(d)**For level ${k}_{3}+1$, ${k}_{3}=2,3,\dots ,{m}_{2}-2$, subdivided into three cases as follows:

**(e)**For level ${m}_{2}+1$, i.e., ${k}_{3}={m}_{2},$

**(f)**For level ${m}_{2}+2$,

## References

- Masanet, E.; Shehabi, A.; Lei, N.; Smith, S.; Koomey, J. Recalibrating global data center energy-use estimates. Science
**2020**, 367, 984–986. [Google Scholar] [CrossRef] [PubMed] - Zhang, Q.; Meng, Z.; Hong, X.; Zhan, Y.; Liu, J.; Dong, J.; Bai, T.; Niu, J.; Deen, M.J. A survey on data center cooling systems: Technology, power consumption modeling and control strategy optimization. J. Syst. Archit.
**2021**, 119, 102253. [Google Scholar] [CrossRef] - Nadjahi, C.; Louahlia, H.; Lemasson, S. A review of thermal management and innovative cooling strategies for data center. Sustain. Comput. Inform. Syst.
**2018**, 19, 14–28. [Google Scholar] [CrossRef] - Koot, M.; Wijnhoven, F. Usage impact on data center electricity needs: A system dynamic forecasting model. Appl. Energy
**2021**, 291, 116798. [Google Scholar] [CrossRef] - Shirmarz, A.; Ghaffari, A. Performance issues and solutions in SDN-based data center: A survey. J. Supercomput.
**2020**, 76, 7545–7593. [Google Scholar] [CrossRef] - Li, Q.L.; Ma, J.Y.; Xie, M.Z.; Xia, L. Group-server queues. In Proceedings of the International Conference on Queueing Theory and Network Applications, Qinhuangdao, China, 21–23 August 2017; pp. 49–72. [Google Scholar]
- Harchol-Balter, M. Open problems in queueing theory inspired by data center computing. Queueing Syst.
**2021**, 97, 3–37. [Google Scholar] [CrossRef] - Barroso, L.A.; Hölzle, U. The case for energy-proportional computing. Computer
**2007**, 40, 33–37. [Google Scholar] [CrossRef] - Kuehn, P.J.; Mashaly, M.E. Automatic energy efficiency management of data center resources by load-dependent server activation and sleep modes. Ad Hoc Netw.
**2015**, 25, 497–504. [Google Scholar] [CrossRef] - Gandhi, A. Dynamic Server Provisioning for Data Center Power Management. Ph.D. Thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA, 2013. [Google Scholar]
- Gandhi, A.; Doroudi, S.; Harchol-Balter, M.; Scheller-Wolf, A. Exact analysis of the M/M/k/setup class of Markov chains via recursive renewal reward. Queueing Syst.
**2014**, 77, 177–209. [Google Scholar] [CrossRef] - Gandhi, A.; Gupta, V.; Harchol-Balter, M.; Kozuch, M.A. Optimality analysis of energy-performance trade-off for server farm management. Perform. Eval.
**2010**, 67, 1155–1171. [Google Scholar] [CrossRef][Green Version] - Gandhi, A.; Harchol-Balter, M.; Adan, I. Server farms with setup costs. Perform. Eval.
**2010**, 67, 1123–1138. [Google Scholar] [CrossRef] - Maccio, V.J.; Down, D.G. On optimal policies for energy-aware servers. Perform. Eval.
**2015**, 90, 36–52. [Google Scholar] [CrossRef][Green Version] - Maccio, V.J.; Down, D.G. Exact analysis of energy-aware multiserver queueing systems with setup times. In Proceedings of the IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, London, UK, 19–21 September 2016; pp. 11–20. [Google Scholar]
- Phung-Duc, T. Exact solutions for M/M/c/setup queues. Telecommun. Syst.
**2017**, 64, 309–324. [Google Scholar] [CrossRef][Green Version] - Phung-Duc, T.; Kawanishi, K.I. Energy-aware data centers with s-staggered setup and abandonment. In Proceedings of the International Conference on Analytical and Stochastic Modeling Techniques and Applications, Cardiff, UK, 24–26 August 2016; pp. 269–283. [Google Scholar]
- Gebrehiwot, M.E.; Aalto, S.; Lassila, P. Optimal energy-aware control policies for FIFO servers. Perform. Eval.
**2016**, 103, 41–59. [Google Scholar] [CrossRef] - Gebrehiwot, M.E.; Aalto, S.; Lassila, P. Energy-performance trade-off for processor sharing queues with setup delay. Oper. Res. Lett.
**2016**, 44, 101–106. [Google Scholar] [CrossRef] - Gebrehiwot, M.E.; Aalto, S.; Lassila, P. Energy-aware SRPT server with batch arrivals: Analysis and optimization. Perform. Eval.
**2017**, 15, 92–107. [Google Scholar] [CrossRef] - Mitrani, I. Service center trade-offs between customer impatience and power consumption. Perform. Eval.
**2011**, 68, 1222–1231. [Google Scholar] [CrossRef] - Mitrani, I. Managing performance and power consumption in a server farm. Ann. Oper. Res.
**2013**, 202, 121–134. [Google Scholar] [CrossRef] - Kamitsos, I.; Ha, S.; Andrew, L.L.; Bawa, J.; Butnariu, D.; Kim, H.; Chiang, M. Optimal sleeping: Models and experiments for energy-delay tradeoff. Int. J. Syst. Sci. Oper. Logist.
**2017**, 4, 356–371. [Google Scholar] [CrossRef] - Hipp, S.K.; Holzbaur, U.D. Decision processes with monotone hysteretic policies. Oper. Res.
**1988**, 36, 585–588. [Google Scholar] [CrossRef] - Lu, F.V.; Serfozo, R.F. M/M/1 queueing decision processes with monotone hysteretic optimal policies. Oper. Res.
**1984**, 32, 1116–1132. [Google Scholar] [CrossRef] - Yang, J.; Zhang, S.; Wu, X.; Ran, Y.; Xi, H. Online learning-based server provisioning for electricity cost reduction in data center. IEEE Trans. Control Syst. Technol.
**2017**, 25, 1044–1051. [Google Scholar] [CrossRef] - Ding, D.; Fan, X.; Zhao, Y.; Kang, K.; Yin, Q.; Zeng, J. Q-learning based dynamic task scheduling for energy-efficient cloud computing. Future Gener. Comput. Syst.
**2020**, 108, 361–371. [Google Scholar] [CrossRef] - Liang, Y.; Lu, M.; Shen, Z.J.M.; Tang, R. Data center network design for internet-related services and cloud computing. Prod. Oper. Manag.
**2021**, 30, 2077–2101. [Google Scholar] [CrossRef] - Xia, L.; Chen, S. Dynamic pricing control for open queueing networks. IEEE Trans. Autom. Control
**2018**, 63, 3290–3300. [Google Scholar] [CrossRef] - Xia, L.; Miller, D.; Zhou, Z.; Bambos, N. Service rate control of tandem queues with power constraints. IEEE Trans. Autom. Control
**2017**, 62, 5111–5123. [Google Scholar] [CrossRef] - Ma, J.Y.; Xia, L.; Li, Q.L. Optimal energy-efficient policies for data centers through sensitivity-based optimization. Discrete Event Dyn. Syst.
**2019**, 29, 567–606. [Google Scholar] [CrossRef][Green Version] - Chi, C.; Ji, K.; Marahatta, A.; Song, P.; Zhang, F.; Liu, Z. Jointly optimizing the IT and cooling systems for data center energy efficiency based on multi-agent deep reinforcement learning. In Proceedings of the 11th ACM International Conference on Future Energy Systems, Virtual Event, Australia, 22–26 June 2020; pp. 489–495. [Google Scholar]
- Li, Q.L. Constructive Computation in Stochastic Models with Applications: The RG-Factorizations; Springer: Beijing, China, 2010. [Google Scholar]
- Cao, X.R. Stochastic Learning and Optimization—A Sensitivity-Based Approach; Springer: New York, NY, USA, 2007. [Google Scholar]
- Xia, L.; Zhang, Z.G.; Li, Q.L. A c/μ-rule for for job assignment in heterogeneous group-server queues. Prod. Oper. Manag.
**2021**, 1–18. [Google Scholar] [CrossRef] - Li, Q.L.; Li, Y.M.; Ma, J.Y.; Liu, H.L. A complete algebraic transformational solution for the optimal dynamic policy in inventory rationing across two demand classes. arXiv
**2019**, arXiv:1908.09295v1. [Google Scholar] - Ma, J.Y.; Li, Q.L. Sensitivity-based optimization for blockchain selfish mining. In Proceedings of the International Conference on Algorithmic Aspects of Information and Management, Dallas, TX, USA, 20–22 December 2021; pp. 329–343. [Google Scholar]
- Xia, L. Risk-sensitive Markov decision processes with combined metrics of mean and variance. Prod. Oper. Manag.
**2020**, 29, 2808–2827. [Google Scholar] [CrossRef] - Puterman, M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming; John Wiley& Sons: New York, NY, USA, 2014. [Google Scholar]
- Budhiraja, A.; Friedlander, E. Diffusion approximations for controlled weakly interacting large finite state systems with simultaneous jumps. Ann. Appl. Probab.
**2018**, 28, 204–249. [Google Scholar] [CrossRef][Green Version]

Cost | Necessary Interpretation |
---|---|

${C}_{1}$ | The power consumption price |

${C}_{2}^{\left(1\right)}$ | The holding cost for a job in Group 1 per unit of sojourn time |

${C}_{2}^{\left(2\right)}$ | The holding cost for a job in Group 2 per unit of sojourn time |

${C}_{2}^{\left(3\right)}$ | The holding cost for a job in the buffer per unit of sojourn time |

${C}_{3}^{\left(1\right)}$ | The setup cost for a server switching from the sleep state to the work state |

${C}_{3}^{\left(2\right)}$ | The transferred cost for a incomplete-service job returning to the buffer |

${C}_{4}$ | The transferred cost for a job in Group 2 is transferred to Group 1 |

${C}_{5}$ | The opportunity cost for each lost job |

R | The service price from the served job |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ma, J.-Y.; Li, Q.-L.; Xia, L.
Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers. *Systems* **2022**, *10*, 27.
https://doi.org/10.3390/systems10020027

**AMA Style**

Ma J-Y, Li Q-L, Xia L.
Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers. *Systems*. 2022; 10(2):27.
https://doi.org/10.3390/systems10020027

**Chicago/Turabian Style**

Ma, Jing-Yu, Quan-Lin Li, and Li Xia.
2022. "Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers" *Systems* 10, no. 2: 27.
https://doi.org/10.3390/systems10020027