Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Abstract

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

1. Introduction

Operators of cloud services, such as Alibaba Cloud (Alicloud) and Amazon Web Services (AWS), rent out computing capacity (virtual machines, VMs) as a cloud service to millions of users. These users, ranging from individuals to large institutions, usually rent and pay only for a set of VMs through a posted fixed-pricing model with service-level agreements (SLAs). In the fixed-pricing market, some users enter into long-term contracts to reserve resources, and many choose to obtain resources on demand (i.e., pay as you use). The utilization of these resources is highly time-varying, compounded with uncertain peak–trough changes [1]. During trough times, VMs, on average, run under 30% capacity in global data centers [2], with the majority of these running under 20% utilization [3]. Resource utilization can have sudden unexpected peak–trough shifts [4]. For example, the applications deployed on these resources have new product launches and marketing campaigns, which suddenly generate a huge workload influx [5]. This high volatility for time-varying resource utilization leads to non-stationary excess capacities, which are surplus capacities after meeting the usage of fixed-pricing markets. Considering the perishable and non-storable nature of services in the cloud, the unused excess computing capacities are not able to be transferred from one period to the next. Excess capacity will lead to the potential waste of resources, not to mention the fact that cloud operators still need to pay for these excess capacities (e.g., electricity, hardware, and maintenance costs).

Managing non-stationary excess capacity without violating the written SLAs has become a significant emerging issue. Therefore, some leading cloud operators, including AWS and Alicloud, have launched a relatively new type of service: low-priority spot (pre-emptive) services. Taking Alicloud as an example, it adopts a variable price that is adjusted every five minutes with the following statements: “Spot (preemptive) instance is the temporarily unused regular service (e.g., Elastic Compute Cloud instances), and is offered at a lower price than on-demand instances. Every spot user is required to indicate a maximum price that the user is willing to pay. Only users with a willingness to pay higher than the current spot price can get the spot cloud service. However, if the market price varies higher than the user’s bid or if supply and demand change, the instances are automatically released [6]”. Because there is no SLA violation for low-priority spot services, the operators can reclaim capacities and interrupt these spot users’ computing jobs when resource scarcity occurs. Therefore, this special real-time pricing mechanism for spot cloud services can be completely adapted to the non-stationarity of excess capacity.

However, due to the non-stationarity of excess capacity with uncertain peak–trough changes, cloud operators cannot directly observe the excess capacity available to spot services for every given spot price decision in advance; that is, when making price decisions for spot cloud services, operators cannot know the excess capacity that can be provided to spot cloud services during the decision period. To avoid revenue losses from offering sub-optimal prices, operators also need to make spot pricing decisions based on all historical excess capacities, which causes spot pricing decisions to face a huge search space and to be difficult to adapt to real-time decision-making requirements for spot pricing.

Dynamic pricing for cloud computing is an attractive scheme and of great interest to the academic community devoted to seeking the optimal dynamic pricing policy. Ben-Yehuda et al. [7] and George et al. [8] have examined the dynamic pricing traces of spot services from Amazon EC2 and Alibaba Cloud, and provided qualitative changes which showed that spot prices are artificially set. Xu et al. [9] developed and analyzed market-driven dynamic pricing mechanisms for spot instances, and adopted a revenue management framework to formulate the revenue maximization problem with dynamic pricing as a stochastic dynamic program. Furthermore, Alzhouri et al. [10] developed a systematic approach toward the dynamic prices of stagnant VMs of both one class and multiple classes to achieve the maximum expected revenue within a finite discrete time horizon. Lei et al. [11] examined a dynamic pricing problem in the face of non-stationary, stochastic, and price-sensitive demand for instances, proposed heuristic methods, and showed the near optimality of the proposed methods when both demand and supply are sufficiently large. Bonacquisto et al. [12] proposed an alternative, dynamic pricing model for selling unused capacity. It is worth pointing out that these works focused on fixed excess capacity while ignoring the non-stationarity of excess capacity. In fact, an efficient real-time spot pricing method should rely on non-stationary excess capacity supply and interruptible demand sides, and will also need an integrated framework that must be considered.

The research on non-stationarity in cloud environments mainly focuses on improving the prediction accuracy of data center resource utilization [13,14]. However, the high volatility in workload makes it difficult to estimate future resource utilization accurately in real time. Additionally, pricing decisions based on predicted resource utilization can only obtain local optimal policies. The non-stationarity in the cloud environment has different workload patterns partitioned by random peak–trough points [15]. When the peak–trough points are detected, pattern structures before and after these points can be estimated based on historical data. Detecting when these peak–trough points happen in a univariate time series has a long, rich history in the statistical literature. Palshikar [16] measured different peak point detection algorithms in any given time series. Tahir et al. [17] investigated the online burst detection technique using sample entropy. Based on changepoint detection thinking, this paper identifies cloud computing change patterns and learns optimal pricing policies for different patterns.

Reinforcement learning (RL) is a simulation-based method that can autonomously learn effective policies for sequential decision problems under uncertainty [18]. As a simulation-based algorithm, decision makers learn how to make optimal pricing decisions by interacting with stochastic simulated environments. Driven by the advantages of optimal learning under uncertainty, RL has become an important tool for revenue management and dynamic pricing problems. Cheng [19] applied the Q-learning approach to dynamic pricing in e-retailing. Gosavii et al. [20] adopted the RL technique to a single-leg airline revenue management problem with multiple fare classes and overbooking. Zhang et al. [21] applied the Q-learning approach to real-time pricing in a smart grid to solve the MDP model without the acquisition of transition probabilities. Alsarhan et al. [22] integrated computing resource adaptation with service admission control based on an RL model. These studies on the application of an RL methodology in revenue management all assume that the environment model is stationary.

The stationarity assumption of the sequential decision problems greatly simplifies real complex environments. Therefore, recent RL works have generalized the framework by incorporating changing environments. Abdallah et al. [23] proposed repeated update Q-learning (RUQL), a variant of Q-learning. It essentially repeats the updates to the Q-values of a state–action pair and is shown to have learning dynamics better suited to non-stationary environment tasks in simulation experiments. Mao et al. [24] considered Q-learning in non-stationary Markov decision processes and proposed a fixed-window size restarted Q-learning (named FWQL). Padakandla et al. [25] proposed a context-detection-based RL algorithm to solve non-stationary environments, which assumed known context change sequences. Very few prior works have considered revenue management problems in non-stationary environment models based on RL algorithms. The research most relevant to this article is that of Rana et al. [26,27], which applied a Q(λ) learning approach to solve the dynamic pricing problem with interdependent demands. This study verifies that the Q(λ) learning approach outperforms the classical Q-learning algorithm in time-varying demand environments.

In contrast to existing studies, in this work, based on the RL technique, we aimed to study how to price spot services in real time to match interruptible demand technically, with non-stationary unused excess capacity. We present here an interruptible demand model for the spot segment market and formulate the real-time spot pricing problem as a non-stationary Markov decision process (MDP) with the non-stationarity of excess capacity. We model non-stationary excess capacity as a piecewise linear pattern. An improved reinforcement learning algorithm, being aware of excess capacity pattern shifts, is proposed.

The main contributions of this paper can be summarized as follows:

• This paper develops a non-stationary real-time pricing decision model from Markov decision process theory, which captures the underlying dynamics of real-time spot cloud pricing. The model framework is much more in line with the spot cloud service schemes practiced by cloud operators.
• An RL methodology is adopted to solve the formulated decision-making model adaptively without the acquisition of transition probabilities. Different from existing ones, a non-stationary Q-learning algorithm is proposed to decide the real-time spot price for handling non-stationarity. Its advantages are manifested in two aspects: (1) online peak–trough point detection and (2) the learning of optimal pricing policies with changing patterns. To the best of our knowledge, this paper is the first study to investigate real-time spot pricing through reinforcement learning methods in a non-stationary cloud environment.
• We conducted experiments based on simulated and real-world non-stationary excess capacity. Experimental validation shows that the proposed solution with which we solve our problem outperforms state-of-the-art approaches. We also provide valuable insights into cloud operators’ capacity management practices.

The remainder of this paper is structured as follows. In Section 2, a non-stationary real-time pricing decision model for spot cloud services is formulated in the framework of an MDP. In Section 3, a non-stationary Q-learning algorithm based on RL is proposed to solve the non-stationary stochastic problem. In Section 4, the experiment validations in the simulated and real-world non-stationary scenarios are presented. Finally, the conclusions of the paper are discussed in Section 5.

2. Non-Stationary Real-Time Pricing Decision Model for Spot Cloud Services

This study focuses on a real-time pricing problem of spot cloud services to maximize non-stationary excess capacity revenue. In this study, we consider a cloud operator and focus on the spot market segment in which users can tolerate service interruptions. For cloud operators, there might be regular and spot users accessing the cloud simultaneously. Spot users mainly need resources for computing purposes and their jobs do not require persistence, whereas regular users look for cloud services with guaranteed service-level agreements. It is possible that competition among regular service operators or among spot service operators exists, but not across the two types. Thus, the behavior of fixed-price regular users is not included in our model. Instead, the time-varying (non-stationary) workload of regular users leads to the non-stationarity of excess capacity available for the spot market. In this section, a real-time spot pricing decision model with non-stationary excess capacity is formulated.

2.1. Model Assumption for Non-Stationary Excess Capacity

In the spot pricing decision-making process, excess capacity is an important reference factor for spot service pricing. The non-stationarity of excess capacity can significantly affect the dynamics of spot pricing decisions. We model the non-stationarity of excess capacity according to the characteristics of the non-stationary utilization of resources in fixed-price regular services (e.g., on-demand and reserved services).

We assume a time horizon, T, as a selling period and discretize the time horizon, T, into identically sized slots. There are T total periods and t indices for the periods, with the time index running forward (t = 1 is the first period and t = T is the last period). Let N denote the total capacity (VMs) of the cloud operator, which is stable in the selling finite horizon, T. $u_t$ represents the resource utilization of a fixed-price regular service in each period, t (t = 1, 2, 3, …, T), and $c_t$ denotes the excess capacity available to spot services in period t. Thus, we have $c_t=N-u_t$. The utilization of resources in fixed-price regular services changes with the changing workloads of computing jobs running on them. The higher the workloads of computing jobs, the higher the utilization of resources. The workloads on fixed-price regular services are non-stationary time series with random peak–trough variations. Some peak–trough variations are certain, and some are uncertain. According to the literature [15], these random peak–trough variations make the utilization of resources non-linear, such as on-and-off trends. These non-linear structures can be partitioned into linear patterns in a relatively short time interval and described as a linear model, as in Equation (1):

$$c_t=\alpha +\beta t+{\epsilon }_t$$

(1)

The coefficients $(\alpha ,\beta )$ are determined by historical excess capacity time series $(c_{t-1},c_{t-2},\dots ,c_1)$. ${\epsilon }_t$ is unobservable shocks. Assume that ${\epsilon }_t$ represents independent and identically distributed random variables with mean zero and variance ${\sigma }^2$. For notational brevity, let $\psi =(\alpha ,\beta )$ denote the estimated parameter vector of excess capacity. The non-linearity structure of the changing excess capacity can be partitioned as different linear patterns by peak–trough points [1]. Therefore, we assume non-stationary excess capacity to be a piecewise linear model with peak–trough points (presented in Equation (2)). We assume that ${\theta }_k$ ($1\le k\le K$) denotes one linear pattern type, where K is the total number of the patterns in time horizon T and $T_k$ denotes the peak or trough point from pattern ${\theta }_{k-1}$ to pattern ${\theta }_k$. Let $t_k$ index the time in pattern ${\theta }_k$. Herein, the non-stationarity model of excess capacity, $c_t$, can be characterized as Equation (2):

$$c_t=\{\begin{array}{ll}{\alpha }_{{\theta }_1}+{\beta }_{{\theta }_1}{t}_1+{\epsilon }_t,& 0\le t_1<t-T_1\\ {\alpha }_{{\theta }_2}+{\beta }_{{\theta }_2}{t}_2+{\epsilon }_t,& 0\le t_2<t-T_2\\ \dots & \\ {\alpha }_{{\theta }_K}+{\beta }_{{\theta }_K}{t}_K+{\epsilon }_t,& 0\le t_K<t-T_K\end{array}$$

(2)

where ${\psi }_{{\theta }_k}=({\alpha }_{{\theta }_k},{\beta }_{{\theta }_k})$ denotes the estimated parameter vector of pattern ${\theta }_k$ in the time interval [0, $t-T_k$]. Let $\mathsf{\Theta }$ ($\mathsf{\Theta }$$=\{{\theta }_1,{\theta }_2,\dots ,{\theta }_K\}$) denote the sequence set of pattern types. Let $\mathsf{{\rm Y}}$ = {$T_1$, $T_2$, …, $T_K$} denote the sequence set of peak–trough points. The estimation of ${\theta }_k$ and $T_k$ is described in detail in Section 3.2.

The workloads of applications deployed on cloud platforms usually have a certain stochastic regularity, but when a sudden business activity occurs, excess resource capacity will suddenly change and enter a new pattern. Therefore, we make the following assumptions about the non-stationarity:

(1)

The non-stationary excess capacity model Equation (2) changes at least once (that is, $K\ge 2$), and the number of pattern types is finite.

(2)

The pattern sequence $\mathsf{\Theta }$, and peak–trough sequence set $\mathsf{{\rm Y}}$, are all unknown by cloud operators and need to be estimated with ongoing time.

2.2. Demand Model of Spot Cloud Services

2.2.1. Real-Time Spot Pricing Scheme

To maximize non-stationary excess capacity’s revenue without violating service-level agreements (SLAs) for fixed-price regular services (e.g., on-demand and reserved instances), the prices of spot cloud services are dynamic in real time with the following pricing scheme [6]:

(1)

The cloud operator will periodically post a new spot price at the beginning of each period, and users need to submit their willingness to pay; that is, the maximum price a customer is willing to pay for each spot instance.

(2)

When the submitted willingness to pay is higher than the spot price, spot users can obtain the spot instances. When the spot price exceeds the submitted willingness to pay or when resource scarcity occurs, cloud operators will interrupt the running spot instances.

(3)

Fees for all running spot instances are automatically deducted from online systems based on the current prices and usage; there is no SLA violation for the interruption of the running spot instances.

We assume that the willingness to pay submitted by spot users can be approximated as their real willingness to pay. The reasons for this are as follows. (1) Whether or not a user’s computing job is interrupted does not depend entirely on the willingness to pay submitted by the user, but also on the non-stationarity of excess capacity. That is, users who submit a high willingness to pay may be interrupted when resource scarcity occurs, and users with a low submitted willingness to pay can be completed without service interruptions when resources are abundant. (2) Cloud services are online services, and the willingness to pay submitted by a spot user is invisible to others; the requirement for computing resources is random. It is difficult for users to collude with each other to drive down the submitted willingness to pay, while users are also unwilling to pay spot prices higher than their willingness to pay. For these two reasons, we assume that the maximum prices submitted by spot users can be approximated as their real willingness to pay.

2.2.2. Estimation of Willingness to Pay

According to the above spot service scheme, in a spot market consisting of temporarily unused instances, spot users must cope with interruptions at all times. Interruptions caused by spot cloud services will degrade the service quality of spot services. As discussed in Ref. [28], service quality is important to intelligent service systems and will affect user experiences. The degraded service quality will reduce the perceived value of spot services, thereby reducing users’ willingness to pay. According to the definition of willingness to pay, willingness to pay is the maximum price that a customer is willing to pay for a good or service. Perceived value is measured by the price that a user is willing to pay; therefore, we assume that spot users’ willingness to pay is equal to the perceived value of one spot instance per time. It is obvious that interruptions will reduce perceived value in addition to willingness to pay.

Let $v_0$ be the perceived value for each spot instance per time unit without an interruption. Let $w_t$ be a user’s willingness to pay for one spot instance per time unit in time t. Let ${\zeta }_t$($0<{\zeta }_t<1$) be a user’s belief about the interruption probability in time t on the cloud platform. Let $\varphi$ be a customer’s sensitivity to interruptions. We assume that a reduction in a customer’s willingness to pay is equal to the interruption risk multiplied by a customer’s sensitivity to interruptions [29]. Therefore, the willingness to pay for one spot instance per time unit in time t is given as follows:

$$w_t=v_0-\varphi \frac{{\zeta }_t}{1-{\zeta }_t}$$

(3)

When there is no interruption, $w_t$ = $v_0$. $\varphi \frac{{\zeta }_t}{1-{\zeta }_t}$ is the perceived interruption cost, which is the reduction in the willingness to pay for a spot instance due to service interruptions. Let ${\zeta }_0$ be the initial perceived interruption probability at the beginning of the time horizon, T, and we assume that it follows a beta distribution, ${\zeta }_0\sim Beta(a_0,b_0)$, where $a_0\ge 0$ and $b_0\ge 0$. With the number of service interruptions changing, the perceived interruption probability, ${\zeta }_t$, will be changed. We assume that this updating process is the Bayesian updating process. Let $n_{t,1}$ denote the number of running spot instances in time t. Let $n_{t,2}$ denote the number of interrupted spot instances in time t. The perceived interruption probability, ${\zeta }_t$, will be updated as follows: ${\zeta }_t\sim Beta(a_{t-1}+n_{t,1},b_{t-1}+n_{t,2})$. Therefore, according to the principle of a geometric distribution, a user can obtain a spot instance after the $\kappa$th request [29]. The expected delay time unit is equal to $\frac{{\zeta }_t}{1-{\zeta }_t}$, which is assumed to be equal to the user’s belief about the average delay [29].

2.2.3. Demand Model of Spot Cloud Services

Based on the above estimation of users’ willingness to pay, we can obtain the willingness to pay for each time period, t. Let $(w_t,m_t)$ denote one request submitted by a spot user in time t. $m_t$ denotes the number of requested spot instances. We assume that the number of virtual machines requested for one spot computing job is independent and identically distributed (i.i.d.) with a mean of ${\mu }_m$ and a standard deviation of ${\sigma }_m$.

Let $p_t$(t = 1, 2, …, T) represent the posted spot price at the beginning of time t. Let $W_t=(w_t^1,w_t^2,\dots ,w_t^i,w_t^{i+1},\dots ,w_t^T)$ denote the up-to-date vector of spot users’ willingness to pay in period t. Let $M_t$ = $(m_t^1,m_t^2,\dots ,m_t^i,m_t^{i+1},\dots ,m_t^T)$ denote the up-to-date vector of spot instances requested by spot users in period t. According to the spot pricing scheme, at the beginning of time t the decision maker checks the historical maximum price set, {$w_t^1,w_t^2,\dots ,w_t^{t-1}$}, and reclaims the spot instances whose willingness to pay, $w_t^i$, is lower than the current spot price, $p_t$. If there is a new spot request ($w_t$, $m_t$) in period t, then i = t, $w_t^i$ = $w_t$, and $m_t^i$ = $m_t$, or else, $w_t^i$ = 0, $m_t^i$ = 0. We construct the binary-valued $l_t^i$ as follows: $l_t^i$ = 1 indicates that a spot user can continue to use spot instances and $l_t^i$ = 0 indicates that the running spot instances are evicted, where i (i = 1, 2…, t − 1). Binary-valued $l_t^i$ is defined in Equation (4) by comparing historical bids, $w_{t-1}^i$($w_{t-1}^i\ne 0$), and the current spot price, $p_t$:

$$l_t^i=\{\begin{array}{ll}1,& w_{t-1}^i\le p_t\\ 0,& w_{t-1}^i>p_t\end{array}$$

(4)

According to Equation (4), the number of spot users whose bid exceeds the current price at the beginning of period t is equal to $n_t={\displaystyle {\sum }_{i=1}^{t-1}(1-l_t^i)}$. Among these users, $n_t$, there will be some users who give up using the service due to the completion of computing jobs and other reasons. Let $\rho$ denote the departure rate of one spot user in period t. We assume that the departure rate, $\rho$, for every spot user is homogeneous. Let $n_t^{quit}$ denote the number of spot users who give up using spot services in period t. This being the case, according to a binomial distribution, the probability of $n_t^q$ in period t is as follows:

$$Pr(n_t^q)=\left(\begin{array}{l}{n}_t^q\\ n_t\end{array}\right){\rho }^{n_t^q}{(1-\rho )}^{(n_t-n_t^q)}$$

(5)

Let $F(p_t)$ denote the probability that the willingness to pay of spot users is lower than the current spot price, $p_t$, and $1-F(p_t)$ denote the probability that the willingness to pay of spot users is higher than the current spot price, $p_t$. Let $d_t$ denote the total amount of running spot instances at the beginning of period t, $q_{t,1}$ denote the number of spot instances interrupted because of the dynamics of spot price, and $q_{t,2}$ denote the number of spot instances released because of the abandoning of the use of spot services. Thus, $q_{t,1}$ = ${\displaystyle {\sum }_{i=1}^{t-1}{l}_t^i}\times m_i$ and $q_{t,2}=n_t^{quit}{\mu }_m$. Let $\mathrm{Pr}(c_{t+1}<d_{t+1})$ denote the probability of spot demand exceeding excess capacity in period t. We assume that a spot user’s arrival follows a Poisson distribution with an arrival rate, $\delta$, in each period t (t = 1, 2, 3, …, T). According to whether there is a new spot request and if there is enough excess capacity to offer a spot price in period t, the dynamic update process of spot service demand has the following three possibilities.

(1) If there is one new spot service request in period t and there is enough excess capacity to offer spot services, the number of running spot instances changes from $d_t$ to $d_{t+1}=d_t-q_{t,1}-q_{t,2}+m_t$ with the following probability:

$$P{r}_1(d_{t+1}|d_t,p_t)=\delta [1-F(p_t)]\times Pr(n_t^{quit})\times Pr(c_{t+1}>d_{t+1})$$

(6)

(2) If there are no new spot service requests in period t and there is enough excess capacity to offer, the number of running spot instances changes from $d_t$ to $d_{t+1}=d_t-q_{t,1}-q_{t,2}$ with the following probability:

$$P{r}_2(d_{t+1}|d_t,p_t)=[\delta F(p_t)+(1-\lambda )]\times Pr(n_t^{quit})\times Pr(c_{t+1}>d_{t+1})$$

(7)

(3) If there is not enough excess capacity to offer spot service requests in period t, a cloud operator will interrupt spot instances to avoid the violation of regular services. The number of interrupted spot instances is equal to $d_{t+1}-c_{t+1}$; therefore, the number of running spot instances changes from $d_t$ to $d_{t+1}=c_{t+1}$ with the following probability:

$$P{r}_3(c_{t+1}|d_t,p_t)=1-P{r}_1(d_{t+1}|d_t,p_t)-P{r}_2(d_{t+1}|d_t,p_t)$$

(8)

For cases (1) and (2), there are $n_{t,1}$ = $d_{t+1}$ and $n_{t,2}$ = $q_{t,1}$; for case (3), there are $n_{t,1}$ = $c_{t+1}$ and $n_{t,2}$ = $c_{t+1}$-$d_{t+1}$. $n_{t,1}$ and $n_{t,2}$ are used to update the perceived interruption probability ${\zeta }_t$ in Section 2.2.2. Note that due to non-stationary resource utilization in fixed-price regular markets, the excess capacity, $c_{t+1}$, is stochastic non-stationary with time. Therefore, the probability, $Pr(c_{t+1}>d_{t+1})$, in Equations (6)–(8) is assumed to be unknown to cloud operators.

2.3. Non-Stationary Markov Decision Process Model for Spot Cloud Pricing

We formulated a real-time spot pricing process on a finite horizon, T, as a non-stationary Markov decision process (MDP), which is an extension of a standard MDP. We define the non-stationary MDP as $M$ = $<S_{\theta },P,{\mathsf{\Gamma }}_{\theta },R,\mathsf{\Theta }>$. $S_{\theta }$ is the finite set of states of a pattern, $\theta$. P is the finite set of spot pricing decisions. ${\mathsf{\Gamma }}_{\theta }$ is the conditional transition probability and $R$ is the immediate reward. We provide detailed definitions of them below:

Excess capacity pattern set, $\mathsf{\Theta }$: $\mathsf{\Theta }$ is the pattern sequence set in a finite horizon, T, where $\mathsf{\Theta }$ = {${\theta }_1$, …, ${\theta }_k$, ${\theta }_{k+1}$, …, ${\theta }_K$}. ${\theta }_k$ is the k-th pattern type. Let J be the number of pattern types in this pattern sequence set. ${\psi }_j$ is the mean parameter vector for each pattern type, j (j = 1, 2, …, J). Let $\mathsf{\Psi }=\{{\psi }_1,{\psi }_2,\dots ,{\psi }_J\}$ denote the parameter vector set. For example, there are two pattern types {1, 2} in one horizon, T, and pattern 1 appears two times. The pattern sequence set, $\mathsf{\Theta }$, is {${\theta }_1$, ${\theta }_2$, ${\theta }_3$} = {1, 2, 1}; the pattern type set, $\mathsf{\Psi }$, is $\{{\psi }_1,{\psi }_2\}$. For each ${\theta }_k$, the decision epoch at which the cloud environment pattern, ${\theta }_k$, changes is the changepoint $T_k$, and we denote the set of the changepoint by using the notation ${\left\{T_k\right\}}_{k\ge 1}$, which is an increasing sequence of random integers. The peak–trough point, $T_k$, is uncertain for cloud operators.

States set, S: S is the state set. $s_t({\theta }_k^j)\in S$, where $s_t({\theta }_k^j)$ = $\{{\theta }_k^j,c_t,d_t\}$. ${\theta }_k^j$ represents that the k-th pattern type is ${\theta }^j$. $c_t$ is the excess capacity (VMs) available to spot services in period t, which is not affected by the spot price, $p_t$. $d_t$ is the number of running spot instances at the beginning of period t, which mainly depends on the spot price, $p_t$. $s_t({\theta }_k^j)$ is the reference factor for spot pricing decisions in period t.

Decisions set, P: P = {$\underset{\_}{p},\dots ,\overline{p}$} is the set of all possible price decisions of one unit of spot services, such as one VM in one unit of time. $p_t\in P$ is the pricing decision in time step t. At the beginning of each decision epoch, t, the seller chooses a price, $p_t$ $\in$ [$\underset{\_}{p}$,$\overline{p}$], where $\underset{\_}{p}$ and $\overline{p}$ are the lowest and highest feasible prices, respectively, with 0 < $\underset{\_}{p}$ < $\overline{p}$ < $p$. Note that $\overline{p}$ is not higher than the price of regular cloud services, $p$.

Transition probability, $\mathsf{\Gamma }$:$\mathsf{\Gamma }={\left\{{\mathsf{\Gamma }}_t^{\theta }\right\}}_{t\in [T],\theta \in [\mathsf{\Theta }]}$ denotes the underlying transition probabilities from $s_t^{{\theta }_k}$ to $s_{t+1}^{\theta {}^{\prime }}$. Specifically, ${\mathsf{\Gamma }}_t^{\theta }$ = ${\mathsf{\Gamma }}_t(s_{t+1}^{\theta {}^{\prime }}|s_t^{{\theta }_k},p_t)$. The transition from one state, $s_t^{{\theta }_k}$ to the next state, $s_{t+1}^{\theta {}^{\prime }}$, not only depends on the spot price, $p_t$, but is also affected by the non-stationary characteristics of excess capacity. Since excess capacity is non-stationary, the transition probability, ${\mathsf{\Gamma }}_t(s_{t+1}^{\theta {}^{\prime }}|s_t^{{\theta }_k},p_t)$, is assumed to be unknown to cloud operators.

Immediate revenue, R: $R={\left\{R_t\right\}}_{t\in [T]}$ is the set of rewards at time step t. $R_t(s_t^{{\theta }_k},p_t)$ represents the immediate revenue gained for every state, $s_t^{{\theta }_k}$, and pricing action, $p_t$. Let $R(s_{t+1}^{\theta {}^{\prime }}|s_t^{{\theta }_k},p_t)$ represent the immediate revenue gained from $s_t^{{\theta }_k}$ to $s_{t+1}^{\theta {}^{\prime }}$ with pricing action, $p_t$. According to the demand model in Section 2.2, the immediate revenue of spot cloud services for one period, t, can be calculated as Equation (9):

$$R(s_{t+1}^{\theta {}^{\prime }}|s_t^{{\theta }_k},p_t)=\{\begin{array}{lll}{p}_t(d_t-q_{t,1}-q_{t,2}+m_t),& if& d_{t+1}=d_t-q_{t,1}-q_{t,2}+m_t\\ p_t(d_t-q_{t,1}-q_{t,2}),& if& d_{t+1}=d_t-q_{t,1}-q_{t,2}\\ p_t{c}_{t+1},& if& d_{t+1}=c_{t+1}\end{array}$$

(9)

where $q_{t,1}$ = ${\displaystyle {\sum }_{i=1}^{t-1}{l}_t^i}\times m_i$ and $q_{t,2}=n_t^{quit}{\mu }_m$, which are described in Section 2.2. If period t is the peak or trough point,$\theta {}^{\prime }={\theta }_{k+1}$; alternatively, $\theta {}^{\prime }={\theta }_k$.

Non-stationary policy, ${\pi }_{\theta }(p|s)$: ${\pi }_{\theta }(p_t|s_t)$ is a non-stationary policy that is a random conditional probability. More precisely, ${\pi }_{\theta }(p_t|s_t)$ may not be equal to ${\pi }_{{\theta }^{{}^{\prime }}}(p_t|s_t)$ for $\theta$$\ne$$\theta {}^{\prime }$, where $\theta$ and $\theta {}^{\prime }$ are two different pattern types. Therefore, a non-stationary policy can choose different actions in the same state. Furthermore, we assume that ${\pi }_{\theta }(p_t|s_t)$ is a stochastic policy, not deterministic, which can prevent users from detecting patterns of spot pricing based on historical dynamic pricing data.

If there is no partitioned pattern, a decision maker needs the history-dependent decision rule to obtain the optimal policy in a non-stationary environment. The non-stationary decision rule is $\pi (p_t|(s_0,s_1,s_2,\dots ,s_t))$, where $(s_0,s_1,s_2,\dots ,s_t)$ is the set of all possible histories observed up to period t. This decision rule, operating in this non-stationary environment, needs to search over the space of randomized history-dependent policies, which is an intractable problem. Therefore, our decision rule, ${\pi }_{\theta }(p_t|s_t)$, with a divided linear pattern, $\theta$, can greatly reduce the computing complexity.

Objective V: Define $V_{{\mathsf{\Psi }}_{\theta }}^{}(s_t^{{\theta }_k})$ to be the value function under policy ${\mathsf{\Psi }}_{\theta }$ at period t. Our objective for the real-time pricing problem is to learn an optimal policy, ${\mathsf{\Psi }}_{\theta }=\left\{{\pi }_{{\theta }^1}(p|s),{\pi }_{{\theta }^2}(p|s),\dots ,{\pi }_{{\theta }^J}(p|s)\right\}$, for every pattern, ${\theta }^j$, such that the long-run expected sum of discounted rewards is maximized throughout the selling horizon, T, which is expressed as Equation (10):

$$V_{{\mathsf{\Psi }}_{\theta }}^{*}(s_1^{{\theta }_k})=\underset{{\mathsf{\Psi }}_{\theta }}{max}E\left[{\displaystyle \sum _{t=1}^T{\gamma }^t}{R}_t(s_{t+1}^{\theta {}^{\prime }}|s_t^{{\theta }_k},p_t)|s_t^{{\theta }_k}\right]$$

(10)

${\rm E}[\cdot ]$ is the expectation operator, and γ $\in$ [0,1] is the discount factor that represents the relative importance of future revenue compared with present revenue.

3. Algorithm Design

We utilized a simulation-based reinforcement learning algorithm, Q-learning [30], to design an adaptive learning algorithm to solve our non-stationary decision model. Q-learning is a value-iteration-based RL method to approximately solve large-scale MDP problems, and it has been successfully applied to many industries, including air travel, the hospitality industry, and car rentals. It can directly learn from the environment without knowing a model of the environment (e.g., state transition probability), and has the ability to respond to dynamic environments through ongoing learning and adaption. Q-learning is guaranteed to converge to an optimal policy in stationary settings; however, directly using this method will yield suboptimal solutions in non-stationary environments. In this section, we present a new RL algorithm, named non-stationary Q-learning (NSQL), which can detect changing excess capacity adaptively and obtain an optimal pricing policy for each pattern, so as to maximize the long-term discounted reward criterion with non-stationary stochastic excess capacity.

3.1. Q-Learning Algorithm

Q-learning aims to find the best action for every given state in order to maximize the expected cumulative reward by interacting with the environment. A learned value function is used to estimate the quality of executing a pricing action, p, when the environment is in a state, s, defined by Q: S × P → R, that is:

$$Q(s,p)={\rm E}\left[{\displaystyle \sum _{x=0}^{T-t}{\gamma }^x{R}_{t+x}|S=s,P=p}\right]$$

(11)

$Q(s,p)$ denotes the discounted expected total reward when starting at a state–action pair, (s, p). A higher Q-value $Q(s,p)$ represents that an action, p, is judged to yield higher long-term revenue in a state, s. The optimal value functions of $Q(s,p)$ and $V(s)$ in Equation (10) are related by Equation (12):

$$V^{*}(s)=\underset{p}{max}{Q}^{*}(s,p)$$

(12)

The basic principle of Q-learning is to assign a Q-value, $Q(s,p)$, to each state–action pair, $(s,p)$, and update $Q(s,p)$ iteratively towards the observed reward plus the max Q-value over all actions, $p{}^{\prime }$, in the resulting state, $s{}^{\prime }$. δ > 0 is the learning rate, which represents how much the algorithm learns from new experiences. The following rule is then used to update the estimate of Q(s, p):

$$Q(s,p)\leftarrow Q(s,p)+\delta \left(R(s,p)+\gamma \underset{p{}^{\prime }}{max}Q(s{}^{\prime },p{}^{\prime })-Q(s,p)\right)$$

(13)

The optimal policy can be obtained as ${\pi }^{*}(s)=\underset{p}{argmax}{Q}^{*}(s,p)$ in a stationary environment. The learned optimal policies may be deterministic or stochastic, but they are all stationary. The agent has the same optimal policy in the same state, s, which is described as $\{{\pi }^{*}(s),{\pi }^{*}(s),\dots ,{\pi }^{*}(s)\}$; however, when the same state, s, is in two totally different scenarios, their spot pricing policies in the same state, s, may be different. Therefore, the learning process of the Q-value should be independent of different patterns, and the Q-value in the same pattern is updated. Therefore, in a non-stationary environment, on the basis of Equation (13), the updated principle of Q-learning is improved as Equation (14):

$$Q_{\theta }(s,p)\leftarrow Q_{\theta }(s,p)+\alpha \left(R(s,p)+\gamma \underset{p{}^{\prime }}{max}{Q}_{\theta {}^{\prime }}(s{}^{\prime },p{}^{\prime })-Q_{\theta }(s,p)\right)$$

(14)

where $\theta$ is the pattern type. The optimal policy is ${\pi }_{\theta }^{*}(s)=\underset{p}{argmax}{Q}_{\theta }^{*}(s,p)$, associated with the pattern, $\theta$. The optimal policy set in a finite horizon, T, is described as $\{{\pi }_{{\theta }^1}^{*}(s),{\pi }_{{\theta }^2}^{*}(s),\dots ,{\pi }_{{\theta }^j}^{*}(s),\dots ,{\pi }_{{\theta }^J}^{*}(s)\}$, where j is the pattern type. For each pattern, ${\theta }^j$, decision makers can obtain real-time data and responses from users by interacting with the actual business environment, after which they can update the Q-value as in Equation (13). However, a decision maker cannot recognize the pattern adaptively in a non-stationary environment. This being the case, in the following section, we design an online changepoint detection method to identify the locations of peak–trough points in non-stationary excess capacity.

3.2. Online Peak–Trough Point Detection in Excess Capacity

Peak–trough points in excess capacity refer to the decision epochs at which an excess capacity pattern model changes suddenly. Peak–trough point detection is carried out in order to discover sudden changes in patterns lying behind time series data. At present, due to the non-stationary nature of cloud computing workloads, there are already a large number of changepoint detection methods. However, most changepoint detections are offline detections. In comparison with offline changepoint detection problems, the distinguishing feature of our problem is the need to detect peak–trough points online. However, it is difficult to predict real peaks or troughs correctly in advance because of the non-stationarity of excess capacity. Therefore, the detection must be made based on the available nearest data, which will result in a certain time delay in the detected peak–trough points.

We detect the peak–trough points in the local pattern, ${\theta }_k$, using the last, $t_k$, historical observations, where k indexes the pattern sequence in the horizon, T. As described in Section 2.1, $t_k$($t_k=t-T_k$) indexes the time of the pattern, ${\theta }_k$. $T_k$ is the estimated peak or trough point from pattern ${\theta }_{k-1}$ to ${\theta }_k$. The decision maker estimates the next changepoint, $T_{k+1}$, according to the last, $t_k$, historical excess capacity in the pattern, ${\theta }_k$. Let D($t_k$) be the last historical excess capacity; that is, D($t_k$) = $\{c_1,c_2,\dots ,c_{t_k}\}$. Let i ($1\le i\le t_k$) index the number in D($t_k$). $c_i$ is the i-th element in D($t_k$).

Sample entropy is a robust volatility measure that was developed by Richman et al. over a decade ago [31]. Let $\mathrm{SampEn}(z,r,t_k)$ be the sample entropy of the latest, $t_k$, historical observations, which represents the volatility of the time series, $D(t_k)$. We arbitrarily intercept the sequence vector, $\{a_1,a_2,\dots ,a_{t_k-z-1}\}$, from the time series, $D(t_k)$; each sequence vector is $a_j=\{c_j,c_{j+1},\dots ,c_{j+z-1}\}$, where 1 ≤ j ≤ $t_k$ – z + 1. T{\displaystyle m}he sample entropy of the time series, $D(t_k)$, is then calculated as Equation (15):

$$\mathrm{SampEn}\left(z,r,t_k\right)=-log\left[\frac{A^z(r)}{A^{z+1}(r)}\right]$$

(15)

where k = 1, 2, …, K. A smaller value of $\mathrm{SampEn}(z,r,t_k)$ indicates more self-similarity or lower volatility in the time series, $D(t_k)$. $A^z(r)$ is computed by Equation (16):

$$A^z(r)=\frac{1}{t_k-z}{\displaystyle \sum _{i=1}^{t_k-z}\frac{1}{t_k-z-1}}{A}_j^z$$

(16)

$A_j^z={\displaystyle \sum _{\kappa =1}^{t_k-z}{X}_{\kappa }}$. If $\left|\right|a_j-a_{\kappa }\left|\right|\le r$, $X_{\kappa }$ = 1; alternatively, $X_{\kappa }$ = 0. $\left|\right|a_j-a_{\kappa }\left|\right|$ is the Euclidean distance between $a_j$ and $a_{\kappa }$. Let $N(t_k)$ be the whole data sequence, $D(t_k)$, and $N(i,t_k)$ be the data sequence, $D(t_k)$, without the i-th element. $B(c_i,t_k)$, presented in Equation (17), computes the difference in the sample entropy of the two sequences $N(t_k)$ and $N(i,t_k)$:

$$B(c_i,t_k)=\mathrm{SampEn}\left[z,r,N(t_k)\right]-\mathrm{SampEn}\left[z,r,N(i,t_k)\right]$$

(17)

We assume that $\eta$ is a threshold value. If $B(c_i,t_k)\ge \eta$, the time, $i+T_k$, over the horizon, T, is the changepoint of the next pattern, k + 1; that is, $T_{k+1}$ = $i+T_k$. When the estimated slope, ${\beta }_k$, in pattern ${\theta }_k$ is a positive value, that is, ${\beta }_k$ > 0, then $T_{k+1}$($T_{k+1}$ = $i+T_k$) is the peak point. When ${\beta }_k$ < 0, then $T_{k+1}$($T_{k+1}$ = $i+T_k$) is the trough point. The peak–trough point sequence set ${\mathsf{{\rm Y}}}_t$ = {$T_1$, $T_2$, …, $T_k$} at period t is updated as Equation (18):

$${\mathsf{{\rm Y}}}_t=\{\begin{array}{lll}\{T_2,T_2,\dots ,T_{k+1}\},& \mathrm{if}& B(c_i,t_k)\ge \eta \\ \{T_2,T_2,\dots ,T_k\},& & otherwise\end{array}$$

(18)

When one peak or trough point is detected at period $T_{k+1}$ ($T_{k+1}$ = $i+T_k$), we can determine the next linear pattern, ${\theta }_{k+1}$, using data from the period $t-$$T_{k+1}$ to the current time, t. ${\psi }_j$ (${\psi }_j$ = $[{\alpha }_j,{\beta }_j]$) is the mean value of the parameter vector of the pattern type, ${\theta }_j$. We can obtain the historical pattern set $\{{\theta }_1,{\theta }_2\dots ,{\theta }_j\}$in period (1, 2, …, t − 1), where j is the total number of pattern types in period (1, 2, …, t − 1). Calculate the Euclidean distance between the parameter vector, ${\psi }_{k+1}$, of the latest pattern, k + 1, and the historical pattern type, ${\psi }_{\varsigma }$($1\le \varsigma \le j$):

$$d_{k+1}=\inf \left\{\right||{\psi }_{k+1}-{\psi }_{\varsigma }||:1\le \varsigma \le j\}$$

(19)

where $\left|\right|\cdot \left|\right|$ denotes the Euclidean norm of a vector. When the Euclidean distance between the parameter vector of the latest pattern, ${\theta }_{k+1}$, and pattern type ${\theta }_{\varsigma }$ ($1\le \varsigma \le j$) is bigger than $\vartheta$, then the pattern type, ${\theta }_{k+1}$, in period t has a considerable difference from existing pattern types; the pattern type ${\theta }_{k+1}$ is a new pattern, named ${\theta }^{j+1}$. If the calculated Euclidean distance, $d_{k+1}$, is less than the constant $\vartheta$, the pattern type ${\theta }_{k+1}$ in period t belongs to the pattern type ${\theta }^{\varsigma }$($1\le \varsigma \le j$), which has the smallest Euclidean distance, $d_{k+1}$. The mean value of the parameter vector for the pattern type set is updated as Equation (20):

$${\mathsf{\Psi }}_t=\{\begin{array}{ll}\{{\psi }_1,{\psi }_2,\dots ,{\psi }_{j+1}\},& if d_{k+1}>\vartheta \\ \{{\psi }_1,{\psi }_2,\dots ,{\psi }_j\},& otherwise\end{array}$$

(20)

Note that real-time pricing policies in different linear patterns, $\theta$, can be completely different. This is another important reason why a linear model is used in this paper to model the type of pattern; that is, when excess capacity shows a growing trend, excess capacity becomes more and more abundant, and the spot service price needs to be reduced. When excess capacity shows a decreasing trend resource capacity will become more and more scarce, and the spot service price will need to be increased.

3.3. Non-Stationary Q-Learning Algorithm for Real-Time Spot Pricing

Based on online peak–trough point detection, a non-stationary Q-learning (NSQL) algorithm is introduced to solve the non-stationary real-time pricing problem. The most ideal state for spot real-time pricing is that there is no idle resource capacity in trough times and that there is no pre-emption in peak times. Faced with uncertain peak–trough and change patterns in non-stationary excess capacity, we needed to design a method for finding the optimal policy. Such a method should be sensitive to environmental changes. We designed non-stationary Q-learning (NSQL), which is a method that can handle learning tasks when excess capacity model information and pattern changes are all not known. The concept of NSQL is in one respect similar to context Q-learning [25]: both methods instantiate new models whenever a peak or trough is detected. However, unlike context Q-learning, which assumes that the pattern of model change(s) is known, in our problem the pattern sequence and pattern structure are all not known in advance. We use online peak–trough point detection, sample entropy, to acquire the burst point and then estimate the non-stationary pattern information. Furthermore, compared with non-stationary excess capacity, the demand for spot cloud services is relatively stable. Therefore, we assume that a decision maker can obtain the demand information. Non-stationary Q-learning (NSQL) simply updates the Q-values of each estimated excess capacity pattern type model under a stationary demand scenario. Additionally, if the method finds that the current excess capacity pattern type is similar to a previous pattern type, it updates the Q-values under the same pattern type. Thus, in this manner, the policy which was learnt and stored earlier (in the form of Q-values) is not lost, and can forget the irrelevant information and remember the relevant information.

The detailed steps of the NSQL algorithm are as follows (presented in Algorithm 1). In step 1, we input the non-stationary excess capacity time series with a finite horizon, NT, which is called dataset $C_{NT}$. The inputted dataset has J pattern types and the corresponding parameter vector for each pattern was ${\psi }_1$ $\to$ ${\psi }_2$ $\to$ … $\to$${\psi }_J$, ${\psi }_j$ representing the mean values of the linear parameter vectors of all excess capacities, c, in the pattern, ${\theta }^j$. The sequence pattern type set, $\mathsf{\Theta }$ = {${\theta }_1,{\theta }_2,\dots ,{\theta }_K$}, and the peak–trough point sequence set, $\mathsf{{\rm Y}}=\{T_1,T_2,\dots ,T_K\}$, are all not known to the decision maker in advance. The decision maker can only learn them with newly obtained data. The spot market environment, {$\lambda$, F(w), G(m)}, is inputted, which is assumed to be known to the decision maker. In step 2, the algorithm starts by initializing the Q-value, the pattern sequence set, $\mathsf{\Theta }$, and the peak–trough point set $\mathsf{{\rm Y}}$, et al. The loop from step 4 to step 11 is then repeated for each episode, n. Within this outer loop, step 4 initializes the state, $s_1^{{\theta }_k}\leftarrow \{{\theta }_k,c_1,d_1\}$, at the beginning of each horizon, T, in each episode, n. Steps 5–10 are repeated in every whole horizon, T. Within this inner loop, step 6 describes the interaction between the decision maker and the environment, and also obtains the immediate revenue. The spot demand and excess capacity are then updated to $d_{t+1}$ and $c_{t+1}$, respectively. Step 8 determines whether the excess capacity, $c_{t+1}$, in period t + 1 changes into a new pattern. If a peak or trough point occurs in period t + 1, the decision maker determines whether the pattern, ${\theta }_{t+1}$, is a new pattern and updates the pattern sequence, $\mathsf{\Theta }$, and $\mathsf{{\rm Y}}$. The Q-value for each state, $s_t^{{\theta }_k}=\{{\theta }_k,c_t,d_t\}$, is then updated. Algorithm 1 is used until the Q-value is converged for each pattern, such that |$Q_{\theta }^n(s_t,p_t)-Q_{\theta }^{n-1}(s_t,p_t)$|$\le$τ.

Algorithm 1: Non-Stationary Q-Learning (NSQL) Algorithm for Real-Time Spot Pricing

Step 1: Input: Non-stationary excess capacity time series $C_{NT}=\{c_1,c_2,\dots ,c_{NT}\}$. The length of the dataset, $C_{NT}$, is NT. Spot demand model: $\delta$, F(w), and G(m). Step 2: Initialization: Initialize the pattern set, $\mathsf{\Theta }$ $\leftarrow$ {${\theta }_k$}, and the peak–trough point sequence set, $\mathsf{{\rm Y}}$$\leftarrow${$T_k$}. $Q_{\theta }(s_t^{\theta },p_t)$$\leftarrow$0 for all (s; p; t; $\theta$) $\in$ S × P × [T] × [$\mathsf{\Theta }$]. Willingness to pay set, $W_1$ = {0, 0, …, 0}; requested VMs set, $M_1$ = {0, 0, …0}. Latest pattern, ${\theta }_k$. Latest changepoint, $T_k$$\leftarrow$$1$. $d_1$$\leftarrow$0. Initial excess capacity, $c_1$. k$\leftarrow$1. t$\leftarrow$1. Step 3: For episode n = 1 to N do: Step 4: Initialize the state, $s_t^{{\theta }_k}\leftarrow \{{\theta }_k,c_t,d_t\}$. Step 5: For t = 1 to T do:

Step 6:  Choose the spot price, $p_t$, based on the $\epsilon$-greedy when the state is $s_t^{{\theta }_k}$. Carry out the spot price, $p_t$, to obtain the immediate revenue, $R_t$, according to the revenue function (Equation (9)) and update to the next demand state, $d_{t+1}$, with the probability model according to the demand model (Equations (6)–(8)). Then, obtain the $c_{t+1}$ from the non-stationary dataset $C_{NT}$. Step 7:  Determine whether there is one changepoint in time interval $[T_k,t]$ according to Equations (15)–(18). If there is one changepoint, $T_{k+1}$, then determine the pattern type, ${\theta }_{k+1}$, according to Equations (19) and (20). Step 8:  If $\theta {}^{\prime }$ occurs the first time, then $Q(\theta {}^{\prime },s_t,p_t)$$\leftarrow$0 for all (s; p)$\in$S×P. Step 9:  Update the Q-value of the pattern, ${\theta }_k$, with the next pattern, $\theta {}^{\prime }$, in period t + 1: $$Q({\theta }_k,s_t^{},p_t)\leftarrow (1-\alpha )Q({\theta }_k,s_t^{},p_t)+\alpha (R(s_t^{},p_t)+\gamma \underset{p{}_{t+1}}{max}Q(\theta {}^{\prime },s_{t+1}^{},p_{t+1})).$$ Step 10:  Update $s_t^{{\theta }_k}\leftarrow \{\theta {}^{\prime },c_{t+1},d_{t+1}\}$; $W_t\leftarrow W_{t+1}$; $M_t\leftarrow M_{t+1}$; and k$\leftarrow$k + 1. Step 11: If $Q_{\theta }^{*}(s_t,p_t)$ is converged, such that |$Q_{\theta }^n(s_t,p_t)-Q_{\theta }^{n-1}(s_t,p_t)$| $\le$ τ, the algorithm stops and outputs $Q_{\theta }^{*}(s_t,p_t)$ for all (s; p; t; $\theta$) $\in$ S × P × [T] × [$\mathsf{\Theta }$]. Alternatively, set n $\leftarrow$ n + 1 and go to Step 3.

After discussing the non-stationary Q-learning algorithm in procedural form (presented in the algorithm), two important elements must be specifically introduced. (1) The exploration rate and the learning rate. In Algorithm 1, we use an $\epsilon$-greedy policy to derive the action with the highest Q(s, p) with a probability of 1-$\epsilon$($\epsilon$ $\in$ [0,1]) and then choose the other actions randomly with a probability of $\epsilon$. The agent is more likely to select the action at random when the $\epsilon$ is closer to 1, and the $\epsilon$ will be continuously reduced during the training process. (2) Remembering and forgetting. Through effective memory and forgetting, the influence of a non-stationary environment on the optimal policy can be well-avoided. In Algorithm 1, the data of the same pattern type are used to update the same Q-value table; that is, for one specific pattern, the historical data of the same pattern type are memorized and another pattern is forgotten, which is different from the solution for a stationary environment in which all historical data can be used for updating the same Q-value table.

4. Experiment Validation

In this section, we evaluate the effectiveness of our proposed NSQL approach on solving real-time spot pricing, after which we measure the impact of volatility of non-stationarity on optimal price and average revenue. All numerical experiments were carried out using the Python programming language.

4.1. Experiment Description and Performance Metric

In many cloud management systems, the observations of cloud computing resource utilization are represented as time series. These time series are generally non-stationary [32]. CPU utilization is the key significant metric for workload on virtual machines and it exhibits extremely non-stationary behavior [1]. Therefore, we use CPU workload traces to evaluate the effectiveness of our proposed approach. Two kinds of non-stationary datasets are introduced as input data: one is a simulated synthetic trace with a non-stationary workload structure, which is based on non-stationary structures summarized by literature [15]; another is a real-world non-stationary trace based on CPU utilization traces for fixed-price regular services from the Microsoft Azure Public Dataset [33]. Based on CPU utilization of regular virtual machines, we can obtain the CPU excess capacity for corresponding virtual machines. As described in Section 2.1, we characterize $c_t$ as the CPU excess resource capacity after satisfying regular virtual machines. Therefore, the relationship between the CPU usage of regular cloud service, $u_t$, and CPU excess capacity, $c_t$, is $c_t=N-u_t$, where $N$ is CPU resource capacity in total.

In our simulated environment, we set decision horizon T = 50, with five minutes for each period, t (t = 1,2, …, T), and prices are charged on a usage time basis (per minute), as discussed in Equation (6). The time decision epoch set is $\mathbb{Z}$ = {1, 2, ⋯, 50}. For the demand of spot cloud service, we assume that each time interval satisfies the Poisson distribution condition one spot user at a time with an arrival rate of $\delta$ = 0.3, the departure rate for every spot user being $\rho$ = 0.3, and the number of requested CPUs per computing job being N(10,1). We assume there is 1 CPU for each spot instance. The perceived value of 1 CPU for spot instance, $v_0$, is heterogenous because users adopt cloud computing for different purposes, and it follows a uniform distribution over the range $U(\underset{\_}{v},\overline{v})$. The interval of the perceived value of one unit spot instance $(\underset{\_}{v},\overline{v})$ is set to (0, 1). Let sensitivity to interruptions, $\varphi$, be 5. The parameter of initial perceived interruption risk, ($a_0$,$b_0$), is set to (0,1). The interval of the spot price in one period is set to (0,1]. If the spot price, $p_t$ = $\overline{p}$ = 1, the price of the spot service is equal to the price of the regular service, which means that no user is willing to adopt the spot service. If the demand for regular and spot services exceeds total resource capacity, the cloud operator will interrupt the insufficient resource from spot service (but not necessarily all).

To measure the effectiveness of our method, a performance metric is needed. With stationary environment assumptions, the RL algorithms are evaluated based on the cumulative reward (the sum of all of the rewards received in the finite horizon) that these learned policies yield. Therefore, for the non-stationary algorithms proposed, in a similar fashion, we compute the cumulative rewards garnered by the algorithms. The cumulative rewards for the evaluations of the algorithms include the following two stages: training and testing. The training phase is to learn the convergent Q-value table for each pattern. At each learning period, the price decision is obtained based on $\epsilon -greedy$, and the Q-value is updated according to Equation (14) until the converged Q-value table is obtained for each pattern. In the testing phase, the yielding power of the convergent Q-values is obtained in the test learning phase. Once our agent was trained, we tested his performance using an average revenue of 10,000 simulated selling horizons. Our convergence criterion is that the expected cumulative revenue becomes stable between two test phases and no longer increases.

4.2. Experiments on Synthetic Non-Stationary Traces

In this set of experiments, we consider synthetic non-stationary workload traces in which typical pattern changes occur alternately and are easy to detect. According to the literature [15], the workload of regular cloud service is mainly characterized by four basic non-stationary patterns: on-and-off, growing, periodic (e.g., sine curve), and unpredictable. Excess capacity is resource capacity after satisfying the workload of regular cloud services. Therefore, we simulate the non-stationary CPU excess capacity trace based on the above four basic non-stationary workload patterns.

We assume that there are 100 virtual machines in total and there is 1 CPU per virtual machine. $c_t$ denotes the CPU excess capacity available to spot services in period t. Subsequently, $c_t=N-u_t$, where $u_t$ is the CPU usage of regular cloud services in period t. According to the non-stationary patterns of the CPU usage for regular cloud services, we simulate the non-stationary patterns of time-varying CPU excess capacity, $c_t$, with Gaussian noise, which are shown in Figure 1a–d. Figure 1a shows an on-and-off structure which includes two linear patterns. One is simulated by $c_t$ = 20 + $\epsilon$, where t $\in$ {1, 2, …, 20} and t $\in$ {36, 37, …, 50}; the other is simulated by $c_t$ = 70 + $\epsilon$, where t $\in$ {21, 22, …, 35}. There are two changepoints in this on-and-off pattern, t = 21 and t = 36. Figure 1b shows a decreasing structure, including three linear patterns, which is the combination of $c_t$ = 70 + $\epsilon$, where t $\in$ {1, 2, …, 14}, $c_t$ = 70 − 2$\times$ t + $\epsilon$, where t $\in$ {15, 16, …, 35}, and $c_t$ = 10 + $\epsilon$, where t $\in$ {36,37,…,50}. There is no changepoint in this pattern. Figure 1c shows a periodic structure, including three linear patterns, which is simulated by a sine curve characterized as $c_t$ = 35 $\times$ (sin(($\pi$/25) $\times$ t) + 40, where t $\in$ {1, 2, …, 50}. There are two changepoints in this pattern. Figure 1d shows an unpredictable structure, including four linear patterns, which is the stochastic combination of the above non-stationary structures. The Gaussian noise term, $\epsilon$, follows $N(0,5^2)$. There are three changepoints in this pattern. Note that the linear pattern denotes the pattern detected by our changepoint detection method.

Note that the simulated time series of non-stationary excess capacity is unknown to the decision maker in our experiment validation and is only used as a simulation environment. For example, excess capacity in period t will be generated according to the simulated time series after the spot service price in period t is made. We compare the learning efficiency of the proposed algorithm (NSQL) with classical QL, repeated update QL (RUQL) [23], Q(λ) [26], FWQL [24], and context QL [25] in four simulated scenarios in Figure 1a–d. Classical QL does not deal with non-stationarity. Q(λ) uses eligibility traces, and FWQL restarts the Q-value according to the fixed-window size, so as to solve the non-stationary problem. RUQL [23] is similar to QL, but its exploration strategy and step size schedule differ from QL. Other details of these RUQL, FWQL, and Q(λ) algorithms can be found in the literature [23,24,26]. The above four QL algorithms do not detect changepoints, unlike our proposed NSQL (as described in Algorithm 1). Context QL, as described in the literature [25], has multivariate changepoint detection for the experience trace, different from our method, which just detects non-stationary univariate data. For the algorithms that involve hyperparameters (such as the learning rate, $\delta$, discount factor, $\gamma$, exploration rate, $\epsilon$, and fixed-window size of FWQL), we performed a grid search to choose the best value that maximizes the total revenue with $\delta$ = 0.3, $\gamma$ = 0.99, and $\epsilon$ = 0.3. As shown in Figure 2, each point represents the average revenue of 10,000 simulated selling horizons, T, in which the spot prices are made by the Q-value table learned from a different number of learning cycles.

From Figure 2a–d, all of the methods designed for non-stationary excess capacity outperform classical QL, Q(λ), and RUQL, which do not explicitly model the changes in excess capacity over time from the convergence and average revenue. For example, as shown in Figure 2a, the learning process of our NSQL algorithm becomes stable at around 80,000 learning cycles, but the classic Q-learning is not convergent, even after 90,000 learning cycles. For classical Q-learning, the reason for its slower convergence is that there is a need to search the large space of non-stationary policies for learning the spot pricing policy in non-stationary environments. Therefore, an exhaustive search over this large space is computationally expensive and requires a higher number of iterations to obtain higher revenue. Different from classical QL, Q(λ), and RUQL, our NSQL algorithm can find an approximately optimal policy over a smaller, restricted set, which narrows the search space, resulting in higher revenue levels with faster convergence. Furthermore, from the results shown in Figure 2a–d, it is clear that NSQL significantly outperforms the other non-stationary algorithms, which are FWQL and context QL. For the context QL method, although both algorithms detect changepoints of non-stationary environments, context QL detects pattern changepoints based on the experience data generated during interactions with the environments. For non-stationary environments with noise, the experience data fluctuate greatly, and find it difficult to be accurate in identifying the points of change. This makes the detection of changepoints through experience data less effective than the detection of a non-stationary univariate.

4.3. Experiments on Real-World Non-Stationary Traces

In this set of experiments, we employed a widely used workload dataset from the Microsoft Azure Compute Cluster [33], which represents the CPU utilization of two million regular VMs. By deducting the CPU utilization from the total amount of virtual computing resources, we obtain the CPU excess resource capacity. In this dataset, the average percent CPU utilization per VM is reported every five minutes. We examined 2045 of them and identified a VM cluster that includes 125 VMs. We displayed a set of 100 candidate VM traces of them over nearly 48 h. We assume 1 CPU for each VM. So, CPU utilization can be seen as CPU consumption. We sum the CPU utilization of 100 VMs and obtain the total CPU excess capacity of the 100 VMs, which is displayed in Figure 3. As is shown in Figure 3, the CPU excess capacity of the VM cluster fluctuates with time, and the fluctuation has certain regularity. That is, the mean value of the CPU excess capacity fluctuates and changes suddenly to another value, which shows extreme non-stationary behavior. We set the interval of each time period t to be 5 min, and there are 500 time periods in the whole time horizon T.

As is shown in Figure 3, the X-axis shows the time periods (five minutes per period) over 48 h, and the Y-axis shows the CPU excess capacity of the VM cluster. From the non-stationary characteristics, we can infer that it is likely that the VM cluster is hosting a restaurant service. When the time periods are between t = 1 and t = 90, the CPU excess capacity is at a high level, indicating that it is not peak dining time and that there are few diners. Thus, there is little workload on virtual machines in this time period. We can infer that this time period is at midnight. When the time periods are between t = 91 and t = 114, the CPU excess capacity is at a low level, which represents an increase in the number of diners, and there is a heavy workload on VMs during this time period. We can infer that these time periods are breakfast time. By analogy, we can explain the meaning of peaks and troughs in other time periods. In summary, these abrupt peak–trough changes represent transitions in the workload on VMs. Notice that the non-stationarity has visible and distinct sudden changes in its CPU excess capacity and has a time-dependent mean, which gives strong evidence for utilizing changepoint detection.

Due to the strong volatility of time-varying excess resources in the real-world dataset, the collected revenue has a high fluctuation. Therefore, we use the mean value, standard deviation (SD), and median of the collected revenue to describe the performances of the algorithms. Note that the real-world time series of non-stationary excess capacity is unknown to the decision maker and is only used as a generation for CPU excess capacity. So, the changepoint detection method is needed for the identification of pattern changepoints. Other parameters in our algorithm have the same settings as the experiment in Section 4.2. The performances of classical QL, Q(λ), RUQL, FWQL, context QL, and NSQL (our method) trained by 80,000 learning cycles are as follows.

From

Table 1. Rewards collected by different methods based on the real-world dataset.

Method	Mean ± SD	Median
NSQL (our method)	4074.12 ± 55.15	4083.06
Context QL	3850 ± 103.13	3865.12
FWQL	3656 ± 125.1	3660.89
RUQL	3380.23 ± 88.99	3401.31
Q(λ)	3351.14 ± 69.1	3353.67
QL	3039 ± 71.3	3042.19

, the non-stationary algorithms, FWQL, context QL, and NSQL outperform the other algorithms without considering non-stationary changes. Additionally, we can see that the average revenue under NSQL (our method) and context QL with changepoint detection outperforms FWQL without a changepoint detection method. Compared to context QL, NSQL not only has a higher mean value of revenue but also has strong stability in terms of revenue level. The reason is that changepoint detection for non-stationary excess capacity is much better than changepoint detection for the experience tuple of the reinforcement learning process.

4.4. Comparison with Different Volatilities of Non-Stationary Excess Capacity

In this section, we further examine the effect of the volatility of non-stationary excess capacity using our method (NSQL). As are shown in Figure 4, two simulated non-stationary scenarios ((a) and (b)) are introduced, which have the same amount of excess capacity with different volatilities. Scenario (a): Peak-to-trough fluctuation with low volatility, which is shown in Figure 4a. The first pattern is simulated by $c_t$ = 45 + 2 $\times$ t + ${\epsilon }_1$, where t $\in$ {1, 2, …, 15}; the second pattern is simulated by $c_t$ = 60 − (t − 15) $\times$ 2 + ${\epsilon }_2$, where t $\in$ {16, 17, …, 30}; and the third pattern is simulated by $c_t$ = 33 + (t − 30) $\times$ 2 + ${\epsilon }_3$, where t $\in$ {31, 32, …, 50}. The Gaussian noise terms, ${\epsilon }_1$, ${\epsilon }_2$, and ${\epsilon }_3$, follow $N(0,5)$. Scenario (b): Peak-to-trough fluctuation with high volatility, which is shown in Figure 4b. The first pattern is in a continuous growth simulated by $c_t$ = 15 + 5 $\times$ t + ${\epsilon }_1$, where t $\in$ {1, 2, …, 15}; the second pattern is a continuous decreasing simulated by $c_t$ = 85 − (t − 15) $\times$ 5 + ${\epsilon }_2$, where t $\in$ {16, 17, …, 30}; and the third is in a continuous growth simulated by $c_t$ = 15 + (t − 30) $\times$ 4 + ${\epsilon }_3$, where t $\in$ {31, 32, …, 50}. The Gaussian noise terms, ${\epsilon }_1$, ${\epsilon }_2$, and ${\epsilon }_3$, follow $N(0,5^2)$.

Once again, the simulated time series of non-stationary excess capacity is unknown to the decision maker and is only used as a simulation environment. For example, excess capacity in period t will be generated according to the simulated time series after the spot service price in period t is made. Figure 5 presents the optimal spot price, total demand, and cumulative revenue within one horizon, T, in two different scenarios.

Figure 5a shows that the higher the volatility of non-stationary excess capacity, the more frequently the price of spot cloud services changes. In addition, we can observe that the optimal price decreases with the excess capacity increasing, and the optimal price increases with the excess capacity decreasing. The reason is that lower prices can increase revenue when the excess capacity is plentiful and higher prices can reduce service interruption risk when the excess capacity is scarce. In a highly volatile scenario, such frequent price changes have better performance in peak shaving and valley filling (presented in Figure 5b). When excess capacity in the operator is at a high level, a lower spot service price is adopted; when excess capacity is low, a higher spot price is adopted. Therefore, the spot pricing learned by our method can adapt to the dynamic volatility of excess capacity without interrupting spot service resources. Figure 5c shows that the higher the volatility of non-stationary excess capacity, the lower the cumulative revenue of spot services. Therefore, in order to maximize the total revenue of cloud resources, operators can reduce fluctuations in excess capacity by reasonably scheduling cloud computing workloads of on-demand and reserved services. For example, in peak times of resource demand, computing workloads with a strong time sensitivity can be served first, and computing requests with a low time sensitivity can be served in the trough period of computing workloads, so as to minimize the volatility of excess capacity, thereby maximizing the total cloud revenue.

5. Conclusions

In this paper, a non-stationary real-time pricing method for spot cloud services with non-stationary excess capacity is presented. Due to the non-stationarity of excess capacity, spot cloud services with pre-emption using real-time pricing were implemented to maximize cloud revenue. Focusing on the spot market segment in which users can tolerate service interruptions, a non-stationary MDP model was formulated. In this model, non-stationary excess capacity is partitioned into the piecewise linear pattern. The non-stationary MDP model fully embodies spot demand dynamic updating with service interruptions, considering non-stationary excess capacity. To solve the stochastic non-stationary problem with incomplete non-stationary excess capacity (without transition probabilities), Q-learning, the most popular and classical reinforcement learning algorithm, was adopted, and a new non-stationary Q-learning (NSQL) algorithm with burst pattern changepoint detection is proposed. A change detection method was used to determine changes in excess capacity and determine the pattern model in conjunction with Q-learning. With the aid of real-time interactions, the proposed Q-learning algorithm adaptively decides real-time spot prices during the online learning process. The simulation results show that the proposed method and algorithm perform well in varying non-stationary excess capacity conditions and give better returns when compared to classical Q-learning and other non-stationary learning algorithms, such as FWQL and context QL.

Future extensions to this work can focus on improving the accuracy of peak–trough detection in the context of large and continuous non-stationary excess capacity models. Such extensions may be useful in recognizing pattern changes in order to better adapt to extremely non-stationary environments.