Complementary in Time and Space: Optimization on Cost and Performance with Multiple Resources Usage by Server Consolidation in Cloud Data Center

Abstract

The recent COVID-19 pandemic has accelerated the use of cloud computing. The surge in the number of users presents cloud service providers with severe challenges in managing computing resources. Guaranteeing the QoS of multiple users while reducing the operating cost of the cloud data center (CDC) is a major problem that needs to be solved urgently. To solve this problem, this paper establishes a cost model based on multiple computing resources in CDC, which comprehensively considers the hosts’ energy cost, virtual machine (VM) migration cost, and SLAV penalty cost. To minimize this cost, we design the following solution. We employ a convolutional autoencoder-based filter to preprocess the VM historical workload and use an attention-based RNN method to predict the computing resource usage of the VMs in future periods. Based on the predicted results, we trigger VM migration before the host enters an overloaded state to reduce the occurrence of SLAV. A heuristic algorithm based on the complementary use of multiple resources in space and time is proposed to solve the placement problem. Simulations driven by the VM real workload dataset validate the effectiveness of our proposed method. Compared with the existing methods, our proposed method reduces the energy consumption of the hosts and SLAV and reduces the total cost by 26.1~39.3%.

1. Introduction

The worldwide pandemic of the COVID-19 virus has forced people to change the way they live, socialize, and work. To maintain social distance, more human activities than ever have migrated online. In December 2021, Kamouri et al. [1] estimated that approximately 56% of the U.S. workforce could move to online or remote work. McKinsey Consumer Pulse [2] demonstrates that the shares of e-commerce in the major countries increased at two to five times the rate before COVID-19. Compared to the pre-lockdown levels, the usage of Internet services increased from 40% to 100% [3]. The usage of video-conferencing services, such as Zoom, has increased ten times [4]. In addition, to contain the pandemic, massive amounts of residential and medical data were sent to data centers for analysis. For example, China uses health QR codes nationwide to monitor and manage population movements with big data. Behind these large-scale digital activities, the support of cloud computing data centers (CDCs) is inseparable. To meet the corresponding computing power needs, people have now embraced cloud computing in all aspects.

With the explosive increase in the use of cloud computing, cloud service providers (CSPs) are also facing severe challenges in the management of CDCs. The problem mainly comes from two aspects: the cost of maintaining the operation of the CDC and the guarantee of Quality of Service (QoS) for users. With the surge in the number of users, to maintain the business needs of existing users and satisfy new users’ experience of cloud services, CSPs must increase investment in their CDCs. For example, SaaS service provider Salesforce increased the cost of its data center by 43% in 2022, which is USD 284 million, to meet higher customer service requirements [5]. Such a large financial investment is a great economic burden for the operation of an enterprise. Therefore, a feasible way to alleviate the economic pressure is to optimize the management of the CDC and improve the utilization of various computing resources in the CDC. The problem, however, brought about when resource utilization is improved is that the competition for different computing resources among the virtualized computing devices of users is enhanced. When the competition on the host with limited resources is intense to a certain extent, it will inevitably lead to a situation in which the user’s QoS drops significantly. It can be said that there is a certain contradiction between reducing the operating cost of the CDC and ensuring the QoS of users. We need to design a feasible way to achieve the tradeoff between the two.

Users run a wide variety of services in the CDC, and the corresponding virtual machine (VM) workloads are constantly changing due to the dynamic changes in services over time. During peak business hours, VMs have increased demands on computing resources such as CPU, memory, disk, and network. During periods of slack in business, the demand for these computing resources by VMs is also greatly reduced. To adapt to the dynamic changes of such workloads, the commonly used solution for managing hosts and VMs in CDCs is to perform server consolidation periodically [6]. By leveraging VM live migration, server consolidation migrates some VMs on hosts with high resource usage to other hosts with low resource usage to achieve load balancing in the CDC as much as possible. On the other hand, SLA is used to quantitatively describe the QoS for the use of VMs [7]. Hence, to ensure QoS, SLA needs to be taken into account to a certain extent when performing server consolidation.

In general, a round of server consolidation is divided into three sub-steps to implement. The first is host workload detection, which picks out overloaded and underloaded hosts in the cluster. Next is VM selection for overloaded hosts. To achieve the goal of reducing the workload on these hosts and the occurrence of SLAV, suitable VMs need to be selected for migration. Finally, the VM placement method is implemented to map an appropriate destination host for each VM to be migrated. VM placement problems often have specific optimization goals, such as using the smallest number of hosts to mount a given number of VMs. After implementing VM placement, underloaded hosts need to be addressed. By migrating all VMs on the underloaded host to other suitable hosts as much as possible, and shutting down or switching these underloaded hosts to an energy-saving state, the host energy cost of the CDC can be further reduced. The operating cost of the CDC involves many aspects, including host energy consumption, VM migration cost in server consolidation, SLAV penalty, cooling, security, storage, and network bandwidth overhead. The parts directly related to the computing resources used by users, including CPU, memory, disk, and network, are host energy consumption, VM migration overhead, and SLAV penalty. Therefore, in this paper, we comprehensively consider the operating cost of a CDC based on multiple computing resources, establish a relevant cost model, and propose a corresponding server consolidation solution. Our contributions are as follows:

(A)

We formally identify the multi-resource-based cost model for server consolidation, which involves host energy consumption, VM migration, and SLAV. Based on the cost model, the optimization problem is given.

(B)

A convolutional auto-encoder-based filter is leveraged to denoise the VM workload trace. Then, we propose an attention-based RNN method to predict the future workloads of the VMs. Based on the prediction results, a host workload detection policy is proposed.

(C)

To minimize the total cost of server consolidation, we propose a VM selection policy and a VM placement algorithm which consider the multi-resource demands of VMs in the present and future.

(D)

We conduct simulations to evaluate the performance of our proposed solution ARP-TSCP. The simulations’ results indicate that ARP-TSCP can reduce host energy consumption by 18.5~30.3%, SLAV cost by 38~52%, and total cost by 26.1~39.3% as compared to the baseline methods.

The remainder of the paper is organized as follows. In Section 2, we review the related work. In Section 3, we formally propose the cost model and then define the optimization problem to be solved. In Section 4, we propose the heuristic algorithms to solve this problem. In Section 5, we evaluate the performance of the proposed method by real VM workload, trace-driven simulations. In Section 6, we conclude the paper and discuss future works.

2. Related Work

In this section, we first survey the energy consumption models and cost models in cloud server consolidation, then we review the server consolidation solutions.

2.1. Sever Consolidation Cost Models

The cost of server consolidation in the cloud is mainly related to the host energy consumption, VM migration, and SLAV.

CPU is one of the most important components in a host, so many previous works [8,9,10,11,12,13,14,15,16,17] have proposed the host energy models based only on CPU usage or processor performance. However, the composition of a host’s energy consumption is not only related to the CPU factor. Therefore, several works [18,19,20,21,22,23,24] have established the host energy consumption models based on the usage of multiple resources, including CPU, memory, disk, and network. These works are of great help to us in building the proposed cost model of this paper. However, these models only consider the energy consumption when a host is running independently and do not consider the additional energy consumption caused by VM migration in server consolidation.

When addressing server consolidation, costs of VM live migration are primarily considered to be related to CPU [6]. Maziku et al. [25] and Dargie et al. [26] respectively pointed out that the duration of VM migration is related to the memory size of the VM being migrated and the network bandwidth of the host. Therefore, in addition to the CPU, other resources used by the VM migration process should also be included in the cost calculation.

Buyya et al. [6] proposed a CPU-based SLAV calculation method, which was widely adopted by many subsequent works [27,28,29,30,31,32,33,34,35]. However, the QoS of using VMs cannot be measured merely by CPU performance, and SLAV must involve the use of multiple resources. Guan et al. [36] proposed SLAV for GPUs for cloud gaming. Some SLAs [37,38] related to network factors have been proposed, but these models are not entirely built in the context of CDC.

2.2. Server Consolidation Solutions

Multiple solutions have been proposed to handle server consolidation.

Lin et al. [24] proposed a host energy consumption model based on CPU, memory, and disk and a centralized server consolidation structure called DEM. The master node is aware of the energy consumption of slave nodes that host VMs to trigger server consolidation. However, this work does not design a VM placement algorithm and does not consider VM migration and SLAV. Li et al. [39] proposed a server consolidation method based on multi-resource constraints. The goal of this method is to reduce energy consumption while ensuring user QoS. This work considers multi-resource constraints in the host workload detection and the design of the VM placement algorithm. However, in terms of the host energy consumption model and SLAV, only the CPU factor is considered. YADAV et al. [31] established a server consolidation model based on CPU usage, aiming to reduce energy consumption and SLAV at the same time. They proposed an adaptive host overloaded detection method and a bandwidth-aware-based VM selection strategy. Naeen et al. [40] proposed a stochastic process-based server consolidation policy to minimize data center costs while satisfying QoS requirements. The proposed energy consumption model, SLAV, and heuristic VM algorithm are based on CPU usage. They also take into account the cost of the host switching between different states. Sayadnavard et al. [41] proposed a server consolidation method based on multi-resource constraints. The goal is to minimize the number of hosts used by VM placement. When selecting a destination host for each VM to be migrated, the Markov chain model is used to determine whether that host would be overloaded soon. Yuan et al. [42] proposed to use the culture multiple-ant-colony algorithm to solve the server consolidation problem, thereby reducing the energy consumption of the hosts. The energy consumption model is based on CPU and does not specifically consider SLAV factors. Mamun et al. [43] proposed a server consolidation problem in the CDC with wireless network structures. They proposed an energy consumption model consisting of CPU-oriented host energy consumption and energy consumption of various network devices in the network topology. A network-ware VM placement method is proposed to reduce energy consumption. However, they did not consider SLAV.

3. Cost Model and Problem Description

In this section, we first formally describe the multiple computational resources based cost model in CDC, and based on this, we describe the problem definition.

3.1. Cost Model

In CDC, the cost related to computational resources mainly involves hosts, VM migrations, and SLAV penalties.

Before giving a specific cost model, we first describe the time and objects of the entire system. There are N heterogeneous hosts in CDCs, forming the host set $H=\{h_1,h_2,\cdots ,h_N\}$. The total resources provided by a host $h_i$ is denoted as $C_i=(c_i^{cpu},c_i^{mem},c_i^{disk},c_i^{net})$, where $c_i^{cpu}$, $c_i^{mem}$, $c_i^{disk}$, and $c_i^{net}$ are the total available usage of CPU, memory, Disk R/W, and network I/O throughput, respectively. There are M running VMs, forming VM set $V=\{v_1,v_2,\cdots ,v_M\}$. When a user creates a VM $v_j$, the submitted resource requirements are denoted as $D_j=(d_j^{cpu},d_j^{mem},d_j^{disk},d_j^{net})$, where $d_j^{cpu}$, $d_j^{mem}$, $d_j^{disk}$, and $d_j^{net}$ are the total requirements of CPU, memory, Disk R/W, and network I/O throughput, respectively.

The life cycle of the CDC is divided into L small and equally long consecutive time segment $\{t_1,t_2,\cdots ,t_L\}$, and each time segment has a length of T. In a certain time segment $t_k$, if host $h_i$ is in the working state, then we have ${\lambda }_{i,k}=1$; otherwise, ${\lambda }_{i,k}=0$. The amount of resources that $h_i$ can provide are denoted as $R_{i,k}=(r_{i,k}^{cpu},r_{i,k}^{mem},r_{i,k}^{disk},r_{i,k}^{net})$. In $t_k$, the resources required by VM $v_j$ are denoted as $S_{j,k}=(s_{j,k}^{cpu},s_{j,k}^{mem},s_{j,k}^{disk},s_{j,k}^{net})$.

We summarize the total cost of CDC for a given lifetime by analyzing the performance of each computing device in each time segment. In addition to the energy consumption cost of the hosts, it is also necessary to consider the cost of VM migration during each server consolidation and the penalty caused by the occurrence of SLAV. We will discuss them separately in the following subsections.

3.1.1. Cost of Host Energy Consumption

Given a host $h_i$, its cost $C_{h_i}$ during operation is mainly related to the electricity fee $EP$ and its power $p_{h_i}$, which is:

$$C_{h_i}=\sum _{k=1}^L(EP\times p_{h_i}\times T\times {\lambda }_{i,k}).$$

(1)

It should be noted that if $h_i$ is powered off or in a power-saving state, its power consumption is negligible, so it will not incur any electricity-related costs.

In CDCs, in order to meet the various business demands of a large number of different users, almost all hosts are high-power and high-performance components. Therefore, it is impractical to model host power consumption only by analyzing the CPU. Basmadjian et al. [44] calculated that the average energy consumption of CPU in the host is about 37%. It should be noted that when users create or apply for a VM in a CDC, they need to directly specify the relevant parameters of the CPU, memory, disk, and network devices. Here, we can think that the host power consumption is related to the four resources: CPU, memory, disk, and network interface card (NIC). Therefore, in this paper, we build the power consumption model of the host based on these four resources:

• CPU power model

Buyya et al. [6] leveraged a linear CPU power consumption model, that is, the current energy consumption of the CPU is directly related to its usage rate. When the CPU usage increases, the power of the host increases linearly. Although the linear model is simple to calculate, the result is lower than the actual power consumption value and the accuracy is not enough [24]. Therefore, we adopt an exponential model to fit the power consumption of the CPU.

Under normal circumstances, even if the working host is in idle state (the CPU usage rate is 0), the CPU will also generate a certain amount of energy consumption. This part of the energy consumption needs to be considered. Hence, the CPU power $p_{i,k}^{cpu}$ of host $h_i$ in $t_k$ can be described as:

$$p_{i,k}^{cpu}=p_{idle,i}^{cpu}+(p_{peak,i}-p_{idle,i})\times {\left(u_{i,k}^{cpu}\right)}^{{\alpha }_i^{cpu}},$$

(2)

where $p_{idle,i}^{cpu}$ is the CPU power of $h_i$ in idle state, $p_{peak,i}$ is the CPU power of $h_i$ when it is running at full load, $u_{i,k}^{cpu}$ is the usage rate of the CPU at any time in $t_k$, and ${\alpha }_i^{cpu}$ is the fit index. Here, we assume that the CPU usage at any time in $t_k$ is equal and is $u_{i,k}^{cpu}$. For a given CPU model, the powers in idle state and at full load are fixed values. According to [45], the value of ${\alpha }_i^{cpu}$ varies with the CPU usage. However, in general, the CPU power can be described more accurately when ${\alpha }_i^{cpu}=0.75$ [45].

• Memory power model

The power consumption of memory is related to its reading and writing. Accurately monitoring the read and write speed of memory in real time is too expensive. An alternative low-overhead solution is to estimate the throughput of memory through last level cache (LLC) miss counter available in processors. However, this approach is still impractical because the specific instructions associated with LLC misses are different in different CPU models [22].

In fact, in all public host or VM workload trace datasets, the provided memory-related data are the memory usage in certain periods of time. Therefore, we use the current footprint of memory (current usage of memory) $u_{i,k}^{mem}$ to estimate the power $p_{i,k}^{mem}$ of host $h_i$:

$$p_{i,k}^{mem}=p_{idle,i}^{mem}+{\alpha }_i^{mem}\times u_{i,k}^{mem},$$

(3)

where $p_{idle,i}^{mem}$ is memory power of $h_i$ in idle state, ${\alpha }_i^{cpu}$ is the fit index, and $u_{i,k}^{mem}$ is the usage rate of memory at any time in $t_k$. Here, we assume that the memory usage at any time in $t_k$ is equal and is $u_{i,k}^{mem}$. According to statistics, for a DDR memory system, when ${\alpha }_i^{mem}=0.3$ W/GB, the memory power can be estimated more accurately [46].

• Disk energy consumption model

The power consumption of disk $p_{i,k}^{disk}$ is directly related to its read and write throughput, we have:

$$p_{i,k}^{disk}={\alpha }_i^r\times u_{i,k}^r+{\alpha }_i^w\times u_{i,k}^w+p_{idle,i}^{disk},$$

(4)

where $u_{i,k}^r$ and $u_{i,k}^w$ are the disk read and write number of bytes of $h_i$ in any time in $t_k$, respectively; ${\alpha }_i^r$ and ${\alpha }_i^w$ are the disk read and write fit index, respectively; and $p_{idle,i}^{disk}$ is the disk power in idle state. Here, we assume that the disk read and write number of bytes at any time in $t_k$ are equal. Kansal et al. [22] discovered that the difference between disk read power and write power is negligible. Hence, Equation (4) can be updated as:

$$p_{i,k}^{disk}=p_{idle,i}^{disk}+{\alpha }_i^{disk}\times u_{i,k}^{disk},$$

(5)

where $u_{i,k}^{disk}=u_{i,k}^r+u_{i,k}^w$, and ${\alpha }_i^{disk}$ is the fit index.

In the actual operation of host, the power consumption of the disk is related to random read and write and sequential read and write. Different read and write modes generate different read and write power consumption. Hence, ${\alpha }_i^{disk}$ is divided into ${\alpha }_i^{seq}$ (sequential read and write) and ${\alpha }_i^{rnd}$ (random read and write). In practice, it is impractical to monitor and detect the read/write mode of the disk. Therefore, for convenience, we can assume that the probability of sequential reads and writes caused by all users is equal to that of random reads and writes. We obtain:

$${\alpha }_i^{disk}=({\alpha }_i^{seq}+{\alpha }_i^{rnd})/2.$$

(6)

Lin et al. [24] found that setting ${\alpha }_i^{seq}=0.07$ W/MB/s and ${\alpha }_i^{rnd}=0.22$ W/MB/s can more accurately estimate the power consumption of the disk.

• Network power model

We mainly discuss the power of NICs. Under the framework of the TCP/IP protocol, we discuss the power NIC $p_{i,k}^{NIC}$, which is mainly based on the number of IP packets (the size of an IP packet is 500 bytes, denoted as $ip\_size$) that it can handle per second. We have:

$$p_{i,k}^{NIC}=p_{idle,i}^{NIC}+{\alpha }_i^{NIC}\times u_{i,k}^{NIC},$$

(7)

where ${\alpha }_i^{NIC}$ is the power consumption of each IP packet processed by $h_i$’s NIC, $u_{i,k}^{NIC}=s_{j,k}^{net}/ip\_size$ is the number of IP packets processed at any time in $t_k$, and $p_{idle,i}^{NIC}$ is the power of NIC in idle state. Here, we assume that the number of IP packets processed by the NIC at any time in $t_k$ is equal. Garcia-Saavedra et al. [47] found that the value of ${\alpha }_i^{NIC}$ should be $1.26\times {10}^{-5}$ joule/packet.

Hence, combining with Equations (2), (3), (5) and (7), we obtain:

$$p_{h_i}=p_{i,k}^{cpu}+p_{i,k}^{mem}+p_{i,k}^{disk}+p_{i,k}^{NIC}.$$

(8)

Hence, combing with Equation (1), we obtain the energy consumption cost of all hosts $C_H$:

$$C_H=\sum _{i=1}^N{C}_{h_i}=\sum _{i=1}^N\sum _{k=1}^L[EP\times (p_{i,k}^{cpu}+p_{i,k}^{mem}+p_{i,k}^{disk}+p_{i,k}^{NIC})\times T\times {\lambda }_{i,k}].$$

(9)

3.1.2. Cost of VM Migration

We assume that at the beginning of each time segment, the CDC performs server consolidation to balance between the CSP’s cost and the user’s performance. VM migration is an important part of server consolidation. The majority of data transferred during VM live migration are from the memory of VM. Although the VM generates several dirty pages during live migration, Dargie et al. [26] found that the energy consumption of VM live migration is positively related to the memory size of the migrating VM. The larger the VM memory at the time of migration, the longer the migration time and the more energy consumption.

In CDC, migrating a VM $v_j$ from host $h_i$ to another host $h_{i^{\prime }}$ requires additional resources provided by $h_i$ to support the process. We assume that $h_i$ reserves enough resources to complete the migration of $v_j$, and $h_{i^{\prime }}$ also reserves enough resources to receive $v_j$.

The work conducted by Buyya et al. [6] on server consolidation is based on the assumption that a VM consumes an additional 10% of CPU usage during migration. In this paper, we extend this assumption to the VM usage with respect to all four computing resources in migration. In addition, in order to speed up batch VM migration and reduce the impact on user services during batch VM migration, we assume that CDC deploys an exclusive NIC and network for migration tasks, and here the size of the exclusive migration bandwidth of $h_i$ is ${MIG\_NET}_i$. Therefore, the VM migrations cloud be completed within a time segment. This part of the cost is denoted as $C_k^{mig\_net}$. In this paper, the total cost of VM migration of a CDC in a given life cycle is denoted as $C_{mig}$. Since the power model of CPU is a nonlinear exponential model, and the power model of other resources are linear, $C_{mig}$ can be described as:

$$C_{mig}=\sum _{k=1}^L(C_k^{vcpu}+C_k^{mig}+C_k^{mig\_net}),$$

(10)

where $C_k^{vcpu}$ is the cost caused by CPU when migrating VMs within $t_k$, and $C_k^{mig}$ is the cost caused by non-CPU resources when migrating VMs within $t_k$.

$C_k^{mig}$ is described as:

$$C_k^{mig}=\sum _{i=1}^N\sum _{j=1}^M\sum _{i^{\prime }=1}^N(EP\times {\gamma }_{j,i,i^{\prime },k}\times p_{j,k}^{mig}\times t_{j,k}^{mig}),$$

(11)

where ${\gamma }_{j,i,i^{\prime },k}$ is a 0-1 indicator, $p_{j,k}^{mig}$ is the power of migrating $v_j$, and $t_{j,k}^{mig}$ is the time spent migrating $v_j$. If $v_j$ is migrated from $h_i$ to $h_{i^{\prime }}$, then ${\gamma }_{j,i,i^{\prime },k}=1$; otherwise ${\gamma }_{j,i,i^{\prime },k}=0$. Furthermore, it is obviously that $t_{j,k}^{mig}<T$. Since VM memory is the majority data transferred during migration, there are:

$$t_{j,k}^{mig}=\frac{s_{j,k}^{mem}}{mig\_b{w}_{i,k}},$$

(12)

where $mig\_b{w}_{i,k}$ is the migration bandwidth size allocated by $h_i$ to $v_j$. We assume that the dedicated migration bandwidth is evenly allocated to each migrated VM within $t_k$ on $h_i$. Hence, for a given $h_i$ and a $h_{i^{\prime }}$, we have:

$$mig\_b{w}_{i,k}=\frac{MIG\_NE{T}_i}{{\sum }_{i=1}^N{\sum }_{j=1}^M{\sum }_{i^{\prime }=1}^N{\gamma }_{j,i,i^{\prime },k}},$$

(13)

then

$$t_{j,k}^{mig}=\frac{s_{j,k}^{mem}\times {\sum }_{i=1}^N{\sum }_{j=1}^M{\sum }_{i^{\prime }=1}^N{\gamma }_{j,i,i^{\prime },k}}{mig\_b{w}_{i,k}\times MIG\_NE{T}_i}.$$

(14)

Then, we substitute Equation (14) into Equation (11).

We let $p_{j,k}^{vmem}$, $p_{j,k}^{vdisk}$, and $p_{j,k}^{vnet}$ be the memory, disk, and network power of $v_j$ within $t_k$, respectively. The migration cost regarding memory of $v_j$ is $0.1\times p_{j,k}^{vmem}=0.1\times {\alpha }_i^{mem}\times s_{j,k}^{mem}$, the migration cost regarding memory of $v_j$ is $0.1\times p_{j,k}^{vdisk}=0.1\times {\alpha }_i^{disk}\times s_{j,k}^{disk}$, and the migration cost regarding network of $v_j$ is $0.1\times p_{j,k}^{vnet}=0.1\times {\alpha }_i^{NIC}\times u_{i,k}^{NIC}$. Hence, there are

$$p_{j,k}^{mig}=0.1\times ({\alpha }_i^{mem}\times s_{j,k}^{mem}+{\alpha }_i^{disk}\times s_{j,k}^{disk}+{\alpha }_i^{NIC}\times u_{i,k}^{NIC}).$$

(15)

Based on the power model in Equation (7), we discuss $C_k^{mig\_net}$.

$$C_k^{mig\_net}=2\times \sum _{i=1}^N(EP\times {\lambda }_{i,k}\times p_{i,k}^{mig\_net}\times T),$$

(16)

where $p_{i,k}^{mig\_net}$ is the power of the dedicated NIC for migration on $h_i$. Since $h_i$ is sending VM data and $h_{i^{\prime }}$ is receiving corresponding data, the NIC power consumption of both needs to be considered. We obtain

$$p_{i,k}^{mig\_net}\times T=({\alpha }_i^{NIC}\times u_{i,k}^{mig\_net}+p_{idle,i}^{N}IC)\times T,$$

(17)

where $u_{i,k}^{mig\_net}\times T$ is the total transferred VM data from $h_i$ to $h_{i^{\prime }1}$ in time segment $t_k$. Then we have:

$$u_{i,k}^{mig\_net}\times T=\sum _{j=1}^M({\gamma }_{j,i,i^{\prime },k}\times s_{j,k}^{mem}).$$

(18)

We substitute Equation (18) into Equation (16) and have:

$$C_k^{mig\_net}=2\times \sum _{i=1}^N[EP\times {\lambda }_{i,k}\times ({\alpha }_i^{NIC}\times \sum _{j=1}^M{\gamma }_{j,i,i^{\prime },k}\times s_{j,k}^{mem}+T\times p_{idle,i}^{N}IC)].$$

(19)

We discuss $C_k^{vcpu}$ in the following paragraph. A 0-1 indicator ${\beta }_{i,j,k}$ is used to mark whether VM $v_j$ is running on host $h_i$ at the beginning of $t_k$. If $v_j$ is running on $h_i$, then ${\beta }_{i,j,k}=1$; otherwise, ${\beta }_{i,j,k}=0$. We assume that the power consumption of the hosts in CDC is mainly used to keep VMs running. Hence, regarding $h_i$, $(p_{peak,i}-p_{idle,i})\times {\left(u_{i,k}^{cpu}\right)}^{{\alpha }_i^{cpu}}$ in Equation (2) can be presented as:

$$(p_{peak,i}-p_{idle,i})\times {\left(u_{i,k}^{cpu}\right)}^{{\alpha }_i^{cpu}}=(p_{peak,i}-p_{idle,i})\times {\left[\sum _{j=1}^M({\beta }_{i,j,k}\times s_{j,k}^{cpu})\right]}^{{\alpha }_i^{cpu}}.$$

(20)

Then, we take VM migrations on $h_i$ into consideration:

$$\begin{array}{c}(p_{peak,i}-p_{idle,i})\times {[\sum _{j=1}^M({\beta }_{i,j,k}\times s_{j,k}^{cpu})+0.1\times \sum _{j=1}^M({\gamma }_{j,i,i^{\prime },k}\times s_{j,k}^{cpu})]}^{{\alpha }_i^{cpu}}\\ =(p_{peak,i}-p_{idle,i})\times {\left\{\sum _{j=1}^M[({\beta }_{i,j,k}+0.1\times {\gamma }_{j,i,i^{\prime },k})\times s_{j,k}^{cpu}]\right\}}^{{\alpha }_i^{cpu}}.\end{array}$$

(21)

It should be noted that if ${\gamma }_{j,i,i^{\prime },k}=1$, then ${\beta }_{i,j,k}=1$.

Then, we substitute Equation (21) into Equation (2). We denote the new $C_H$ caused by updated $p_{i,k}^{cpu}$ as $C_H^{\prime }$.

3.1.3. SLAV Penalty

In CDCs, once SLAV emerges, then the CSP must compensate the relevant users in some form. This is an effective method for CSPs to guarantee QoS after users pay for cloud services. This part of the overhead needs to be included in the cost of CDC.

We expand the CPU-based SLA definition [13] to CPU SLAV, memory SLAV, disk SLAV, and network SLAV, which are denoted as $SLA{V}_{cpu}$, $SLA{V}_{mem}$, $SLA{V}_{disk}$, $SLA{V}_{net}$, respectively.

$$SLA{V}_{cpu}=\frac{1}{N}\sum _{i=1}^N\frac{T_i^{s,cpu}}{T_i^{a,cpu}}\times \frac{1}{M}\sum _{j=1}^M\sum _{k=1}^L\frac{u{d}_{i,k}^{d,cpu}}{s_{i,k}^{r,cpu}},$$

(22)

where $T_i^{s,cpu}$ is CPU SLAV duration caused by CPU overloaded on $h_i$, and $T_i^{a,cpu}$ is the working duration of $h_i$, and $d_i^{d,cpu}$ is the size of the unsatisfied demand for CPU resources as a result of $v_j$ migration in $t_k$.

Similarly, we formally define $SLA{V}_{mem}$, $SLA{V}_{disk}$ and $SLA{V}_{net}$ as follows:

$$SLA{V}_{mem}=\frac{1}{N}\sum _{i=1}^N\frac{T_i^{s,mem}}{T_i^{a,mem}}\times \frac{1}{M}\sum _{j=1}^M\sum _{k=1}^L\frac{u{d}_{i,k}^{d,mem}}{s_{i,k}^{r,mem}},$$

(23)

$$SLA{V}_{disk}=\frac{1}{N}\sum _{i=1}^N\frac{T_i^{s,disk}}{T_i^{a,disk}}\times \frac{1}{M}\sum _{j=1}^M\sum _{k=1}^L\frac{u{d}_{i,k}^{d,disk}}{s_{i,k}^{r,disk}},$$

(24)

and

$$SLA{V}_{net}=\frac{1}{N}\sum _{i=1}^N\frac{T_i^{s,net}}{T_i^{a,net}}\times \frac{1}{M}\sum _{j=1}^M\sum _{k=1}^L\frac{u{d}_{i,k}^{d,net}}{s_{i,k}^{r,net}},$$

(25)

where the definitions of corresponding parameters are similar to those of $SLA{V}_{cpu}$, and no further explanation will be given here.

In a given life cycle of CDC, the penalty cost caused by the appearance of SLAV is

$$C_{SLAV}=pu{n}_{cpu}\times SLA{V}_{cpu}+pu{n}_{mem}\times SLA{V}_{mem}+pu{n}_{disk}\times SLA{V}_{disk}+pu{n}_{net}\times SLA{V}_{net},$$

(26)

where $pu{n}_{cpu}$, $pu{n}_{mem}$, $pu{n}_{disk}$, and $pu{n}_{net}$ are the CPU, memory, disk, and network SLAV penalty price index, respectively.

3.2. Problem Description

In Section 3.1, we discuss the cost models involved in the operating cost of computing equipment in CDC, and they are host energy consumption cost $C_H$, the VM migration cost $C_{mig}$, and SLAV penalty cost $C_{SLAV}$. In this paper, our goal is to minimize the associated operating cost C of CDC. Based on above models, we have:

$$MINC=C_H^{\prime }+\sum _{k=1}^L(C_k^{mig}+C_k^{mig\_net})+C_{SLAV}.$$

(27)

We name this problem as the Minimizing Computing Resources Cost problem (MCRC problem) in server consolidation. The constraints of MCRC problem are:

$$\sum _{i=1}^N{\beta }_{i,j,k}=1,\forall j,\forall k,$$

(28)

$$\sum _{i^{\prime }=1}^N{\gamma }_{j,i,i^{\prime },k}=1,i\ne i^{\prime },\forall j,\forall k,$$

(29)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{cpu}\le s_i^{cpu},\forall i,\forall k,$$

(30)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{mem}\le s_i^{mem},\forall i,\forall k,$$

(31)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{disk}\le s_i^{disk},\forall i,\forall k,$$

(32)

and

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{net}\le s_i^{net},\forall i,\forall k.$$

(33)

Equation (28) indicates that any VM can only run on a unique host in any given time segment. Equation (29) indicates that the destination host of any VM migration exists and is unique. In Equations (30)–(33) indicate that the CPU, memory, disk and network resources provided by each host to the VMs cannot exceed its own resource limits, respectively.

Now we analyze the complexity of the MCRC problem. Considering a simple case that satisfies the following conditions. If the hosts in the CDC are homogeneous, the resource requirements of any VM $v_j$ in any time segment $t_k$ are fixed values and satisfy In Equations (30)–(33), then the VM migration cost and SLAV penalty cost are both 0, and the objective function of the MCRC problem is:

$$MINC=C_H.$$

(34)

Apparently, the MCRC problem in this simple case can be reduced to the Bin-Packing problem. Since Bin-Packing problem is NP-hard, the MCRC problem is also NP-hard.

4. Solution for MCRC Problem

Since the MCRC problem is NP-hard, we propose a heuristics solution in this paper based on the traditional three-step method for dealing with server consolidation. The first step is to detect the overloaded and underloaded host; the second step is to select suitable VMs from the overloaded host to be migrated; the third step is to select appropriate hosts VMs to be migrated.

Before performing host overloaded detection and VM selection, we first predict the future workload changes of the VMs based on their historical workloads. The purpose of this is to balance loads of hosts before they become overloaded and cause the occurrence of SLAV, thereby reducing cost.

4.1. VM Workload Prediction

Before predicting the future workload of a VM, we first need to preprocess the history of the workload. In Section 3.1.1, we assume that in a certain time segment $t_k$, the usage of a given resource by a VM is constant. In practice, this phenomenon is rare. For example, assuming that the sampling interval for workload records is five minutes (five minutes is the sampling interval used by most public workload trace data sets), then the CPU usage of the VM during these five minutes will vary according to business conditions. The following situation is likely to occur. During the 299 s after the $(k-1)$-th sampling, the CPU usage of a VM is extremely low. However, at the 300th s (k-th sampling), the CPU usage of the VM instantly increases to 90%. This record does not reflect that the VM’s CPU has been under heavy load for the past five minutes. This means two points: (1) there is a certain deviation between the historical sampling records and the actual use of resources by the VM; (2) The assumption in Section 3.1.1 has a certain deviation from the actual use of resources by the VM. To minimize the impact of these biases on the final result, we can consider the presence of noise in the workload records. In this paper, we leverage the convolutional auto-encoder (CAE) to build a filter algorithm to preprocess the workload of the VM and then adopt the Attention-based RNN method for prediction.

4.1.1. A Convolutional Auto-Encoder-Based Filter

CAE has been shown to process time series data well [48].

We use the CAE based filter to process the four resource usages of the VM, respectively. We take the historical time series data of CPU usage by VM $v_j$ as an example to illustrate the proposed CAE based filter. But we will not describe the mathematical principles of CAE in detail in this paper.

Let the CPU usage record of $v_j$ in l consecutive time periods as time series data $\{s_{j,1}^{cpu},s_{j,2}^{cpu},\cdots ,s_{j,l}^{cpu}\}$. After being processed by the encoder and decoder of the CAE based filter, the denoised time series data of CPU usage is $\{{s^{\prime }}_{j,1}^{cpu},{s^{\prime }}_{j,2}^{cpu},\cdots ,{s^{\prime }}_{j,l}^{cpu}\}$. The neural network structure of the CAE based filter is shown in Figure 1.

As shown in Figure 1, both the encoder and decoder are three-layer network structures, and the activation function is $Tanh\left(\right)$. In our actual use, the input is data collected for 48 consecutive time periods (4 h) and normalized. The data collected in the last 5000 consecutive time periods are used for training. Figure 2 shows the result of denoising the CPU usage of a VM in the Alibaba CDC [49] with the proposed CAE based filter.

4.1.2. An Attention-Based RNN Prediction Method

The time series data denoised by the CAE based filter are used to predict the resource usage of the VM in the future time period. The proposed prediction method is based on attention mechanism, named as the ARP algorithm, and is based on our previous work [50].The ARP algorithm predicts the usage of four resources by VMs at the same time, without separate processing.

The ARP algorithm consists of two parts: the encoder and the decoder. In the encoder, an attention module is used to adaptively select the relevant series. Then in the decoder, the relevant encoder hidden states is selected via another attention module [50]. We demonstrate the neural network structures of the encoder and the decoders in Figure 3 and Figure 4, respectively.

Let $S_{j,l}={\{s_{j,l}^{cpu},s_{j,l}^{mem},s_{j,l}^{disk},s_{j,l}^{net}\}}^{\top }$, $s_{j,l}^{cpu}={\{s_{j,1}^{cpu},s_{j,2}^{cpu},\cdots ,s_{j,L}^{cpu}\}}^{\top }$ (L is size of the window), $s_{j,l}^{mem}={\{s_{j,1}^{mem},s_{j,2}^{mem},\cdots ,s_{j,L}^{mem}\}}^{\top }$, $s_{j,l}^{disk}={\{s_{j,1}^{disk},s_{j,2}^{disk},\cdots ,s_{j,L}^{disk}\}}^{\top }$, $s_{j,l}^{net}={\{s_{j,1}^{net},s_{j,2}^{net},\cdots ,s_{j,L}^{net}\}}^{\top }$. The attention mechanism in the encoder calculates the encoder attention weight ${\alpha }_l^q$ based on the previous $l-1$ hidden state, and then get $S_{j,l}^{\prime }=\{{\alpha }_l^1\times s_{j,l}^{cpu},{\alpha }_l^2\times s_{j,l}^{mem},{\alpha }_l^3\times s_{j,l}^{disk},{\alpha }_l^4\times s_{j,l}^{net}\}$. Then $S_{j,l}^{\prime }$ is fed to the LSTM unit in the encoder. The attention mechanism in the decoder calculates the attention weight ${\beta }_l^l$ based on the previous $l-1$ hidden state. At time l, for L$S_{j,l}^{\prime }$, their attention weights are ${\beta }_l^1$ to ${\beta }_l^L$ at time l, respectively. The input information is represented as a weighted sum of the encoder hidden states across all the time steps, and then is fed to the LSTM unit of the decoder. The output result of LSTM is the prediction of the next time period. For the specific explanation and mathematical description of the model, such as the calculation method of the attention weights, please refer to our previous work [50].

As the examples, Figure 5, Figure 6 and Figure 7 demonstrate, respectively, the results of the CPU part after using the ARP algorithm (L = 10) to predict the resource usages of three different VMs in the Alibaba CDC [49].

After accurately predicting the resources usage of each VM in the next period, we can perform host workload detection and VM selection.

4.2. Host Workload Detection

The purpose of host overloaded detection is to avoid and eliminate the fierce competition of VMs for resources, thereby reducing the occurrence of SLAV. Common host overloaded detection methods are divided into two categories, static threshold method, and dynamic threshold method. Regarding the static threshold method, the overloaded thresholds for various resources are set as fixed values. When the usages exceed the thresholds, the host is overloaded, and SLAV occurs. At this time, the VMs must be migrated to reduce the load. Regarding the dynamic threshold method, various statistical methods are used to analyze the usages of computing resources by VMs or hosts, to determine whether the competition for resources by VMs is intense, and whether the host is overloaded. The advantage of the static threshold method is that the host resources are fully utilized, but the disadvantage is that once overloaded, more overhead is required to solve the SLAV. The advantage of the dynamic threshold method is that it can effectively reduce the SLAV, but sometimes the use of host resources is not sufficient. Therefore, we combine the advantages of the two and propose an ARP-based fixed threshold overloaded detection method (ARP-FT method).

ARP-FT is a dual detection method. At the beginning of the time segment $t_k$, ARP-FT first detects whether the usage of various resources on $h_i$ exceeds the given thresholds, and then, based on the prediction result of the ARP algorithm, we judge whether the usage of various resources on $h_i$ exceeds the given thresholds in the next period.

Let the overloaded thresholds $T{H}_{up}=\{T{H}_{up}^{cpu},T{H}_{up}^{mem},T{H}_{up}^{disk},T{H}_{up}^{net}\}$, where $T{H}_{up}^{cpu}$, $T{H}_{up}^{mem}$, $T{H}_{up}^{disk}$, and $T{H}_{up}^{net}$. They are all in the interval $(0,1)$. For host $h_i$, it is overloaded in time segment $t_k$ when the following inequalities hold:

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{cpu},T{H}_{up}^{cpu}\times c_i^{cpu},$$

(35)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{mem}>T{H}_{up}^{mem}\times c_i^{mem},$$

(36)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{disk}>T{H}_{up}^{disk}\times c_i^{disk},$$

(37)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{net}>T{H}_{up}^{net}\times c_i^{net},$$

(38)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{cpu}>T{H}_{up}^{cpu}\times c_i^{cpu},$$

(39)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{mem}>T{H}_{up}^{mem}\times c_i^{mem},$$

(40)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{disk}>T{H}_{up}^{disk}\times c_i^{disk},$$

(41)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{net}>T{H}_{up}^{net}\times c_i^{net}.$$

(42)

Let the underloaded thresholds $T{H}_{down}=\{T{H}_{down}^{cpu},T{H}_{down}^{mem},T{H}_{down}^{disk},T{H}_{down}^{net}\}$, where $T{H}_{down}^{cpu}$, $T{H}_{down}^{mem}$, $T{H}_{down}^{disk}$, and $T{H}_{down}^{net}$. They are all in the interval $(0,1)$. For host $h_i$, it is underloaded in time segment $t_k$ when the following inequalities hold:

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{cpu}<T{H}_{down}^{cpu}\times c_i^{cpu},$$

(43)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{mem}<T{H}_{down}^{mem}\times c_i^{mem},$$

(44)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{disk}<T{H}_{down}^{disk}\times c_i^{disk},$$

(45)

$$\sum _{j=1}^M{\beta }_{i,j,k}\times s_j^{net}<T{H}_{down}^{net}\times c_i^{net},$$

(46)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{cpu}<T{H}_{down}^{cpu}\times c_i^{cpu},$$

(47)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{mem}<T{H}_{down}^{mem}\times c_i^{mem},$$

(48)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{disk}<T{H}_{down}^{disk}\times c_i^{disk},$$

(49)

$$\sum _{j=1}^M{\beta }_{i,j,k+1}\times s_j^{net}<T{H}_{down}^{net}\times c_i^{net}.$$

(50)

4.3. VM Selection

VM selection is for overloaded hosts. The reason why we use the ARP algorithm is to avoid host overloading and SLAV as much as possible, rather than reactively responding to SLAV after its occurrence. Therefore, we assume that in time segment $t_{k+1}$, there would be SLAV and overloaded hosts in the CDC, but not many. Regarding VM selection in $t_k$ for $h_i$, our priority is to select the VMs that may cause $h_i$ to be overloaded during time segment $t_{k+1}$ and form a list of VMs to be migrated. After the migration of these VMs is completed, if $h_i$ is still overloaded in $t_k$, it will be processed accordingly.

Given a host $h_i$ overloaded in the beginning of certain future time segment $t_l$, we define its overloaded type $ol\_typ{e}_{i,l}$. We check which inequalities in Equations (35)–(38) are satisfied by $h_i$, so as to determine which resources in cpu, memory, disk, and network are involved in its overload state. Let the CPU, memory, disk, and network overloaded marks be $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$, respectively. In time segment $t_l$, they are denoted as ${\mathbf{A}}_{i,l}$, ${\mathbf{B}}_{i,l}$, ${\mathbf{C}}_{i,l}$, and ${\mathbf{D}}_{i,l}$, respectively, and their corresponding values are $V\left({\mathbf{A}}_{i,l}\right)=\frac{c_i^{cpu}-r_{i,l}^{cpu}}{T{H}_{up}^{cpu}\times c_i^{cpu}}$, $V\left({\mathbf{B}}_{i,l}\right)=\frac{c_i^{mem}-r_{i,l}^{mem}}{T{H}_{up}^{mem}\times c_i^{mem}}$, $V\left({\mathbf{C}}_{i,l}\right)=\frac{c_i^{disk}-r_{i,l}^{disk}}{T{H}_{up}^{disk}\times c_i^{disk}}$, and $V\left({\mathbf{D}}_{i,l}\right)=\frac{c_i^{net}-r_{i,l}^{net}}{T{H}_{up}^{net}\times c_i^{net}}$. If $h_i$ satisfies inequalities (35) and (37), that is, $V\left({\mathbf{A}}_{i,l}\right)>1$ and $V\left({\mathbf{C}}_{i,l}\right)>1$, then its overloaded type sequence can be denoted as $\left({\mathbf{A}}_{i,l}{\mathbf{C}}_{i,l}\right)$. We sort the overloaded resources in $h_i$’s overloaded type sequence in descending order according to the overloaded mark value. For example, if the initial overloaded type sequence of $h_i$ is $\left({\mathbf{A}}_{i,l}{\mathbf{B}}_{i,l}{\mathbf{C}}_{i,l}{\mathbf{D}}_{i,l}\right)$, and $V\left({\mathbf{A}}_{i,l}\right)=0.1$, $V\left({\mathbf{B}}_{i,l}\right)=0.3$, $V\left({\mathbf{C}}_{i,l}\right)=0.05$, and $V\left({\mathbf{D}}_{i,l}\right)=0.2$, then the sorted overloaded type sequence is $\left({\mathbf{B}}_{i,l}{\mathbf{D}}_{i,l}{\mathbf{A}}_{i,l}{\mathbf{C}}_{i,l}\right)$. The sorted overloaded type sequence is $ol\_typ{e}_{i,l}$ of $h_i$ in $t_l$. Regarding the above instance, $ol\_typ{e}_{i,l}=\left({\mathbf{B}}_{i,l}{\mathbf{D}}_{i,l}{\mathbf{A}}_{i,l}{\mathbf{C}}_{i,l}\right)$.

Given a VM $v_j$, we define its workload type $wl\_typ{e}_{j,l}$ in $t_l$. Let its CPU, memory, disk, and network usage marks be $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$ and $\mathbf{D}$, respectively. In time segment $t_l$, they are denoted as ${\mathbf{A}}_{j,l}$, ${\mathbf{B}}_{j,l}$, ${\mathbf{C}}_{j,l}$, and ${\mathbf{D}}_{j,l}$, respectively, and their corresponding values are $V\left({\mathbf{A}}_{j,l}\right)=\frac{s_{j,l}^{cpu}}{c_{max}^{cpu}}$, $V\left({\mathbf{B}}_{j,l}\right)=\frac{s_{j,l}^{mem}}{c_{max}^{mem}}$, $V\left({\mathbf{C}}_{j,l}\right)=\frac{s_{j,l}^{disk}}{c_{max}^{disk}}$, and $V\left({\mathbf{D}}_{j,l}\right)=\frac{s_{j,l}^{net}}{c_{max}^{net}}$, where $c_{max}^{cpu}=max\left\{c_i^{cpu}\right|i\in [1,N]\}$, $c_{max}^{mem}=max\left\{c_i^{mem}\right|i\in [1,N]\}$, $c_{max}^{disk}=max\left\{c_i^{disk}\right|i\in [1,N]\}$, and $c_{max}^{net}=max\left\{c_i^{net}\right|i\in [1,N]\}$. Let the initial workload type sequence of VM $v_j$ be $\left({\mathbf{A}}_{j,l}{\mathbf{B}}_{j,l}{\mathbf{C}}_{j,l}{\mathbf{D}}_{j,l}\right)$. We sort the initial sequence in descending order according to the values of resource usage marks, and the sorted workload type sequence is $wl\_typ{e}_{j,l}$. For example, if $V\left({\mathbf{A}}_{j,l}\right)=0.1$, $V\left({\mathbf{B}}_{j,l}\right)=0.09$, $V\left({\mathbf{C}}_{j,l}\right)=0.15$, and $V\left({\mathbf{D}}_{j,l}\right)=0.02$, then $wl\_typ{e}_{j,l}=\left({\mathbf{C}}_{j,l}{\mathbf{A}}_{j,l}{\mathbf{B}}_{j,l}{\mathbf{D}}_{j,l}\right)$. Then, we define the complementary workload type $cwl\_typ{e}_{j,l}$ of $v_j$ in $t_l$ as the reverse sequence of $wl\_typ{e}_{j,l}$.

For the overloaded type of $h_j$, we select as few VMs as possible to migrate, so that the values of ${\mathbf{A}}_{j,l}$, ${\mathbf{B}}_{j,l}$, ${\mathbf{C}}_{j,l}$, and ${\mathbf{D}}_{j,l}$ are all less than or equal to 1. For example, if the overloaded type of $h_i$ is $\left(\mathbf{D}\mathbf{A}\right)$, then the VMs with the workload type $(\mathbf{D}\mathbf{A}\ast \ast )$ should be selected for migration as much as possible. Therefore, we need to use $\left(\mathbf{D}\mathbf{A}\right)$ as the reference to sort the VMs on $h_i$.

Given marks ${\mathbf{A}}^{\prime }$, ${\mathbf{B}}^{\prime }$, ${\mathbf{C}}^{\prime }$, and ${\mathbf{D}}^{\prime }$, the corresponding values of host $h_i$ and VM $v_j$ in $t_l$ are $V\left({\mathbf{A}}_{i,l}^{\prime }\right)=r_{i,l}^{cpu}-T{H}_{up}^{cpu}\times c_i^{cpu}$, $V\left({\mathbf{B}}_{i,l}^{\prime }\right)=r_{i,l}^{mem}-T{H}_{up}^{mem}\times c_i^{mem}$, $V\left({\mathbf{C}}_{i,l}^{\prime }\right)=r_{i,l}^{disk}-T{H}_{up}^{disk}\times c_i^{disk}$, $V\left({\mathbf{D}}_{i,l}^{\prime }\right)=r_{i,l}^{net}-T{H}_{up}^{net}\times c_i^{net}$, $V\left({\mathbf{A}}_{j,l}^{\prime }\right)=s_{j,l}^{cpu}$, $V\left({\mathbf{B}}_{j,l}^{\prime }\right)=s_{j,l}^{mem}$, $V\left({\mathbf{C}}_{j,l}^{\prime }\right)=s_{j,l}^{disk}$, and $V\left({\mathbf{D}}_{j,l}^{\prime }\right)=s_{j,l}^{net}$. Given the overloaded reference $RF=({\mathbf{E}}_1\cdots {\mathbf{E}}_p)$, where ${\mathbf{E}}_g\in \{{\mathbf{A}}^{\prime },{\mathbf{B}}^{\prime },{\mathbf{C}}^{\prime },{\mathbf{D}}^{\prime }\}$, and the sort of ${\mathbf{E}}_1\cdots {\mathbf{E}}_p$ corresponds to that of $ol\_typ{e}_{j,l}$. The resources usage of $v_j$ based on $RF$ is denoted as $R{U}_j=({\mathbf{e}}_{j,1}\cdots {\mathbf{e}}_{j,p})$, where ${\mathbf{e}}_{j,g}\in \{{\mathbf{A}}^{\prime },{\mathbf{B}}^{\prime },{\mathbf{C}}^{\prime },{\mathbf{D}}^{\prime }\}$ and the sort of ${\mathbf{e}}_{j,1}\cdots {\mathbf{e}}_{j,p}$ also corresponds to that of $ol\_typ{e}_{j,l}$.

Regarding two different VMs $v_j$ and $v_{j^{\prime }}$, if $v_j$ is better than $v_{j^{\prime }}$ based on $RF$, denoted as $v_j>v_{j^{\prime }}$, then the following is met:

$$\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j,g}\right)|>\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j^{\prime },g}\right)|,$$

(51)

where ${\alpha }_g^{RF}$ is the overloaded reference weight. Resources with higher overloaded levels should be prioritized to reduce load; hence the value of the corresponding ${\alpha }_g^{RF}$ should be larger. We let the value of ${\alpha }_g^{RF}$ be equal to the value of resource mark in $wl\_typ{e}_{j,l}$ that corresponds to the ${\mathbf{E}}_g$ of the same subscript g. For instance, assuming $wl\_typ{e}_{j,l}=\left({\mathbf{C}}_{j,l}{\mathbf{A}}_{j,l}\right)$, then $RF=\left({\mathbf{E}}_1{\mathbf{E}}_2\right)=\left({\mathbf{C}}_{j,l}^{\prime }{\mathbf{A}}_{j,l}^{\prime }\right)$, ${\alpha }_1^{RF}=V\left({\mathbf{C}}_{j,l}\right)$, and ${\alpha }_2^{RF}=V\left({\mathbf{A}}_{j,l}\right)$.

If $v_j$ is equivalent to $v_{j^{\prime }}$ based on $RF$, denoted as $v_j=v_{j^{\prime }}$, then the following is met:

$$\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j,g}\right)|=\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j^{\prime },g}\right)|.$$

(52)

If $v_j$ is worse than $v_{j^{\prime }}$ based on $RF$, denoted as $v_j<v_{j^{\prime }}$, then the following is met:

$$\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j,g}\right)|<\prod _{g=1}^p{\alpha }_g^{RF}\times |V\left({\mathbf{E}}_g\right)-V\left({\mathbf{e}}_{j^{\prime },g}\right)|.$$

(53)

According to the above given relationship between VMs based on $RF$, we sort the VMs on $h_i$ from ’good’ to ’bad’ as an ordered list. Subsequently, the VMs are sequentially selected from the ordered list and placed in the list of VM to be migrated. After each selection, we determine whether $h_i$ is still overloaded after removing the resources required by the selected VM. If it is overloaded, continue to select VMs in order; otherwise, stop VM selection.

4.4. VM Placement

In order to make full use of the resources of a host, we should fully consider the competition of various VMs for resources when placing them on the same host. This competition is divided into two aspects: space and time of resources usage.

In a given period of time, competition should be evenly distributed over different resources, rather than a situation where multiple VMs scramble for the same one or two resources while leaving others idle. We illustrate this with a simple example. Assume that the case where there are only two resources, CPU and memory, is considered. The CPU and memory provided by host H are both 10 units, and the VMs to be placed are V1–V7. The CPU and memory requirements $(s^{cpu},s^{mem})$ of these VMs are $(2.40,1.16)$, $(2.46,1.11)$, $(2.46,0.97)$, $(0.73,2.72)$, $(1.05,2.13)$, $(0.87,1.91)$, and $(2.65,0.97)$, respectively. Figure 8 demonstrates two methods of VM placement, which are shown in Figure 8a,b, respectively. Obviously, the placement in Figure 8a is more reasonable because this method allows the VMs to make full use of the two resources of the host H. In Figure 8b, the VMs’ scramble for resources is concentrated on CPU, so that the utilization rate of H’s CPU has reached the highest point, while the utilization rate of memory resources has not exceeded half, resulting in waste.

In the two adjacent time periods, the user’s business needs change, and the usage of certain resources of the host may change greatly. When we implement VM placement in $t_k$, we must consider the resource usage of the VM in $t_{k+1}$ to avoid the overloaded state of the destination host in $t_{k+1}$ after migration. We illustrate this with a simple example. For the convenience of discussion, only CPU is considered. During time period $t_k$, the CPU usage of host H is $30\%$. For $t_{k+1}$, the predicted value of the CPU usage of host H is $60\%$. The VMs to be placed are V1–V5. Their required usage rate of CPU in $t_k$ and $t_{k+1}$, $\{s_k,s_{k+1}\}$, are $\{30\%,15\%\}$, $\{20\%,10\%\}$, $\{15\%,10\%\}$, $\{20\%,30\%\}$, and $\{10\%,5\%\}$, respectively. Figure 9 demonstrates two methods of VM placement, which are shown in Figure 9a,b, respectively. Obviously, the placement in Figure 9a is more reasonable. In two consecutive time periods, H’s CPU load did not exceed the upper limit. The VM placement in Figure 9b lets H load more VMs in $t_k$. However, in the $t_{k+1}$ period, the CPU load of H exceeds the upper limit, leaving H in the overloaded state, and SLAV occurs, so VM migration must be used to reduce the load. The placement of VMs for H in Figure 9b increases the operating cost of the CDC.

Based on this competition in resource space and time, we design a resources- complementary-usage-based VM placement strategy. There are two goals we want to achieve: (1) to ensure that the resources of the target host can be fully utilized during the $t_k$ period and (2) after placing the VMs on the given host $h_i$, this host would not be overloaded in $t_{k+1}$.

Given a host $h_i$, we define its workload type $WL\_typ{e}_{i,k}$ in $t_k$. Let its CPU, memory, disk, and network usage marks be $\mathbf{AW}$, $\mathbf{BW}$, $\mathbf{CW}$, and $\mathbf{DW}$, respectively. In time segment $t_k$, they are denoted as ${\mathbf{AW}}_{i,k}$, ${\mathbf{BW}}_{i,k}$, ${\mathbf{CW}}_{i,k}$, and ${\mathbf{DW}}_{i,k}$, respectively, and their corresponding values are $V\left({\mathbf{AW}}_{i,k}\right)=\frac{r_{i,k}^{cpu}}{c_{max}^{cpu}}$, $V\left({\mathbf{BW}}_{i,k}\right)=\frac{r_{i,k}^{mem}}{c_{max}^{mem}}$, $V\left({\mathbf{CW}}_{i,k}\right)=\frac{r_{i,k}^{disk}}{c_{max}^{disk}}$, and $V\left({\mathbf{DW}}_{i,k}\right)=\frac{r_{i,k}^{net}}{c_{max}^{net}}$. Let the initial workload type sequence of host $h_i$ be $\left({\mathbf{AW}}_{i,k}{\mathbf{BW}}_{i,k}{\mathbf{CW}}_{i,k}{\mathbf{DW}}_{i,k}\right)$. We sort the initial sequence in ascending order according to the values of resource usage marks, and the ordered workload type sequence is $WL\_typ{e}_{j,l}$. For example, if $V\left({\mathbf{AW}}_{i,k}\right)=0.1$, $V\left({\mathbf{BW}}_{i,k}\right)=0.09$, $V({\mathbf{CW}}_{i,k}=0.15$, and $V\left({\mathbf{DW}}_{i,k}\right)=0.02$, then $WL\_typ{e}_{j,l}=\left({\mathbf{DW}}_{i,k}{\mathbf{BW}}_{i,k}{\mathbf{AW}}_{i,k}{\mathbf{CW}}_{i,k}\right)$. Then, we define the complementary workload type $CWL\_typ{e}_{i,k}$ of $h_i$ in $t_k$ as the reverse sequence of $WL\_typ{e}_{j,l}$. In the production environment, the power consumption of the four resources in a host is sorted from largest to smallest as CPU, memory, disk, and network. Therefore, in a host workload type sequence, if two resource usages are equal, these two resources are sorted according to the order of their resource power consumption levels. Hence, we have $4!=24$ workload types.

We divided all non-overloaded hosts (including those underloaded ones) into 24 groups by the workload types. Then, the 24 groups of hosts are arranged in descending order by the number of hosts included to form a list:

$HG=\{WLT\_Host\_Lis{t}_1,WLT\_Host\_Lis{t}_2,\cdots ,WLT\_Host\_Lis{t}_{2}4\}$.

Next, we sort the hosts in each $WLT\_Host\_Lis{t}_i$. We compare two hosts of the same workload type as follows. Given the workload type $\left({\mathbf{W}}_{i,1}{\mathbf{W}}_{i,2}{\mathbf{W}}_{i,3}{\mathbf{W}}_{i,4}\right)$, where ${\mathbf{W}}_{i,p}\in \{\mathbf{AW},\mathbf{BW},\mathbf{CW},\mathbf{DW}\}$. Regarding two different host $h_i$ and $h_{i^{\prime }}$ of the same workload type:

If $h_i$ is better than $h_{i^{\prime }}$, denoted as $h_i>h_{i^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{i,p}\right)>\prod _{p=1}^{4}V\left({\mathbf{W}}_{i^{\prime },p}\right).$$

(54)

If $h_i$ is equivalent to $h_{i^{\prime }}$, denoted as $h_i=h_{i^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{i,p}\right)=\prod _{p=1}^{4}V\left({\mathbf{W}}_{i^{\prime },p}\right).$$

(55)

If $h_i$ is worse than $h_{i^{\prime }}$, denoted as $h_i<h_{i^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{i,p}\right)<\prod _{p=1}^{4}V\left({\mathbf{W}}_{i^{\prime },p}\right).$$

(56)

Hosts in the same group, according to the above method of comparison, are sorted in descending order to form an ordered group. All ordered groups are denoted as:

$\{WLT\_Host\_List\_sorte{d}_1,WLT\_Host\_List\_sorte{d}_2,\cdots ,WLT\_Host\_List\_sorte{d}_{24}\}$.

For each host given a workload type, in order to take full advantage of all its resources, it is best to pick and place VMs of the complementary work type that are the same as the host’s workload type. For instance, if the host is $WL\_typ{e}_{i,k}=\left(\mathbf{DW}\mathbf{BW}\mathbf{CW}\mathbf{AW}\right)$, then $CWL\_typ{e}_{i,k}=\left(\mathbf{AW}\mathbf{CW}\mathbf{BW}\mathbf{DW}\right)$, and the VMs of the complementary work type $\left(\mathbf{A}\mathbf{C}\mathbf{B}\mathbf{D}\right)$ should be selected and placed on this host. Hence, we divide all VMs to be migrated into 24 groups according to their complementary workload types, denoted as:

$VG=\{cwlt\_VM\_Lis{t}_1,cwlt\_VM\_Lis{t}_1,\cdots ,cwlt\_VM\_Lis{t}_{24}\}$.

Next, we sort the VMs in each groups. We compare two VMs of the same complementary workload type as follows. Given workload type $\left({\mathbf{W}}_{j,1}{\mathbf{W}}_{j,2}{\mathbf{W}}_{j,3}{\mathbf{W}}_{j,4}\right)$, where ${\mathbf{W}}_{j,p}\in \{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}\}$. Regarding two different VMs $v_j$ and $v_{j^{\prime }}$ of the same complementary workload type:

If $v_j$ is better than $v_{j^{\prime }}$, denoted as $v_j>v_{j^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{j,p}\right)>\prod _{p=1}^{4}V\left({\mathbf{W}}_{j^{\prime },p}\right).$$

(57)

If $v_j$ is equivalent to $v_{j^{\prime }}$, denoted as $v_j=v_{j^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{j,p}\right)=\prod _{p=1}^{4}V\left({\mathbf{W}}_{j^{\prime },p}\right).$$

(58)

If $v_j$ is worse than $v_{j^{\prime }}$, denoted as $v_j<v_{j^{\prime }}$, then the following is met:

$$\prod _{p=1}^{4}V\left({\mathbf{W}}_{j,p}\right)<\prod _{p=1}^{4}V\left({\mathbf{W}}_{j^{\prime },p}\right).$$

(59)

VMs in the same group, according to the above method of comparison, are sorted in descending order to form an ordered group. All ordered groups are denoted as:

$V{G}^{\prime }=\{cwlt\_VM\_List\_sorte{d}_1,cwlt\_VM\_List\_sorte{d}_2,\cdots ,cwlt\_VM\_List\_sorte{d}_{24}\}$.

We select a set of hosts $WLT\_Host\_List\_sorte{d}_x$ from $HG$ in sequential order and obtain a VM group $cwlt\_VM\_List\_sorte{d}_y$ that is the same as its complementary workload type. We select a host from $WLT\_Host\_List\_sorte{d}_x$ in sequence and take out VMs from $cwlt\_VM\_List\_sorte{d}_y$ in sequence and put them into the host. At this point, we need to judge whether the host is overloaded both in the current time period and the next time period after the VM is placed. If the answer is no, the VM can be placed on this host and will be removed from $cwlt\_VM\_List\_sorte{d}_y$; otherwise, skip to the next VM in order.

Traversing $cwlt\_VM\_List\_sorte{d}_y$ for the current host, we then process the next host in order. The above algorithm is named the Time and Space Complementary VM placement algorithm (TSCP algorithm), and its pseudo-code is shown in Algorithm 1.

Algorithm 1 TSCP algoritm.

Input: hostlist, vmlist Output: allocation of the VMs 1: $HG$← get_HG(hostlist) 2: for each $WLT\_Host\_Lis{t}_i$ in $HG$ do 3:  $WLT\_Host\_List\_sorte{d}_i$← get_sorted($WLT\_Host\_Lis{t}_i$) 4: end for 5: $VG$← get_VG(vmlist) 6: $V{G}^{\prime }$← get_VG’($VG$) 7: for each $cwlt\_VM\_Lis{t}_i$ in $V{G}^{\prime }$ do 8:  $cwlt\_VM\_List\_sorte{d}_y$← get_sorted($cwlt\_VM\_Lis{t}_i$) 9: end for 10: for each $WLT\_Host\_List\_sorte{d}_i$ do 11:  $cwlt\_VM\_List\_sorte{d}_{same}$← FindSameWLT($WLT\_Host\_List\_sorte{d}_i$,$V{G}^{\prime }$) 12:  for each HOST in $WLT\_Host\_List\_sorte{d}_i$ do 13:   for each VM in $cwlt\_VM\_List\_sorte{d}_{same}$ do 14:    if HOST is not overloaded with hosting VM in $t_k$ and $t_{k+1}$ then 15:     allocation.add(VM,HOST) 16:    end if 17:   end for 18:  end for 19: end forreturn allocation

After all the above operations are completed, if there are still some VMs not migrated, perform the following steps:

(Step 1)

Implementing the TSCP algorithm again but skipping the steps of VM grouping and sorting;

(Step 2)

If there are still some VMs not migrated, implement the First-Fit VM placement algorithm for these VMs;

(Step 3)

If there are still some VMs not migrated, it indicates that the resources of all working hosts are not enough, and the energy-saving mode hosts should be booted up.

After this, we perform underloaded host detection. If there are underloaded hosts, then we leverage the TSCP algorithm to place all the VMs on them.

5. Performance Evaluation

In this section, we evaluate the performance of our proposed solution with a real VM workload trace-driven simulation.

5.1. Experiment Setup

Based on the energy consumption analysis and statistics of each component of the host in the data center by Minartz et al. [51] and Jin et al. [52], we simulated three sizes of large, medium, and small hosts: $H_{large}$, $H_{medium}$, and $H_{small}$, respectively. Their resource parameters are shown in

Table 1. Resource parameters of the hosts.

Host Type	CPU	Memory	Disk Throughput	Network Throughput
$H_{large}$	4 times Intel Xeon Northwood CPU (single core)	8 GB	399 MB/s	1 GB/s
$H_{medium}$	3 times Intel Xeon Northwood CPU (single core)	6 GB	266 MB/s	1 GB/s
$H_{small}$	2 times Intel Xeon Northwood CPU (single core)	4 GB	133 MB/s	1 GB/s

, and their power parameters are shown in

Table 2. Power parameters of the hosts.

Host Type	Value	CPU (kW)	Memory (kW)	Disk (kW)	NIC (kW)
$H_{large}$	$p_{peak}$	0.232	0.21736	0.02106	0.002
	$p_{idle}$	0.1124	0.17576	0.01326	0.00078
$H_{medium}$	$p_{peak}$	0.174	0.10868	0.01404	0.002
	$p_{idle}$	0.843	0.08788	0.00884	0.00078
$H_{small}$	$p_{peak}$	0.116	0.05434	0.00702	0.002
	$p_{idle}$	0.562	0.04394	0.00442	0.00078

. We simulated a CDC consisting of 200 $H_{small}$ hosts, 100 $H_{medium}$ hosts, and 50 $H_{large}$ hosts.

The VM workload trace dataset is provided by the Alibaba CDC [49]. The dataset contains workload traces of 4000 VMs over 8 consecutive days. The traces in the record are generated by sampling every five minutes. Each trace records the CPU usage, memory usage, disk I/O, and network throughput of a VM at a sampling moment. We selected the traces of 1000 VMs in one day (a total of 288 time segments) from the dataset to simulate the users’ business demands for VMs. The simulation is implemented on the CDC simulator CloudMatrix Lite [53], and the CAE-based filter and the ARP algorithm is based on PyTorch [54].

We set the electricity price as $EP=0.25\$/$kWh, the SLAV penalty is a static price $pu{n}_{cpu}=pu{n}_{mem}=pu{n}_{disk}=pu{n}_{net}=0.01\$$. A VM migration costs 10% extra resources; hence, the host reserves 10% resources for migrations. Based on this, we set $T{H}_{up}^{cpu}=T{H}_{up}^{mem}=$$T{H}_{up}^{disk}=T{H}_{up}^{net}=0.9$. We also set $T{H}_{down}^{cpu}=T{H}_{down}^{mem}=T{H}_{down}^{disk}=T{H}_{down}^{net}=0.1$.

5.2. Evaluation

In the evaluation, four overloading detection algorithms (MAD [6], IQR [6], and LR [6]), three VM selection algorithms (MMT [6,31,39], MC [6,55], and RS [6]), and one VM placement algorithm (PABFD [6]) are combined as nine methods to compare with our proposed solution. The PABFD placement algorithm and its corresponding energy consumption model only take into account the host’s CPU. Therefore, we modify it here to suit our multi-resource scenario. The new PABFD placement algorithm is PABFDM, see Algorithm 2 for the pseudo-code.

Algorithm 2 PABFDM algorithm.

Input: hostList, vmList Output: allocation of the VMs 1: vmList.sortDecreasingUtilization() 2: for each VM in vmList do 3:  minPower ← MAX 4:  for host in hostList do 5:   if no SLAV on this host and this host meets the CPU, memory, disk and network resource requirement for VM then 6:    power ← estimatePower(host,VM) 7:    if power < minPower then 8:     minPower←power 9:    end if 10:   end if 11:  end for 12:  allocation.add(VM,host) 13: end for  return allocation

The naming rule of the baseline methods is ’xx-xx’. For instance, LR-MMT indicates that the combination uses LR for overloading detection and MMT for VM selection. We denote our proposed method as ARP-TSCP.

The metrics involved are host-operation-related costs, SLAV cost, and the number of VM migrations. Since the CPU part of the VM migration energy consumption belongs to the host’s energy consumption during calculation, we use the number of VM migrations to indirectly measure the migration cost.

Figure 10 demonstrates the energy consumption comparison of all hosts using ARP-TSCP and the nine baseline methods in 288 consecutive periods. Figure 11 demonstrates the comparison of the total energy consumption of all hosts using these two methods to consolidate servers throughout the day. It can be seen in Figure 10 that, throughout most of the period, the host energy consumption produced by ARP-TSCP is lower than that produced by the baseline methods. Therefore, in the comparison of the total host energy consumption, ARP-TSCP outperforms the baseline methods. The total host energy consumption generated by executing ARP-TSCP in one day is about 18.5% lower than that generated by executing LR-MMT (the best result of all baseline methods) and is about 30.3% lower than that of MAD-RS (the worst result of all baseline methods), as shown in Figure 11.

Figure 12 demonstrates the comparison of the total SLAV of CPU, memory, disk, and network generated by ARP-TSCP and the baseline methods in one day respectively. As can be seen from Figure 12, in terms of reducing the SLAV of CPU and memory, ARP-TSCP has obvious improvement over all baseline methods. We can argue that based on the workload prediction method and eliminating potential overloaded states in advance is a feasible approach. In addition, the memory SLAV caused by performing ARP-TSCP is much smaller than that of other methods. This is because the baseline methods are essentially designed based only on CPU resource. The baseline host overload detection strategies only consider the CPU overloading situation, and the premise of the PABFD algorithm assumes that the host can provide unlimited memory resources for VMs. In the actual production environment, the competition between VMs for host memory resources is very fierce, resulting in a high frequency of memory SLAV. Most of the VMs in the Alibaba VM workload dataset are not disk IO intensive and require more CPU and memory resources [49]. In addition, the NIC throughput rate of 1 GB/s can meet the business needs of many users most of the time. Therefore, in terms of both disk and network, the related SLAV caused by all methods is less than that of CPU and memory.

As shown in Equations (22)–(25), in the SLAV calculation formula, the entire life cycle of the host is involved, so we do not show the SLAV generated in each time period. In Figure 13, we compare the total cost of the SLAV penalty for all methods over a full day. It can be seen that in the case of $pu{n}_{cpu}=pu{n}_{mem}=pu{n}_{disk}=pu{n}_{net}$, the SLAV penalty cost caused by ARP-TSCP is about 38% lower than that of LR-MMT (the best result of all baseline methods) and is about 52% lower than that of MAD-RS (the worst result of all baseline methods). Due to the importance of CPU and memory in actual use as well as their larger proportion of power consumption of the host, if the values of $pu{n}_{cpu}$ and $pu{n}_{mem}$ are greater than the values of $pu{n}_{disk}$ and $pu{n}_{net}$, then ARP-TSCP will have a lower SLAV penalty cost. The advantage will be more obvious.

Figure 14 demonstrates the comparison of the number of VM migrations triggered by all methods in 288 consecutive periods in CDC. Figure 15 demonstrates the comparison of the total number of VM migrations caused by using these ten methods to consolidate servers throughout the day. As can be seen from Figure 14, in different periods, the number of VM migrations induced by ARP-TSCP is sometimes higher than that induced by the baseline methods, and it is sometimes lower than that induced by them. Therefore, as shown in Figure 15, the total amount of VM migrations induced by ARP-TSCP does not have much advantage over that of induced by the baseline methods throughout the day, and the two are very close.

In each period, ARP-TSCP and the baseline methods trigger VM migration for different reasons, respectively. ARP-TSCP actively considers the situation of overloaded hosts and SLAV occurrence in the next period and initiates VM migrations to deal with it in advance. The baseline methods passively respond to the host overloaded phenomenon generated in the current period. The total numbers of VM migrations are very close, but the mechanisms of triggering migration are different. Such differences are reflected in the final result, which is that the amount of SLAV occurrences caused by ARP-TSCP is much smaller than that caused by LR-MMT, as shown in Figure 13.

Compared with the baseline methods, our proposed ARP-TSCP has better improvements in host energy consumption, SLAV occurrence, and VM migration cost. Therefore, in the final total cost comparison, ARP-TSCP outperforms the other methods. As shown in Figure 16, compared to LR-MMT (the best result of all baseline methods), ARP-TSCP reduces the CDC operating cost by about 26.1%, and compared to MAD-RS (the worst result of all baseline methods), ARP-TSCP reduces the CDC operating cost by about 39.3%. This is because the comprehensive performance of ARP-TSCP is better than that of the baseline methods in all aspects.

6. Conclusions

In this paper, we focus on reducing CDC operating costs while ensuring user SLA requirements. We first established a cost model based on multiple computing resources in the CDC, taking into account the host energy cost, VM migration cost, and SLAV penalty cost. Based on this model, we define the MCRC problem in server consolidation. We devised the following step-by-step solution to deal with this problem. We employ a convolutional autoencoder-based filter to preprocess the workload of VMs and to reduce noise in the sampling record. Subsequently, an attention-based RNN method is used to predict the usage of computing resources by VMs in the future period, allowing us to trigger VM migration before the host enters the overloaded state to reduce the occurrence of SLAV. We design a heuristic algorithm based on the complementary use of multiple resources in space and time, ARP-TSCP, to solve the VM placement problem. Finally, simulations driven by real VM workload traces verify the effectiveness of our proposed method. Compared with the existing server consolidation methods, ARP-TSCP can reduce host energy consumption from 18.5% to 30.3%, SLAV cost by 38% to 52%, and total cost by 26.1% to 39.3%.

In the future, we will first consider a more comprehensive cost model, such as taking into account multi-core CPU, the network topology of CDC, use of network equipment, and cooling system. We will also design a forecasting method that can predict the resource usage of equipment over multiple time periods in the future, thereby further reducing the generation of SLAV.