Energy Aware Virtual Machine Scheduling in Data Centers

Power consumption is a primary concern in modern servers and data centers. Due to varying in workload types and intensities, different servers may have a different energy efficiency (EE) and energy proportionality (EP) even while having the same hardware configuration (i.e., central processing unit (CPU) generation and memory installation). For example, CPU frequency scaling and memory modules voltage scaling can significantly affect the server’s energy efficiency. In conventional virtualized data centers, the virtual machine (VM) scheduler packs VMs to servers until they saturate, without considering their energy efficiency and EP differences. In this paper we propose EASE, the Energy efficiency and proportionality Aware VM SchEduling framework containing data collection and scheduling algorithms. In the EASE framework, each server’s energy efficiency and EP characteristics are first identified by executing customized computing intensive, memory intensive, and hybrid benchmarks. Servers will be labelled and categorized with their affinity for different incoming requests according to their EP and EE characteristics. Then for each VM, EASE will undergo workload characterization procedure by tracing and monitoring their resource usage including CPU, memory, disk, and network and determine whether it is computing intensive, memory intensive, or a hybrid workload. Finally, EASE schedules VMs to servers by matching the VM’s workload type and the server’s EP and EE preference. The rationale of EASE is to schedule VMs to servers to keep them working around their peak energy efficiency point, i.e., the near optimal working range. When workload fluctuates, EASE re-schedules or migrates VMs to other servers to make sure that all the servers are running as near their optimal working range as they possibly can. The experimental results on real clusters show that EASE can save servers’ power consumption as much as 37.07%–49.98% in both homogeneous and heterogeneous clusters, while the average completion time of the computing intensive VMs increases only 0.31%–8.49%. In the heterogeneous nodes, the power consumption of the computing intensive VMs can be reduced by 44.22%. The job completion time can be saved by 53.80%.

traffic has been the catalyst for an approximately 4.5% increase in worldwide server shipment and data center deployment annually from 2011 to 2015 on average [1].Such an increase in server installation also contributes to the increase of energy consumption of data centers if the server energy efficiency remains.Nowadays, energy consumption of servers is becoming the major concern for data center operation and management [2].Figure 1 plots prediction on electricity consumption of data centers in U.S. from some data sources.
Energies 2019, 12, x 2 While server energy efficiency has improved much since last decade due to technical breakthroughs, the explosion of cloud computing, online services, social media networks, and Internet traffic has been the catalyst for an approximately 4.5% increase in worldwide server shipment and data center deployment annually from 2011 to 2015 on average [1].Such an increase in server installation also contributes to the increase of energy consumption of data centers if the server energy efficiency remains.Nowadays, energy consumption of servers is becoming the major concern for data center operation and management [2].Figure 1 plots prediction on electricity consumption of data centers in U.S. from some data sources.Although today's servers are more energy efficient than ever before, their power consumption under a low or idle workload accounts for more than 20% of the power at 100% workload for typical servers.Current data centers (DCs) of giant Internet companies like Google, Facebook, Microsoft, and Alibaba are highly energy efficient and have reached the power usage effectiveness (PUE) of 1.1 [6,7].However, for small-and medium-sized data centers, they are not always in a high-load operating state and thus the servers still consume a lot of energy when they are idle.These smalland medium-sized data centers account for more than 95% percentage of total installed servers [3,8].In such smaller scale data centers, server utilization is much lower than large scale data centers and their power use effectiveness (PUE) is also lower.Moreover, incoming requests are intermittent and workloads fluctuate on a daily or weekly basis.Therefore, energy proportional computing has emerged in both industry and academia.Energy proportionality (EP) is a proposed metric to measure the relationship of server power consumption and its utilization [9][10][11][12].Ideally, the power consumption of the server with EP = 1.0 should be fully proportionate to its utilization when the server is working.For example, the power at idle or low utilization ideally should also be almost zero or proportional compared to power at 100% workload.That is, the power consumption of the server with utilization of 10% should be one-tenth of its power consumption with utilization of 100%.In a typical server system, the processor is more energy proportional than memory and other components [13,14].Different hardware configuration may also affect the energy proportionality of the whole server system.Therefore, good knowledge of a server's energy proportionality and efficiency can help workload placement and job scheduling to achieve power and/or energy minimization in data centers.
Figure 2 gives the EP values and the SPECpower score of 509 commercial servers from the public valid SPECpower_ssj results ordered by hardware availability year (until 2017Q4 on 7 December 2017) [15].The authors will use SPECpower and SPECpower_ssj interchangeably in the remainder of this article.We can observe that the energy efficiency and EP of commercial servers Although today's servers are more energy efficient than ever before, their power consumption under a low or idle workload accounts for more than 20% of the power at 100% workload for typical servers.Current data centers (DCs) of giant Internet companies like Google, Facebook, Microsoft, and Alibaba are highly energy efficient and have reached the power usage effectiveness (PUE) of 1.1 [6,7].However, for small-and medium-sized data centers, they are not always in a high-load operating state and thus the servers still consume a lot of energy when they are idle.These small-and medium-sized data centers account for more than 95% percentage of total installed servers [3,8].In such smaller scale data centers, server utilization is much lower than large scale data centers and their power use effectiveness (PUE) is also lower.Moreover, incoming requests are intermittent and workloads fluctuate on a daily or weekly basis.Therefore, energy proportional computing has emerged in both industry and academia.Energy proportionality (EP) is a proposed metric to measure the relationship of server power consumption and its utilization [9][10][11][12].Ideally, the power consumption of the server with EP = 1.0 should be fully proportionate to its utilization when the server is working.For example, the power at idle or low utilization ideally should also be almost zero or proportional compared to power at 100% workload.That is, the power consumption of the server with utilization of 10% should be one-tenth of its power consumption with utilization of 100%.In a typical server system, the processor is more energy proportional than memory and other components [13,14].Different hardware configuration may also affect the energy proportionality of the whole server system.Therefore, good knowledge of a server's energy proportionality and efficiency can help workload placement and job scheduling to achieve power and/or energy minimization in data centers.
Figure 2 gives the EP values and the SPECpower score of 509 commercial servers from the public valid SPECpower_ssj results ordered by hardware availability year (until 2017Q4 on 7 December 2017) [15].The authors will use SPECpower and SPECpower_ssj interchangeably in the remainder of this article.We can observe that the energy efficiency and EP of commercial servers have improved in the last decade from the beginning of the first SPECpower_ssj benchmark results released in 2007.
For example, the average EP improved significantly from 0.32 to 0.91 (2007 to 2017); the average server energy efficiency improved from 489 to 11742 (ssj_ops per watt).
Energies 2019, 12, x 3 have improved in the last decade from the beginning of the first SPECpower_ssj benchmark results released in 2007.For example, the average EP improved significantly from 0.32 to 0.91 (2007 to 2017); the average server energy efficiency improved from 489 to 11742 (ssj_ops per watt).With improvements in energy efficiency and EP, recent commercial servers can achieve their peak energy efficiency at a lower utilization than 100%.Table 1 is a sample SPECpower_ssj result published in 2017Q4 [15].The server achieves peak energy efficiency 13,845 ssj_ops per watt at 60% utilization.The energy efficiency is 12,424 ssj_ops per watt at 100% utilization.Figure 3 illustrates the percentage of occurrences at peak energy efficiency per year.From 2004 to 2009, the servers achieved their peak energy efficiency at 100% utilization.The utilization spots for peak energy efficiency have varied since 2010.Most importantly, the latest manufactured servers tend to achieve peak energy efficiency at non-100% utilization.Among 30 published SPECpower_ssj results in 2017 (through 7 December 2017), 27 servers had hardware availability in 2017.Fourteen of the servers achieved peak energy efficiency at 70% utilization.Using VM scheduling and migration to keep the cluster server running at peak efficiency, we can improve the overall energy efficiency of the cluster and the whole data center.With improvements in energy efficiency and EP, recent commercial servers can achieve their peak energy efficiency at a lower utilization than 100%.Table 1 is a sample SPECpower_ssj result published in 2017Q4 [15].The server achieves peak energy efficiency 13,845 ssj_ops per watt at 60% utilization.The energy efficiency is 12,424 ssj_ops per watt at 100% utilization.Figure 3 illustrates the percentage of occurrences at peak energy efficiency per year.From 2004 to 2009, the servers achieved their peak energy efficiency at 100% utilization.The utilization spots for peak energy efficiency have varied since 2010.Most importantly, the latest manufactured servers tend to achieve peak energy efficiency at non-100% utilization.Among 30 published SPECpower_ssj results in 2017 (through 7 December 2017), 27 servers had hardware availability in 2017.Fourteen of the servers achieved peak energy efficiency at 70% utilization.Using VM scheduling and migration to keep the cluster server running at peak efficiency, we can improve the overall energy efficiency of the cluster and the whole data center.In data centers, servers are often provisioned and configured for peak workload and peak performance.Underutilized resources waste energy when working in lower utilization for non-energy proportional servers.Virtualization consolidation has been widely deployed in current data centers for service consolidation and energy reduction through resource multiplexing.In a multitenant cloud computing environment, cloud service providers pack as many VMs as possible to keep the physical server running at higher resource utilization regardless of the underlying server's EP.The packing policy masks the energy efficiency potentials of recent manufactured servers.In these cases, EP-blind scheduling wastes energy as it deteriorates system performance and service quality.
In this paper, we propose EASE, the energy efficiency and proportionality aware VM scheduling framework.Before scheduling, EASE identifies server's energy efficiency and EP by executing customized computing intensive, memory intensive, and hybrid benchmarks on the server.The rationale of EASE is to schedule VMs to servers to keep them working around their peak energy efficiency point, i.e., the near optimal working range.When workload fluctuates, EASE re-schedules or migrates VMs to other servers to make sure that all the servers are running as near their optimal working range as they possibly can.EASE can also calculate the global energy efficiency and energy savings of the data center through energy consumption data via an intelligent platform management interface (IPMI) enabled power data aggregation.Extensive experiments on a cluster were conducted, including computing intensive, memory intensive, and mixed workload VMs.The experimental results show that power consumption can save around 37.07%-49.98% in the homogeneous node cluster.The average completion time of the computing intensive workload increases 0.31%-8.49%.In the heterogeneous nodes, the power consumption of the computing intensive jobs can be reduced by 44.22%.In addition, the job completion time can be saved by 53.80%.The results presented in this article provide useful insight for the power and energy management of virtualized data centers, especially for heterogeneous servers.At the same time, EASE can serve as a reference for green data center operations based on server energy efficiency, EP, and workload characteristics.
The rest of the paper is organized as follows.In Section 2 we describe energy proportionality and the PEEP metric.In Section 3, we describe EASE, including its methodology and components for VM scheduling and migration.We provide experiment results, observations, and insights for different platforms with various configurations in Section 4. In Section 5 we review some related work on server energy proportionality and identify our unique approach and results.We conclude the paper with some future research directions in Section 6.In data centers, servers are often provisioned and configured for peak workload and peak performance.Underutilized resources waste energy when working in lower utilization for non-energy proportional servers.Virtualization consolidation has been widely deployed in current data centers for service consolidation and energy reduction through resource multiplexing.In a multitenant cloud computing environment, cloud service providers pack as many VMs as possible to keep the physical server running at higher resource utilization regardless of the underlying server's EP.The packing policy masks the energy efficiency potentials of recent manufactured servers.In these cases, EP-blind scheduling wastes energy as it deteriorates system performance and service quality.

SPECpower Benchmark
In this paper, we propose EASE, the energy efficiency and proportionality aware VM scheduling framework.Before scheduling, EASE identifies server's energy efficiency and EP by executing customized computing intensive, memory intensive, and hybrid benchmarks on the server.The rationale of EASE is to schedule VMs to servers to keep them working around their peak energy efficiency point, i.e., the near optimal working range.When workload fluctuates, EASE re-schedules or migrates VMs to other servers to make sure that all the servers are running as near their optimal working range as they possibly can.EASE can also calculate the global energy efficiency and energy savings of the data center through energy consumption data via an intelligent platform management interface (IPMI) enabled power data aggregation.Extensive experiments on a cluster were conducted, including computing intensive, memory intensive, and mixed workload VMs.The experimental results show that power consumption can save around 37.07%-49.98% in the homogeneous node cluster.The average completion time of the computing intensive workload increases 0.31%-8.49%.In the heterogeneous nodes, the power consumption of the computing intensive jobs can be reduced by 44.22%.In addition, the job completion time can be saved by 53.80%.The results presented in this article provide useful insight for the power and energy management of virtualized data centers, especially for heterogeneous servers.At the same time, EASE can serve as a reference for green data center operations based on server energy efficiency, EP, and workload characteristics.
The rest of the paper is organized as follows.In Section 2 we describe energy proportionality and the PEEP metric.In Section 3, we describe EASE, including its methodology and components for VM scheduling and migration.We provide experiment results, observations, and insights for different platforms with various configurations in Section 4. In Section 5 we review some related work on server energy proportionality and identify our unique approach and results.We conclude the paper with some future research directions in Section 6.

SPECpower Benchmark
Currently, the most authoritative industry standard organizations for performance measurements are the Transaction Processing Performance Council (TPC), the Standard Performance Evaluation Corporation (SPEC), and the Storage Performance Council (SPC).SPEC released the first industry standard benchmark, SPECPower_ssj, which measures power consumption related to performance for servers.Workload of SPECpower evaluates energy efficiency and performance running server-side Java applications for small-and medium-sized servers at graduated utilization levels.SPECpower reports power consumption for servers at different utilization levels (from 100% utilization to idle in 10% intervals) over a set time.The server's overall energy efficiency score is calculated by division of the sum of 11 average active power values (including idle) and the sum of 10 performances to power ratio values.Table 1, which is a sample result of SPECpower_ssj2008 published in 2017Q4, represents a server with hardware availability of Aug-2017.In a SPECpower result, the utilization is not the CPU utilization.The performance to power ratio at each target load level is often called the server's energy efficiency at that load level with unit of ssj_ops per watt.For the last column, performance to power ratio in Table 1, 12,424 is the energy efficiency at peak utilization (100%), 13,845 is the peak energy efficiency, and the server's overall energy efficiency score is 12,120.

Energy Proportionality Calculation And Its Implications
This article uses the EP metric proposed by Ryckbosch et al. in [10] as shown in Equation (1).
where: Area real is the area covered by a real server's power curve (green zone in Figure 4).Area real is the area covered by an ideal server (black solid line zone in Figure 4).
Energies 2019, 12, x 5 Currently, the most authoritative industry standard organizations for performance measurements are the Transaction Processing Performance Council (TPC), the Standard Performance Evaluation Corporation (SPEC), and the Storage Performance Council (SPC).SPEC released the first industry standard benchmark, SPECPower_ssj, which measures power consumption related to performance for servers.Workload of SPECpower evaluates energy efficiency and performance running server-side Java applications for small-and medium-sized servers at graduated utilization levels.SPECpower reports power consumption for servers at different utilization levels (from 100% utilization to idle in 10% intervals) over a set time.The server's overall energy efficiency score is calculated by division of the sum of 11 average active power values (including idle) and the sum of 10 performances to power ratio values.Table 1, which is a sample result of SPECpower_ssj2008 published in 2017Q4, represents a server with hardware availability of Aug-2017.In a SPECpower result, the utilization is not the CPU utilization.The performance to power ratio at each target load level is often called the server's energy efficiency at that load level with unit of ssj_ops per watt.For the last column, performance to power ratio in Table 1, 12,424 is the energy efficiency at peak utilization (100%), 13,845 is the peak energy efficiency, and the server's overall energy efficiency score is 12,120.

Energy Proportionality Calculation And Its Implications
This article uses the EP metric proposed by Ryckbosch et al. in [10] as shown in Equation 1.
Where:  is the area covered by a real server's power curve (green zone in Figure 4).

𝐴𝑟𝑒𝑎
is the area covered by an ideal server (black solid line zone in Figure 4).  1 and an ideally energy proportional server (power normalized to power at 100% utilization).
We can approximate the EP value of the server as in Table 1 by calculating 10 subareas (10 utilization level from 10% to 100%) in its power-utilization curve in Figure 4. Without loss of generality, we can estimate any server's EP value given its power curve.From equation 1 and Figure 4 we can see that higher EP means that: (1) the server has lower idle power; (2) it consumes less power than an ideal server at specific utilization level; or (3) both.
In our previous work [4], we identified the relationship between energy efficiency, EP, and processor architecture, hardware configuration, and server performance.Moreover, for servers with higher EP (EP > 0.8), the non-linearity of energy consumption corresponding to its performance is becoming more dominant.At the same time, the energy efficiency curve of the server is nonlinear.1 and an ideally energy proportional server (power normalized to power at 100% utilization).
We can approximate the EP value of the server as in Table 1 by calculating 10 subareas (10 utilization level from 10% to 100%) in its power-utilization curve in Figure 4. Without loss of generality, we can estimate any server's EP value given its power curve.From Equation ( 1) and Figure 4 we can see that higher EP means that: (1) the server has lower idle power; (2) it consumes less power than an ideal server at specific utilization level; or (3) both.In our previous work [4], we identified the relationship between energy efficiency, EP, and processor architecture, hardware configuration, and server performance.Moreover, for servers with higher EP (EP > 0.8), the non-linearity of energy consumption corresponding to its performance is becoming more dominant.At the same time, the energy efficiency curve of the server is nonlinear.
Figure 3 shows that the peak energy efficiency spot from the latest server has shifted from 100% utilization to lower utilization.Table 2 lists the statistics of peak energy efficiency occurrence of all 509 servers with SPECpower_ssj results since 2007.Although 334 servers achieve their peak energy efficiency at 100% utilization, 95.2% of these servers (i.e., 318 servers) were manufactured before 2013.Only 4.8% (i.e., 16 servers) were manufactured between 2013 and 2018.The EP curves from 509 servers show that: (1) For all 334 servers that achieve peak energy efficiency at 100% utilization, their EP curves are always above the EP curve of the ideal server.(2) For the remaining 175 servers that achieve peak energy efficiency at non-100% utilization, six servers are always above the ideal server, 113 servers intersect with the ideal curve once, and 56 servers intersect with the ideal curve twice.Their energy efficiency and EP are listed in Table 3.We give some representative EP curves in Figure 5. From Figures 4 and 5 we derive two implications as follows: • Implication #1: For a server whose power curve intersects with the ideal power curve before 100% utilization, there exists at least one work range where its energy efficiency is larger than the ideal server.This is defined as an optimal working range (OWR).
When in the utilization interval (40%, 90%), the server in Table 1 has an energy efficiency which is larger than an ideal server (see Figure 4).Table 4 lists its power and percentage compared to an ideal server.For example, at 50% utilization, its power consumption normalized to 100% utilization is 45.7%.For an ideal server, its power consumption at 50% utilization should also be 50%.The working range in Table 4 identified by the EP curve is the range that the server should work in. Implication #2: The server achieves its peak energy efficiency at the utilization spot where its power is the lowest normalized to ideal server.Table 4 shows that the server at 60% utilization consumes only 90% power compared to an ideal server, which is the lowest during all working range.Therefore, the server achieves its peak energy efficiency at 60% utilization (see Table 1).

The PEEP Metric
The PEEP metric is the ratio of peak energy efficiency over energy efficiency at peak utilization [16].For example, the PEEP value of the server with SPECpower results in Table 1 is 13,845/12,424 = 1.114.The PEEP value indicates the offset of a server's peak energy efficiency with respect to its energy efficiency at 100% utilization.PEEP values equal to 1.0 mean that the server performs peak energy efficiency at peak utilization (100% utilization).A PEEP value larger than 1.0 means that the server performs peak energy efficiency other than 100% utilization.Table 5 lists the counts of servers with a different utilization spot where peak energy efficiency occurs.Among 175 servers with PEEP > 1.0, 81 servers achieve their peak energy efficiency at 70% utilization.• Implication #2: The server achieves its peak energy efficiency at the utilization spot where its power is the lowest normalized to ideal server.
Table 4 shows that the server at 60% utilization consumes only 90% power compared to an ideal server, which is the lowest during all working range.Therefore, the server achieves its peak energy efficiency at 60% utilization (see Table 1).

The PEEP Metric
The PEEP metric is the ratio of peak energy efficiency over energy efficiency at peak utilization [16].For example, the PEEP value of the server with SPECpower results in Table 1 is 13,845/12,424 = 1.114.The PEEP value indicates the offset of a server's peak energy efficiency with respect to its energy efficiency at 100% utilization.PEEP values equal to 1.0 mean that the server performs peak energy efficiency at peak utilization (100% utilization).A PEEP value larger than 1.0 means that the server performs peak energy efficiency other than 100% utilization.Table 5 lists the counts of servers with a different utilization spot where peak energy efficiency occurs.Among 175 servers with PEEP > 1.0, 81 servers achieve their peak energy efficiency at 70% utilization.Figure 6 draws the average and median values of PEEP, EP, and the SPECpower score of the 509 servers.These are grouped by their utilization spot where peak energy efficiency occurs.Servers achieving their peak energy efficiency at lower utilization have higher PEEP and EP values than those at higher utilization.Figure 7 presents the EP and PEEP value for the 175 servers whose PEEP values are larger than 1.0.Figure 7 can be quantified as 0.85 based on Equation 2: Figure 7 also suggests that servers whose peak energy efficiencies do not coincide at 100% utilization will have higher EP values, higher PEEP values, and vice versa.
Equations 4 and 5 show that power saving may correlate with a server's EP. Figure 8 shows the 175 servers whose PEEP values are larger than 1.0.This can be quantified as 0.84 by Equation 3: This article shows that servers should perform in an optimal working range.However, for a data center with thousands of servers, it is more important to select which server to run multiple candidates, especially when the candidate servers have optimal working ranges.Equations ( 2)- (4) show that choosing a server with higher EP and PEEP values can save more power.This is the basic rationale of workload placement and scheduling in data centers with heterogeneous servers whose EP is also heterogeneous.      Figure 7 presents the EP and PEEP value for the 175 servers whose PEEP values are larger than 1.0.Figure 7 can be quantified as 0.85 based on Equation 2: Figure 7 also suggests that servers whose peak energy efficiencies do not coincide at 100% utilization will have higher EP values, higher PEEP values, and vice versa.
Equations 4 and 5 show that power saving may correlate with a server's EP.
Figure 8 shows the 175 servers whose PEEP values are larger than 1.0.This can be quantified as 0.84 by Equation 3: This article shows that servers should perform in an optimal working range.However, for a data center with thousands of servers, it is more important to select which server to run multiple candidates, especially when the candidate servers have optimal working ranges.Equations ( 2)- (4) show that choosing a server with higher EP and PEEP values can save more power.This is the basic rationale of workload placement and scheduling in data centers with heterogeneous servers whose EP is also heterogeneous. Figure 7 also suggests that servers whose peak energy efficiencies do not coincide at 100% utilization will have higher EP values, higher PEEP values, and vice versa.
Equations ( 4) and (5) show that power saving may correlate with a server's EP. Figure 8 shows the 175 servers whose PEEP values are larger than 1.0.This can be quantified as 0.84 by Equation (3):

Energy Efficiency and Proportionality Aware Virtual Machine Scheduling
We provide the energy aware virtual machine scheduling flow chart in EASE in Figure 9.

Server Energy Efficiency Identification
Hardware configuration of different servers in the data center is usually heterogeneous.The operating system and workloads on each server also differ.The server's energy efficiency differs when running workloads under different configurations, including processor DVFS, memory bank powering mode, NIC transmitting settings, etc. [17].To implement the energy efficiency and proportionality aware VM scheduling, we must understand the energy efficiency and proportionality of each server in data centers.Although SPECpower_ssj score is an important reference source, not all servers in a data center have their own SPECpower_ssj score.Moreover, data center workload is very diverse and dynamic.This is not represented comprehensively by the SPECpower_ssj benchmark.Therefore, the authors have developed a customized benchmark for server energy efficiency identification.It has the following merits: This article shows that servers should perform in an optimal working range.However, for a data center with thousands of servers, it is more important to select which server to run multiple candidates, especially when the candidate servers have optimal working ranges.Equations ( 2)-( 4) show that choosing a server with higher EP and PEEP values can save more power.This is the basic rationale of workload placement and scheduling in data centers with heterogeneous servers whose EP is also heterogeneous.

Energy Efficiency and Proportionality Aware Virtual Machine Scheduling
We provide the energy aware virtual machine scheduling flow chart in EASE in Figure 9.

Energy Efficiency and Proportionality Aware Virtual Machine Scheduling
We provide the energy aware virtual machine scheduling flow chart in EASE in Figure 9.

Server Energy Efficiency Identification
Hardware configuration of different servers in the data center is usually heterogeneous.The operating system and workloads on each server also differ.The server's energy efficiency differs when running workloads under different configurations, including processor DVFS, memory bank powering mode, NIC transmitting settings, etc. [17].To implement the energy efficiency and proportionality aware VM scheduling, we must understand the energy efficiency and proportionality of each server in data centers.Although SPECpower_ssj score is an important reference source, not all servers in a data center have their own SPECpower_ssj score.Moreover, data center workload is very diverse and dynamic.This is not represented comprehensively by the SPECpower_ssj benchmark.Therefore, the authors have developed a customized benchmark for

Server Energy Efficiency Identification
Hardware configuration of different servers in the data center is usually heterogeneous.The operating system and workloads on each server also differ.The server's energy efficiency differs when running workloads under different configurations, including processor DVFS, memory bank powering mode, NIC transmitting settings, etc. [17].To implement the energy efficiency and proportionality aware VM scheduling, we must understand the energy efficiency and proportionality of each server in data centers.Although SPECpower_ssj score is an important reference source, not all servers in a data center have their own SPECpower_ssj score.Moreover, data center workload is very diverse and dynamic.This is not represented comprehensively by the SPECpower_ssj benchmark.Therefore, the authors have developed a customized benchmark for server energy efficiency identification.It has the following merits: (1) Representative of Data Center Workloads: We use SPECpower_ssj, PrimeSearch [17], STREAM [18], and their mixture.The workload is also proportional to the server's hardware configuration.It varies with multiple workload intensities to simulate a real multitenant virtualized cloud computing environment.(2) Easy Implementation: Recent commercial servers are equipped with IPMI support through which we get the real time power consumption from embedded sensors in a server mainboard.This includes system power, processor power, and memory bank power.With simulation workload-and IPMI-based power consumption, we can get the server's energy efficiency and proportionality under different workload types and workload intensity levels.In the authors' implementation, IPMI-based power data acquisition has negligible overhead on the physical server's system utilization and extra power consumption (< 0.25 watt).
For each server, we find the energy efficiency and proportionality by running computing intensive, memory intensive, and hybrid workload, respectively.It is a single synthetic benchmark program to measure the amount of memory and corresponding computational rate of the simple vector kernel.The CPU of the computer system runs much faster than memory.As this progresses, more program performance will be limited to memory bandwidth rather than CPU computing performance.(3) Hybrid Workload: The authors use SPECpower_ssj2008 as the hybrid workload to stress the server's components, including CPU and memory synthetically.

VM Workload Characterization
Previous sections discussed how each server may have different energy efficiency and proportionality.In addition, each one must work in its own optimal working range.Therefore, it is necessary to determine each VM's workload type before scheduling it to the target physical server.In doing so, we can ensure that it is running in the optimal working range.
It is necessary to identify the workload characteristics of the VM.It is impossible for a VM to consume only CPU resources without memory resources in a real cloud computing environment.Therefore, an empirical setting for VM type identification is used in Table 6.This setting is based on empirical observations and experiences in testing environments and real data sets [19].They can be updated by customized parameters.In the future version of EASE, the scheduler could learn and adjust it according to the actual workload trace data in data centers.The workload characterization procedure is done by the system profiling techniques like sysstat tool to collect the CPU and memory usage which indicates the workload type and intensity.For each VM, EASE will undergo workload characterization procedure by tracing and monitoring their resource usage including CPU, memory, disk, and network and determine whether it is computing intensive, memory intensive, or hybrid workload.

Virtual Machine Scheduling and Migration
In traditional data centers, VMs are packaged into many servers.The underutilized servers are powered off.As described, the latest manufactured servers achieve their peak energy efficiency at non-100% utilization.The effectiveness of reducing power consumption by increasing resource utilization will become increasingly unsatisfactory.Moreover, servers with low EP can be grouped together to run at their optimal working range to achieve better EP in cluster wide.Therefore, in EASE, we schedule VMs to servers with the best optimal working range.
EASE contains the following components: 1.
Server Monitoring and Power Data Collection: CPU and memory contribute to most of the dynamic power consumption in a server.Therefore, the monitoring module of EASE is to collect real-time power consumption, CPU utilization, memory utilization, and other system access and activities statistics.The physical server provides the physical machine CPU, memory, input/output (I/O), and other hardware resource utilization using the sysstat utility.

2.
Benchmark Execution: EASE runs benchmark tests on dedicated servers under multiple software and hardware configurations.It also collects performance data.

3.
Energy Efficiency and Proportionality Calculation: Based on system monitoring data and benchmark execution results, the energy efficiency and proportionality are calculated in real time.

4.
VM Scheduling: When EASE gets each server's current running status and optimal working range, it determines if the server is overloaded or underloaded.For new VMs, EASE looks for the appropriate target physical server to keep them working in an optimal working range.For existing VMs, EASE first characterizes the machine's workload type (i.e., computing intensive, memory intensive, or hybrid) and attempts to migrate the VM to an appropriate target physical server.Servers with different hardware and software configurations have different energy efficiency and proportionality.Therefore, if there are multiple servers with different resources (i.e., resource type, capacity, and configuration), EASE first sorts the servers with available resources by resource type in descending order of energy efficiency, EP, and PEEP values.Next, it selects a target host according to the priority that the target host can reach the peak energy efficiency or energy efficiency will be higher in its optimal working range.
The VM scheduling algorithm is listed in Figure 10.We assume that the scheduler can always find a target server receiving VM with enough free CPUs, memory, and disk to run the migrated VM.
To ensure that the data center is running in optimal working range, EASE monitors and evaluates all the servers.As it checks that it is running in its optimal working range, it may migrate VMs to other servers.
Live migration is perfectly implemented in state-of-the-art VM monitoring.Therefore, the authors did not consider VM migration overheads.Moreover, the migration is only conducted inside a data center network.The quality of service-related issues (i.e., service interruption and data dependence redirection) are negligible.
The server is overloaded if it is not running in its optimal working range and nears the upper bound of its optimal working range.The VM on the overloaded server will migrate until the server returns to its optimal working range.Similarly, the server is underloaded if it is not running in its optimal working range and nears the lower bound of its optimal working range.The newly created VMs or to-be-migrated VMs will be scheduled to the underloaded server until the server is running in its optimal working range.
resources by resource type in descending order of energy efficiency, EP, and PEEP values.Next, it selects a target host according to the priority that the target host can reach the peak energy efficiency or energy efficiency will be higher in its optimal working range.
The VM scheduling algorithm is listed in Figure 10.We assume that the scheduler can always find a target server receiving VM with enough free CPUs, memory, and disk to run the migrated VM.For power minimization, the first goal is to migrate out VMs on overloaded servers.This ensures that there is no overloaded server.The second goal is to keep servers running in optimal working range.If there are multiple underloaded servers, we merge VMs on underloaded servers to use as few as possible servers.
Figure 11 lists the VM migration algorithm.The rationale to migrate the VM with the largest resource requirement on an overloaded server is to get it back to optimal working range as soon as possible with minimal migration operations.For example, if we migrate a VM with smallest resource requirement, the server may still be in overloaded status.After migration, EASE updates the original server status information.It will continue the above process until there is no overloaded server.The following section conducted experiments on real clusters with EASE-enabled VM scheduling.
Energies 2019, 12, x 12 To ensure that the data center is running in optimal working range, EASE monitors and evaluates all the servers.As it checks that it is running in its optimal working range, it may migrate VMs to other servers.
Live migration is perfectly implemented in state-of-the-art VM monitoring.Therefore, the authors did not consider VM migration overheads.Moreover, the migration is only conducted inside a data center network.The quality of service-related issues (i.e., service interruption and data dependence redirection) are negligible.
The server is overloaded if it is not running in its optimal working range and nears the upper bound of its optimal working range.The VM on the overloaded server will migrate until the server returns to its optimal working range.Similarly, the server is underloaded if it is not running in its optimal working range and nears the lower bound of its optimal working range.The newly created VMs or to-be-migrated VMs will be scheduled to the underloaded server until the server is running in its optimal working range.
For power minimization, the first goal is to migrate out VMs on overloaded servers.This ensures that there is no overloaded server.The second goal is to keep servers running in optimal working range.If there are multiple underloaded servers, we merge VMs on underloaded servers to use as few as possible servers.
Figure 11 lists the VM migration algorithm.The rationale to migrate the VM with the largest resource requirement on an overloaded server is to get it back to optimal working range as soon as possible with minimal migration operations.For example, if we migrate a VM with smallest resource requirement, the server may still be in overloaded status.After migration, EASE updates the original server status information.It will continue the above process until there is no overloaded server.The following section conducted experiments on real clusters with EASE-enabled VM scheduling.

Experimental Platform
In order to evaluate EASE's feasibility and performance, this article conducts experiments on the following platforms (see Table 7).The physical platform and the VM run CentOS 7.2; the hypervisor is KVM/QEMU.Each VM is configured with two vCPUs, 4GB memory, and 20GB hard disk space.We developed a simulator to evaluate our algorithm's scalability in our work in

Experimental Platform
In order to evaluate EASE's feasibility and performance, this article conducts experiments on the following platforms (see Table 7).The physical platform and the VM run CentOS 7.2; the hypervisor is KVM/QEMU.Each VM is configured with two vCPUs, 4GB memory, and 20GB hard disk space.We developed a simulator to evaluate our algorithm's scalability in our work in progress.Due to space limitations, we only provide experiments on three machines in our lab.However, EASE can scale well in large data centers with hundreds or thousands of servers for two reasons.(1) EASE can identify server's energy efficiency and energy proportionality offline and in advance.This process only runs once and can be done automatically after server deployment before service in production.

Experimental Results
Homogeneous Cluster.Since the hardware configurations of the homogeneous servers are the same, the server's energy efficiency and proportionality are the same for the same workload.For example, Figure 12 shows the power consumption when running different workloads on server #3.

Experimental Results
Homogeneous Cluster.Since the hardware configurations of the homogeneous servers are the same, the server's energy efficiency and proportionality are the same for the same workload.For example, Figure 12 shows the power consumption when running different workloads on server #3.
It is important to note that when 12 VMs are running STREAM workload concurrently on server #3, the power consumption (138 watts) is lower than when eight VMs are running (each VM continues running STREAM load).This is mainly due to overheads introduced when the kernel virtual machine (KVM) is in memory virtualization, which results in a sharp decline in the efficiency of memory virtualization.A large number of STREAM computing tasks are blocked, resulting in multiple VMs in an idle and waiting state.This eventually leads to reduced server power consumption.This can be observed from the completion time of the STREAM in Figures 13-15.It is important to note that when 12 VMs are running STREAM workload concurrently on server #3, the power consumption (138 watts) is lower than when eight VMs are running (each VM continues running STREAM load).This is mainly due to overheads introduced when the kernel virtual machine (KVM) is in memory virtualization, which results in a sharp decline in the efficiency of memory virtualization.A large number of STREAM computing tasks are blocked, resulting in multiple VMs in an idle and waiting state.This eventually leads to reduced server power consumption.This can be observed from the completion time of the STREAM in Figures 13-15.
Energies 2019, 12, x 14      The shorter STREAM calculation time (calculation time of single Copy, Add, Scale, or Triad are generally not longer than one minute and the total run time is generally 10 min) is significantly less than the PrimeSearch completion time (3)(4)(5)(6)(7)(8)(9)(10)(11)(12) VMs is about 1620 sec, 24 VMs is 2980 sec).Even in the concurrent running of VMs with STREAM and PrimeSearch, the server power consumption remains significantly less than the sum of the power consumption of two servers running separately.
Figure 16 shows the maximum memory access bandwidth for STREAM running inside each VM while running STREAM separately and running STREAM and PrimeSearch concurrently.The shorter STREAM calculation time (calculation time of single Copy, Add, Scale, or Triad are generally not longer than one minute and the total run time is generally 10 min) is significantly less than the PrimeSearch completion time (3)(4)(5)(6)(7)(8)(9)(10)(11)(12) VMs is about 1620 sec, 24 VMs is 2980 sec).Even in the concurrent running of VMs with STREAM and PrimeSearch, the server power consumption remains significantly less than the sum of the power consumption of two servers running separately.Energies 2019, 12, x 15 Figure 16 shows the maximum memory access bandwidth for STREAM running inside each VM while running STREAM separately and running STREAM and PrimeSearch concurrently.
Table 8 shows the homogeneous node cluster's power consumption and performance comparison using EASE.The experimental results on server #3 show that power consumption can achieve 37.07%-49.98%savings in both the computing intensive and memory intensive workload.
The average completion time of the computing intensive workload only increases from 0.31% to 8.49%.Although the completion time of the memory intensive workload is deteriorated (increased 4.8% to 67.8%) in most cases (except the case of eight PrimeSearch VMs and eight STREAM VMs concurrently), the maximum completion time reduced by 7.96% compared to STREAM alone.The reason may be that when all VMs run the same memory intensive workloads, the hypervisor's memory virtualization and page replacement performance deteriorate due to a different ratio between VM numbers and physical processor count.Heterogeneous Cluster.Figures 17 and 18 demonstrate how runtime configuration affects the server's energy efficiency using server #1 and server #3 under different processor frequency.As can be seen from these figures, the EP, peak power consumption, and peak EP of the three heterogeneous servers are different (i.e., server #3 has higher EP and EE than the others).It should be noted that Figures 17 and 18 do not suggest that only the CPU impacts a server's energy efficiency and proportionality.Instead, these show that runtime configuration, like frequency scaling, can affect server's energy efficiency and proportionality.
Tables 9 and 10 show the power consumption of different scheduling algorithms.They also show the average completion time of a PrimeSearch workload on the heterogeneous virtualized Table 8 shows the homogeneous node cluster's power consumption and performance comparison using EASE.The experimental results on server #3 show that power consumption can achieve 37.07%-49.98%savings in both the computing intensive and memory intensive workload.The average completion time of the computing intensive workload only increases from 0.31% to 8.49%.Although the completion time of the memory intensive workload is deteriorated (increased 4.8% to 67.8%) in most cases (except the case of eight PrimeSearch VMs and eight STREAM VMs concurrently), the maximum completion time reduced by 7.96% compared to STREAM alone.The reason may be that when all VMs run the same memory intensive workloads, the hypervisor's memory virtualization and page replacement performance deteriorate due to a different ratio between VM numbers and physical processor count.
Heterogeneous Cluster.Figures 17 and 18 demonstrate how runtime configuration affects the server's energy efficiency using server #1 and server #3 under different processor frequency.As can be seen from these figures, the EP, peak power consumption, and peak EP of the three heterogeneous servers are different (i.e., server #3 has higher EP and EE than the others).It should be noted that Figures 17 and 18 do not suggest that only the CPU impacts a server's energy efficiency and proportionality.Instead, these show that runtime configuration, like frequency scaling, can affect server's energy efficiency and proportionality.
Energies 2019, 12, x 16 cluster consisting of the three types of servers.The power consumption can be saved by 44.22%; the task completion time can be saved by 53.80%.After using EASE, the power consumption saving is 48.27%; task completion time saving is 30.49%.cluster consisting of the three types of servers.The power consumption can be saved by 44.22%; the task completion time can be saved by 53.80%.After using EASE, the power consumption saving is 48.27%; task completion time saving is 30.49%.Tables 9 and 10 show the power consumption of different scheduling algorithms.They also show the average completion time of a PrimeSearch workload on the heterogeneous virtualized cluster consisting of the three types of servers.The power consumption can be saved by 44.22%; the task completion time can be saved by 53.80%.After using EASE, the power consumption saving is 48.27%; task completion time saving is 30.49%.As presented in Table 10, EASE schedules the VM to the server with high energy efficiency and high EP.Although it costs 10.48% more time than packing scheduling, the power consumption is reduced by 45.53%.Therefore, EASE can be applied in a virtualized data center with limited power consumption or power capping.
Table 11 shows the performance of EASE for mixed workload cases.EASE saves power by 46.90% and 7.67% compared to the initial (without EASE) and packing scheduling, respectively.It saves completion time by 52.29% and 0.27% compared to the initial (without EASE) algorithm and packing scheduling, respectively.For the memory-based workload, EASE reduces the maximum bandwidth by −10.70% and −0.34% compared to the initial (without EASE) and the packing scheduling of STREAM, respectively.As for average completion time, EASE is increased by 40.9% and 4.64% compared to the initial (without EASE) and intermediate scheduling of STREAM,

Work
Thanks to rapid development and technological breakthroughs of hardware, server energy efficiency has significantly improved.However, in many scenarios, dynamic voltage frequency scaling (DVFS) and dynamic power management (DPM) are still used for single server's power aware adaptation and performance tuning [20][21][22][23].Those approaches dynamically adjust the running state for a single resource (i.e., CPU or memory) or focus on single workloads.They ignore the interdependence of different resources and their impact on server performance.In addition, they ignore power consumption when running different types of applications.
For data center power management, the main goal is to coordinate all resources for multiplexing and service provisioning.In a virtualized data center, it is more challenging to initiate DVFS because VM has no knowledge of underlying hardware chips although virtualization enables service consolidation and power reduction.The power and energy aware scheduler must balance the overall system energy consumption and performance.It must also improve resource utilization while trying to satisfy service level agreements (SLAs).Genetic algorithm, Ant Colony algorithm, linear programming, adaptive heuristics, and utility-based approaches are proposed for resource scheduling and VM migration [24][25][26][27][28][29][30][31][32][33][34].For typical air-cooled data centers, researchers propose thermal-aware workload allocation strategy with respect to the chip temperature constraint [35] and use computational fluid dynamics (CFD) to model and validate Airflow in a Data Center [36].However, state-of-the-art VM scheduling approaches do not consider the server's energy efficiency and EP.
Server's power consumption varies when its utilization level changes.Moreover, a server consumes large amount of power even when it's idle.Thus energy proportional computing is proposed.Energy proportionality typically refers to that the system's power consumption changes with the workload level.The research results in [37] show that it is difficult to achieve the energy proportional computing on a single server.The reason is that CPUs, RAMs, graphics cards and other hardware on a single server consume about 70% of peak energy consumption even in idle state [38].Therefore, energy proportional computing is more achievable in data center scale where multiple servers are located together for service provisioning.
The latest commercial servers achieve their peak energy efficiency at non-100% utilization.Therefore, it is possible to save more energy when the server is underutilized compared to the 100% utilization case [38][39][40][41].Moreover, efficiency in hypervisor itself can also impact virtualized server's energy efficiency, such as inter-domain communication performance [42].However, EP requires more exploration for energy efficiency and proportionality-aware VM scheduling.
This paper proposes EASE, the energy efficiency and proportionality-aware VM scheduling, for virtualized cloud data centers.EASE first identifies server energy efficiency and EP by running customized and standard benchmarks.It schedules VMs to servers with the highest energy efficiency or the servers with the highest PEEP value to save more power and energy.Our work is based on the energy efficiency and energy proportionality characteristics of each server in data centers.And we schedule virtual machines in virtualized environment rather than jobs to physical servers.Our core contribution is that we integrate the energy efficiency and energy proportionality awareness in the scheduling framework.According the real benchmarking results, our algorithms outperform related work in energy minimization and performance optimization.

Conclusions and Future Work
A typical data center contains heterogeneous servers manufactured in different generations from different vendors.However, the energy efficiency and proportionality-aware scheduling approach mask heterogeneity, the complexity of hardware configuration, and performance tuning.The authors propose EASE to schedule VMs according to workload characteristics, energy efficiency, and proportionality of servers in a data center.EASE keeps the servers running in their optimal range in terms of energy efficiency through VM scheduling and migration.It can provide server consolidation and ensure the quality of service guarantee of multi-tenants.In addition, it can reduce energy consumption in virtualized data centers while not sacrificing significant application performance inside VMs.Experiments on real homogeneous clusters show that EASE achieves 37.07%-49.98%power consumption savings for computing and memory intensive workloads.The average completion time of the workload is only increased by 0.31% to 8.49%.Experiments on heterogeneous clusters show that the power consumption of the computing intensive workload can be reduced by 44.22%; the task completion time can be saved by 53.80%.
The use of EASE on recent server platforms equipped with power management options (i.e., IPMI, intelligent power distribution cabinet with power capping PDM or PDU) shows that it can be easily integrated into existing power management systems to deploy energy efficiency-aware VM scheduling.For legacy servers without IPMI capability, EASE may function well if it can read power measurements from power meters in real time if the power meter has real time data acquisition function.We are developing an energy efficiency simulator for data centers to evaluate our algorithm's scalability [43].Moreover, in the future, we also would like to adapt EASE to diverse workload types by self-learning workload characterization in data centers.

Figure 1 .
Figure 1.Prediction on electricity consumption of data centers in U.S. Data extrapolated on [1,3-5].GeSI SMARTer 2020 data is extrapolated by 7% annual increase rate of Global data centers emission (GtCO2e) with the baseline of NRDC 2011 data.

Figure 1 .
Figure 1.Prediction on electricity consumption of data centers in U.S. Data extrapolated on [1,3-5].GeSI SMARTer 2020 data is extrapolated by 7% annual increase rate of Global data centers emission (GtCO2e) with the baseline of NRDC 2011 data.

Figure 2 .
Figure 2. Energy Proportionality (EP) and energy efficiency (EE) trend of commercial servers.

Figure 2 .
Figure 2. Energy Proportionality (EP) and energy efficiency (EE) trend of commercial servers.

Figure 3 .
Figure 3. Distribution of server utilization spot with peak energy efficiency.

Figure 3 .
Figure 3. Distribution of server utilization spot with peak energy efficiency.

Figure 4 .
Figure 4. Comparison of EP curve (or power curve) of the server in Table1and an ideally energy proportional server (power normalized to power at 100% utilization).

Energies 2019, 12 , x 8 Figure 6 .
Figure 6.PEEP, EP, and EE with respect to utilization where peak energy efficiency occurs.

Figure 7 .
Figure 7. Scattering of PEEP and EP.

Figure 7
Figure 7 presents the EP and PEEP value for the 175 servers whose PEEP values are larger than 1.0.Figure 7 can be quantified as 0.85 based on Equation (2): Figure 7 presents the EP and PEEP value for the 175 servers whose PEEP values are larger than 1.0.Figure 7 can be quantified as 0.85 based on Equation (2):

Figure 6 .
Figure 6.PEEP, EP, and EE with respect to utilization where peak energy efficiency occurs.

Figure 8 .
Figure 8. Scattering of power saving and EP.

Figure 9 .
Figure 9. Flow chart of energy aware virtual machine scheduling.

Figure 8 .
Figure 8. Scattering of power saving and EP.

Figure 9 .
Figure 9. Flow chart of energy aware virtual machine scheduling.

Figure 9 .
Figure 9. Flow chart of energy aware virtual machine scheduling.

Figure 17 .
Figure 17.Server #1 energy efficiency and proportionality under different frequency scaling configurations.

Figure 18 .
Figure 18.Server #3 energy efficiency and proportionality under different frequency scaling configurations.

Figure 17 .
Figure 17.Server #1 energy efficiency and proportionality under different frequency scaling configurations.

Figure 18 .
Figure 18.Server #3 energy efficiency and proportionality under different frequency scaling configurations.

Figure 18 .
Figure 18.Server #3 energy efficiency and proportionality under different frequency scaling configurations.
Comparison of EP curve (or power curve) of the server in Table

Table 2 .
Peak energy efficiency occurrence by utilization spot.
Note: One server achieves peak energy efficiency at both 80% and 90% utilization.Therefore, 509 servers have 510 peak energy efficiency spots.

Table 3 .
Energy efficiency and proportionality of servers with non-100% peak energy efficiency.

Table 4 .
More energy efficient working range of the server in Table1.

Table 5 .
Count of utilization where peak energy efficiency occurs.Figure6draws the average and median values of PEEP, EP, and the SPECpower score of the 509 servers.These are grouped by their utilization spot where peak energy efficiency occurs.Servers achieving their peak energy efficiency at lower utilization have higher PEEP and EP values than those at higher utilization.

Table 4 .
More energy efficient working range of the server in Table1.

Table 5 .
Count of utilization where peak energy efficiency occurs.

Algorithm 1: Virtual machine scheduling algorithm based on energy efficiency and proportionality in EASE Input: virtual machines and available server set Output: scheduling between VMs and servers  Energy efficiency and proportionality calculation for each workload type (computing intensive ， memory intensive or hybrid) and hardware software configurations  for each server in available server set  calculate its peak energy efficiency and optimal working range  sort the server by energy efficiency, EP and PEEP  end for  for each VM  workload characterization by server tracing and monitoring  schedule it to available server with highest PEEP  end for  update VM set  update available server set Figure 10.
VM scheduling in EASE.

Algorithm 2: virtual machines migration Input: server set S with VM running set V Output: server set S' with VM running set V' 1. 𝐦𝐚𝐱 = 𝐨𝐩𝐭𝐢𝐦𝐚𝐥 𝐰𝐨𝐫𝐤𝐢𝐧𝐠 𝐫𝐚𝐧𝐠𝐞 𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝 //set the upper threshold of server load for each server 2. for server in S 3. get the load status 𝐖 𝐢
Figure 11.VM migration in EASE.

Table 7 .
Server types and experimental platform configuration.

Table 7 .
Server types and experimental platform configuration.

Table 8 .
Power consumption and performance comparison of different scheduling algorithms.

Table 8 .
Power consumption and performance comparison of different scheduling algorithms.

Table 9 .
Power consumption and performance comparison of heterogeneous cluster.

Table 10 .
Power consumption and performance comparison of heterogeneous cluster.

Table 9 .
Power consumption and performance comparison of heterogeneous cluster.

Table 10 .
Power consumption and performance comparison of heterogeneous cluster.

Table 9 .
Power consumption and performance comparison of heterogeneous cluster.

Table 10 .
Power consumption and performance comparison of heterogeneous cluster.

Table 11 .
Power consumption and performance comparison of a heterogeneous cluster before and after scheduling (mixed load).