Carbon-E ﬃ cient Virtual Machine Placement Based on Dynamic Voltage Frequency Scaling in Geo-Distributed Cloud Data Centers

: The tremendous growth of big data analysis and IoT (Internet of Things) has made cloud computing an integral part of society. The prominent problem associated with data centers is the growing energy consumption, which results in environmental pollution. Data centers can reduce their carbon emissions through e ﬃ cient management of server power consumption for a given workload. Dynamic voltage frequency scaling (DVFS) can be applied to control the operating frequencies of the servers based on the workloads assigned to them, as this approach has a cubic increment relationship with power consumption. This research work proposes two DVFS-enabled host selection algorithms for virtual machine (VM) placement with a cluster selection strategy, namely the carbon and power-e ﬃ cient optimal frequency (C-PEF) algorithm and the carbon-aware ﬁrst-ﬁt optimal frequency (C-FFF) algorithm.The main aims of the proposed algorithms are to balance the load among the servers and dynamically tune the cooling load based on the current workload. The cluster selection strategy is based on static and dynamic power usage e ﬀ ectiveness (PUE) values and the carbon footprint rate (CFR). The cluster selection is also extended to non-DVFS host selection policies, namely the carbon- and power-e ﬃ cient (C-PE) algorithm, carbon-aware ﬁrst-ﬁt (C-FF) algorithm, and carbon-aware ﬁrst-ﬁt least-empty (C-FFLE) algorithm. The results show that C-FFF achieves 2% more power reduction than C-PEF and C-PE, and demonstrates itself as a power-e ﬃ cient algorithm for CO 2 reduction, retaining the same quality of service (QoS) as its counterparts with lower computational overheads.


Introduction
Datacenters are critical infrastructures that amalgamate vast computing and storage resources, offering online computing as and when needed. Virtualization techniques embedded in grid computing platforms aid data centers in providing computing resources as a service to customers [1]. The growing energy consumption is a significant problem in data centers. The consumption of energy is increasing by about 10%-12% per year [2]. Synchronized power and resource management are essential to assist data centers in conserving energy while providing the required quality of service (QoS) for hosted applications [3]. It is very much advantageous to maximize server utilization to lower energy consumption [4]. Virtual machine (VM) consolidation is performed to accomplish auto scaling, resulting in reduced energy consumption [5,6].It is essential to maintain the maximum number of 2 of 24 servers possible in a running state to satisfy service level agreements (SLAs), which account for more than 80% of information technology (IT) budgets. The power consumption of an idle server is two-thirds of its energy consumption with 100% utilization at full load [7][8][9]. It is noteworthy that idle power and dynamic power consumption utilization levels vary based on different power models of physical servers. The energy reduction achieved by shrinking the number of existing resources through VM consolidation may result in lower resource availability, jeopardizing the credibility of the provider. Utilization of servers at high voltage results in high temperatures and shorter lifetimes. Resource utilization should be optimized based on the computing capacities of the servers in order to reduce idle and active server power consumption [10]. Considering the above, minimum power consumption is achieved through optimum central processing unit (CPU) utilization of the servers with our proposed algorithms.

Related Works
Complications in workload allotment in servers with reduced power consumption mean that optimal power management is required, which is dependent on the arrival rate of the tasks and the processor's power-to-frequency relationship. Thus, it is vital to perform a quantitative analysis of the association between dynamic voltage frequency scaling (DVFS) and power consumption to optimize the use of servers [11]. The worst-fit decreasing (WFD) strategy has been proposed as a load balancing approach for task allocation and energy consumption reduction [12], where by DVFS-based fixed discrete CPU utilization levels were considered and 34% power reduction was achieved [13]. A polynomial complexity algorithm was presented, with the assumption that the energy consumption of servers with lower workloads is comparably less than higher workloads [14]. When optimizing power consumption, most of the research work has focused on optimizing the CPU and cooling devices, as they are the components that consume the most power. The CPU consumes 46% and the cooling device consumes 15% of the total power in a data center [15]. The DVFS-based approach can be used when there is a lower workload and no need to run the servers at their maximum performance level [16][17][18]. The job scheduling approach was used for workload management to achieve maximum utilization of servers with reduced energy consumption [19][20][21]. Many heuristic methods for VM placement have been used with constrained combinatorial optimization problems for different objectives, such as to identify energy-efficient hosts for VMs [22][23][24], to reduce the number of migrations [25], and to increase the number of idle hosts [26,27]. In heterogeneous environments, heuristic techniques generally cannot guarantee optimal long-term solutions [28]. Reducing power consumption through the VM migration approach involves limiting the number of powered servers operating at highest utilization level.This approach is not energy free, rather it is dependent on VM size and bandwidth [29]. The energy requirement for the live migration of idle VMs is estimated using their proposed power model [30]. The consecutive sequential migration of several VMs also has an energy impact [31]. DVFS is mainly applied to non-critical workloads to improve energy-efficient scheduling of idle servers or light-loaded servers [32]. The genetic algorithm-based model was proposed for VM placement to minimize energy consumption [33]. A multiobjective model of the VM placement problem was proposed to maximize resource utilization and minimize energy consumption and network traffic, with VM placement formulated as a bin-packing problem and the network traffic reduction formulated as a quadratic assignment problem, with resources constraints [34]. An optimization model with multiobjective formulation was considered to maximize server utilization and minimize the number of active servers with memory and CPU resource constraints [35]. The algorithm was designed in order to form an optimal initial population and to reduce the search space, and was evaluated on a small-scale data center. The author of [36] disagreed with the work performed in [35], insisting on the need for an exhaustive approach in order to arrive at an optimal solution for difficult NPproblems. Approaches for bin-packing-based modified best fit decreasing (BFD) placement and dynamic placement of VMs were considered to reduce operation costs and environmental impacts [37]. In the modified BFD approach, the VMs with the best utilization were placed in the physical machine (PM) with the least energy

•
Selection of data centers and clusters is performed based on the PUE and CO 2 emission rate, aiming to reduce the overall carbon footprints of the data centers; • Load balancing is done by identifying a feasible server with a minimal operating frequency for the current workload with the required quality of service, aiming to reduce hot spots in CPU heat dissipation, which have a direct impact on hardware lifetime and performance; • The impacts of static and dynamic power usage effectiveness (PUE) on placement decisions are analyzed, along with cooling load power impacts.
The rest of this paper is organized as follows. In Section 1, the general facts about power consumption in data centers are outlined. Section 2 surveys several closely associated research approaches related to this work. Sections 3 and 4 detail the system models and the research problem formulation. Subsequently, Section 5 elaborates on the algorithms proposed in this research work for solving the formulated stochastic problem. Then, the experimental set-up is presented in Section 6. Section 7 presents the simulation results and discussions about the significance of load balancing using the optimal frequency and the dynamic cooling load. Finally, Section 8 presents the findings of this research work.

Power Model
The power consumption of the processor is directly proportional to the frequency. Hardware-based solutions to the problem of power consumption have reached a saturation point. The energy efficiency Appl. Sci. 2020, 10, 2701 4 of 24 of multi core systems is entirely dependent on the workload. Cores that have no activity will experience static power loss, which is directly related to the supply voltage. Running a core at maximal workload means it will use the highest frequency and voltage, resulting in high power consumption. By distributing the workload among the processors, the work is completed in the same amount of time with less power consumption. In multi core systems, the only way to address this problem is to maintain the optimal CPU frequency with the minimum energy consumption ratio by distributing the workload. Operating the processor at minimum frequency is a sensible and more reasonable model for achieving minimum power requirements. There is a collective impact on P-states and workload activity on processor temperature [47]. A linear relationship exists between power consumption and the temperature of a processor in a well-cooled environment. DVFS is used to scale the supply voltage and frequency to prevent power wastage. As DVFS has a direct influence on the power consumption and temperature, it can be used as a workable thermal and power control mechanism.
The relation R between the j th PM and i th VM indicates whether VMi is placed in PMj, as below: The service level agreement (SLA) is measured using the ratio of virtual machine acceptance (RVA), calculated as: Where N represents the total number of VM requests submitted and T(R) represents the total number of VM requests accepted and mapped to available PMs. This is derived as:

Problem Formulation
The VM request is in the form of a triplet (f, r, e), where f F represents the reserved frequency, r R represents the resource requirement, and e I represents the execution interval. Consider M heterogeneous servers, with each containing discrete frequencies (f 0 , f 1 , f 2 , f 3 , f 4 , . . . f k ) with utilization (U 0 , U 1 , U 2 , U 3 . . . , U k ),where U 0 = 0% (idle), U k = 100%, and fixed dynamic power consumption (P 0 ,P 1 , P 2 P 3 P 4 . . . , P k ). Here, U 0 is considered as the idle state, with power consumption P 0 . Let S = {S 1 , S 2 , S 3 . . . .S M } represent M servers for each S j , where jε [1,M] with utilization (U j,0 , U j,1 , U j,2 , U j,3 . . . , U j,k ), and power consumption (P j,0 , P j,1 , P j,2 , P j,3 , P j,4 . . . , P j,k ) can be characterized as a triplet (CU j , CP j , C j ); where CU j is the current utilization of server S j , CP j is the power consumption of server S j with utilization state CU j , and C j is the total processing capacity of Sj.
The relation R between the j th PM and i th VM indicates whether VM i is placed in PM j , as below: The service level agreement (SLA) is measured using the ratio of virtual machine acceptance (RVA), calculated as: where N represents the total number of VM requests submitted and T(R) represents the total number of VM requests accepted and mapped to available PMs. This is derived as:

Server Power
The VM request of the ith VM in request Queue (ReqQ), (f i , r i , e i ), remains constant throughout the execution. Here, e i is the total number of intervals reserved by the VM for the resource (f i × r i ). The power consumption of the jth physical machine with utilization U l at time t is represented as P j,l (t) and derived as [48]: where U j,l < CU j (t) < U j,l+1 , 0 ≤ l ≤ k, where l represents the operating frequency, CU j (t) is the current utilization of the j th server at time t, and k is the number of discrete frequencies. The energy consumption of the j th PM with utilization u within interval [0, I] can be calculated as: The total energy consumption of the M number of PMs within a reservation interval [0, I] can be calculated as:

Cooling Power
The cooling device power consumption contributes to the maximum electricity consumption of the data center. Dynamic tuning of the cooling load based on the current workload may help reduce power consumption in data centers, which will have a direct impact on the PUE. The cooling power cannot be ignored, as it prevents service disruption caused by the heat generated by servers [49]. To analyze the power consumption of a cooling device, standard computer room air conditioning (CRAC) units are considered in this work. The power consumption of the chiller does not change much with regard to the outside air temperature or IT load [50]. The coefficient of performance (CoP) is the measure used to compute the efficiency of the cooling unit to determine its cooling load. The CoP is the ratio (d/w) of heat removed (for server load d) to the quantity of work (w) needed to remove the heat. A larger CoP indicates better efficiency, meaning less work is required to remove a greater amount of heat. The CoP of the CRAC unit is a changeable value that increases in proportion to the increase of the supply air temperature in the CRAC unit [51].
The total carbon footprint (TCF) generated at time t, including overhead power, is formulated as: where tdc, tc, M and N represent the numbers of data centers, clusters, machines, and requests, respectively. The overall energy consumption of all servers in a cluster (PS c ) within the interval [0, T], partitioned as a sequence of reservation intervals (ri) in the form of (t ri , t ri+1 ] (ri {0,1, . . . ri-1}), is formulated as: The CoP for the CRAC unit can be modeled as in [52]: where T sup = (current_temperature − safe_temperature) The PUE of the data center can be calculated as: The total overhead power (OP) of a cluster (c) is calculated as: The objective function TCF(t) is subject several limitations. The total number of VMs allocated to a machine should not exceed the servers computing (U) and memory capacity (mem), as follows: The relation R between VMs and PMs is many-to-one, meaning R ⊆ N × M if: The total energy (eng) consumed is supposed to be within the limit of the available brown energy (B) at the data center, as follows: The total brown energy consumed is supposed to be within the limits of the cloud provider's agreed upon grid electricity consumption (G): Total available B ≤ Total assigned G

RAMAlgorithm
The high-level design of the resource allocation management (RAM) algorithm executed in the management node is presented in Algorithm 1. The functionality of Algorithm 1 can be grouped Appl. Sci. 2020, 10, 2701 8 of 24 into two sections. In Section 1, lines 2-4 perform VM-to-PM mapping using the placement algorithm. In Section 2, lines 5 and 6 perform resource deallocation for every interval.

Carbon-and Power-Efficient Optimal Frequency VM Placement (C-PEF)
The C-PEF algorithmic approach is detailed in Algorithm 2. The proposed strategy allocates the new VMs to feasible servers, ensuring:(i)carbon-efficient clusters based on the PUE and carbon footprint rate (CFR);(ii) the power-efficient optimal operating frequency of servers; and(iii) a minimum increase in overall power after allocation.

Carbon-and Power-Efficient Optimal Frequency VM Placement (C-PEF)
The C-PEF algorithmic approach is detailed in algorithm 2. The proposed strategy allocates the new VMs to feasible servers, ensuring:(i)carbon-efficient clusters based on the PUE and carbon footprint rate (CFR);(ii) the power-efficient optimal operating frequency of servers; and(iii) a minimum increase in overall power after allocation.

Algorithm 2: CPEF Carbon and Power-Efficient Optimal Frequency VM Placement
Input: Clusterlist, Hostlist, ReqQ Output: TargetVMQ 1 while VM in ReqQ do 2 Totclusterlist ← Get the clusters from the Clusterlist of all datacenters; 3 sort the clusters in Totclusterlist in ascending order of (PUE*CFR) using Equation (7); 4 For cluster in Totclusterlist do 5 Mhostlist ← Get the Hostlist from cluster; 6 For freq in freqstep do 7 For host in Mhostlist do 8 maxu ← Get utilization equivalent to using Equation (4); 9 cur uti ← Get current utilization using Equation (4); 10 rem uti ← maxu-cur-uti; 11 if feasible-host for VM then 12 P1 ← Get dynamic power of the host using Equation (4); 13 P2 ← Get dynamic power of the host with VM placement using Equation (4); 14 ∆P ← P2-P1; 15 R1 ← Get minimum remaining task execution time; 16 ∆R ← Execution time of VM -R1; 17 Selected-H ost · add(host); 18 if Selected H ost≠ NULL then 19 Sort the Selected H ost in non decreasing order of ∆P ; 20 choosy host ← Get the host from Selected Host with positive ∆R; 21 Selected H ost ← Get the difference between Selected-H ost and choosy-host; 22 S exp pow ← Selected H ost[first] · P2 × Execution time of VM ; 23 For choosy host do 24 Pow1 ←Get the total dynamic power without the task corresponding to R1 and with placement of VM ; C-exp-Pow ← P2 × R1 + Pow1 × ∆R; 25 Sort choosy host in non decreasing order on C-exp-Pow; 26 Desthost ← Get host corresponding to minimum of S exp pow and C-exp-P ow; 27 T argetVMQ · add(V M, Desthost); 28 Skip freq, cluster, and go to VM loop 29 Return TargetVMQ; Appl. Sci. 2020, 10, 2701 9 of 24 The aim of the C-PEF algorithm is to distribute the load within the cluster. Each server is set to its minimum utilization level. The utilization level is increased gradually when the VM allocation is not feasible at the current utilization level. The greedy selection of the destination hosts for the VMs among the feasible hosts is based on a minimum increase in overall power consumption at the current utilization level. The utilization of each node is reduced to an extent by distributing the load without performance compromise to avoid hotspots due to CPU turbulence. Each node is utilized at the required minimum utilization level as much as possible. Algorithm 2 receives the Clusterlist of all data centers, the Hostlist of each cluster, and the VM resource request through ReqQ. Lines 2 and 3 consolidate the entire cluster list into the Totclusterlist. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE × CFR. The greedy search, considering power limited to the current utilization level, is performed in line 6 of Algorithm 2. The feasible host system with nominal operating frequency for VM placement is identified as the SelectedHost. The difference in dynamic power before and after VM placement, ∆P, is calculated in line 14 of Algorithm 2. The power consumption P2 is not constant throughout the execution of the VM, as it depends on the next incoming and outgoing tasks of the machine to which it is allocated. As the incoming task is not known in advance, the known details of outgoing tasks based on the remaining execution time and utilization level are used effectively to predict the dynamic power. This approach has an impact if there is a time gap between the first request submission and the next.
The destination host (Desthost) is identified based on the new VM(NVM) execution time and the next outgoing task's remaining execution time in lines 21-33 in algorithm 2.

Carbon-Aware First-Fit Optimal Frequency VM Placement (C-FFF)
The C-FFF algorithmic approach is detailed in algorithm 3. The aim of the C-FFF algorithm is to distribute the load within the cluster. Each server is set to its minimum utilization level. The utilization level is increased gradually when the VM allocation is not feasible at the current utilization level. This approach differs from C-PEF in terms of host selection. There is no greedy selection performed for the minimum increase in overall power consumption, and the VM is placed in the firstfit host when feasibility is confirmed at the current utilization level. Algorithm 3 receives the Clusterlist of all data centers, the Hostlist of each cluster, and the VM resource request through ReqQ. Lines 2 and 3 consolidate the entire cluster list into the Totclusterlist. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The C-FFF algorithm differs from C-PEF in terms of host selection. C-FFF does not use the greedy approach on the feasible host with minimum power, as with C-PEF; instead, it places the VM in the first feasible host that is limited to the current p-state. For n VM requests, m number of hosts, f frequency levels, and c number of clusters, the complexity of the algorithm is derived as O(nfmclog(c)).

Carbon-and Power-Efficient VM Placement (C-PE)
The cluster selection is the same as with C-FFF, meaning it is based on PUE and CFR. The standard power-efficient algorithm does consider the DVFS and remaining execution time for outgoing tasks for VM allocation [25]. In this work, the C-PE LGORITHM performs cluster selection similarly to C-PEF and C-FFF, but differs in its host selection policy. The aim of the C-PE algorithm is to find a feasible host for a VM, considering the maximum utilization level. The host selection is based on a minimum increase in overall power consumption (i.e., minimum ΔP). The Selected Hosts are sorted based on estimated ΔP (line 14of C-PEF). The destination host is selected as in algorithm 2, with maximum utilization. The algorithm complexity with n VM requests, c clusters, and m nodes is derived as O(n(clog(c)+c(m+mlog(m)+m))). The final complexity is expressed as O(ncmlog(m)).

Carbon-Aware First-Fit Optimal Frequency VM Placement (C-FFF)
The C-FFF algorithmic approach is detailed in Algorithm 3. The aim of the C-FFF algorithm is to distribute the load within the cluster. Each server is set to its minimum utilization level. The utilization level is increased gradually when the VM allocation is not feasible at the current utilization level. This approach differs from C-PEF in terms of host selection. There is no greedy selection performed for the minimum increase in overall power consumption, and the VM is placed in the first-fit host when feasibility is confirmed at the current utilization level. Algorithm 3 receives the Clusterlist of all data centers, the Hostlist of each cluster, and the VM resource request through ReqQ. Lines 2 and 3 consolidate the entire cluster list into the Totclusterlist. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The C-FFF algorithm differs from C-PEF in terms of host selection. C-FFF does not use the greedy approach on the feasible host with minimum power, as with C-PEF; instead, it places the VM in the first feasible host that is limited to the current p-state. For n VM requests, m number of hosts, f frequency levels, and c number of clusters, the complexity of the algorithm is derived as O(nf mclog(c)).

Carbon-and Power-Efficient VM Placement (C-PE)
The cluster selection is the same as with C-FFF, meaning it is based on PUE and CFR. The standard power-efficient algorithm does consider the DVFS and remaining execution time for outgoing tasks for VM allocation [25]. In this work, the C-PE algorithm performs cluster selection similarly to C-PEF and C-FFF, but differs in its host selection policy. The aim of the C-PE algorithm is to find a feasible host for a VM, considering the maximum utilization level. The host selection is based on a minimum increase in overall power consumption (i.e., minimum ∆P). The Selected Hosts are sorted based on estimated ∆P (line 14of C-PEF). The destination host is selected as in Algorithm 2, with maximum utilization. The algorithm complexity with n VM requests, c clusters, and m nodes is derived as O(n(clog(c) + c(m + mlog(m) + m))). The final complexity is expressed as O(ncmlog(m)).

Carbon-Aware First-Fit Least-Empty VM Placement (C-FFLE)
This approach performs data center and cluster selections similarly to C-PEF, C-FFF, and C-PE, but differs in terms of its host selection policy. The C-FFLE algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The host selection is based on the first-fit strategy, whereby the hosts are ordered based on the least available resources. This approach does not perform any greedy searching for minimum power heuristic methods; instead, it uses VM best-fit heuristic methods based on resource requirements for node selection.

Carbon-Aware First-Fit Least-Empty VM Placement (C-FFLE)
This approach performs data center and cluster selections similarly to C-PEF, C-FFF, and C-PE, but differs in terms of its host selection policy. The C-FFLE algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The host selection is based on the first-fit strategy, whereby the hosts are ordered based on the least available resources. This approach does not perform any greedy searching for minimum power heuristic methods; instead, it uses VM best-fit heuristic methods based on resource requirements for node selection.

Algorithm 3: C-FFF Carbon-Aware First-Fit Least-Empty VM Placement
Input: Clusterlist, Hostlist, ReqQ Output: TargetVMQ 1 while VM in ReqQ do 2 Totclusterlist ←Get the clusters from the Clusterlist of all datacenters; 3 sort the clusters in Totclusterlist in ascending order of (PUE*CFR) using Equation (7); 4 For cluster in Totclusterlist do 5 Mhostlist ←Get the Hostlist from cluster; 6 For freq in freqstep do 7 For host in Mhostlist do 8 maxu ← Get utilization of freq using Equation 4 with freq as l; 9 cur-uti ←Get current utilization of the host using Equation (4); 10 rem-uti ←maxu -cur-uti; 11 if feasible host for VM then 12 TargetVMQ •add(VM, host); 14 Break host, freq and cluster loop and go to 1 ; 16Return TargetVMQ.

5.6.Carbon-Aware First-Fit VM Placement (C-FF)
The cluster selection by the C-FF algorithm is similar to C-PEF,C-FFF,C-FFLE, and C-PE algorithms, but differs in terms of the host selection policy. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for data center selection and sorts the Totclusterlist in ascending order based on PUE×CFR. It uses first-fit heuristic methods for host selection.

Experimental Environment and Assumptions
Considering the expense and time incurred in the evaluation of large-scale experiments in real time, Matlab software is used to simulate the environment. Each reservation interval is assumed to have duration of 300 seconds. The input request is accepted at the beginning of each reservation cycle. A data center with heterogeneous systems with different power models capable of provisioning multiple VMs is considered. The virtual resource size is not known and the VM request has no limitations. The VM is assumed to be active throughout the execution time. All the tasks are considered to be CPU-intensive. The power consumption of a task is measured by its CPU utilization, as this is considered to consume a significant fraction of energy. All the machines are assumed to be in an off state when not in use. The VM's resource requirements are assumed to be constant throughout the reservation interval. The data center's safe operating temperature is considered to be 23 °C. The peak IT load (server only) power estimation for the data center is 52 kW for the physical machine specifications given in Table 1 [53].The data center is assumed to have a floor space of approximately 500 square feet. The total electricity power requirement is calculated as 124 kW (including cooling and lighting load). The CPU power consumption for all servers should not exceed

Carbon-Aware First-Fit VM Placement (C-FF)
The cluster selection by the C-FF algorithm is similar to C-PEF,C-FFF,C-FFLE, and C-PE algorithms, but differs in terms of the host selection policy. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for data center selection and sorts the Totclusterlist in ascending order based on PUE×CFR. It uses first-fit heuristic methods for host selection.

Experimental Environment and Assumptions
Considering the expense and time incurred in the evaluation of large-scale experiments in real time, Matlab software is used to simulate the environment. Each reservation interval is assumed to have duration of 300 s. The input request is accepted at the beginning of each reservation cycle. A data center with heterogeneous systems with different power models capable of provisioning multiple VMs is considered. The virtual resource size is not known and the VM request has no limitations. The VM is assumed to be active throughout the execution time. All the tasks are considered to be CPU-intensive. The power consumption of a task is measured by its CPU utilization, as this is considered to consume a significant fraction of energy. All the machines are assumed to be in an off state when not in use. The VM's resource requirements are assumed to be constant throughout the reservation interval. The data center's safe operating temperature is considered to be 23 • C. The peak IT load (server only) power estimation for the data center is 52 kW for the physical machine specifications given in Table 1 [53]. The data center is assumed to have a floor space of approximately 500 square feet. The total electricity power requirement is calculated as 124 kW (including cooling and lighting load).
The CPU power consumption for all servers should not exceed 17.3kW. The cooling load concerning CPU utilization is limited to 12.11 kW [54]. The data centers are assumed to be powered only by grid energy sources. Physical Machine and VM Reservation Modeling Table 1 shows model of physical machines with varying power models to simulate heterogeneity and configurations of heterogeneous systems taken from the SPEC power benchmark [53] used in the simulation. Table 2 presents the power consumption, with equal CPU utilization distribution ranging from 0% to 100%. The power calculation for the periods in between intervals is estimated based on Equation (4). For example, the power consumption with 13% CPU utilization for power model 1 is between 10% and 20%,while the resulting power is 64.14 W based on ((13%−10%)/(20%−10%) ×(66−63)) + 63 with reference to Table 2. To evaluate the proposed algorithms, 4 small-scale data centers with 100 heterogeneous systems are used to model infrastructure-as-a-service(IaaS).The VM characteristics for elastic compute units (ECU) shown in Table 3 are used to model the virtual machine reservations. Each data center is assumed to have 2 clusters with varying values for carbon footprint rates. The carbon footprint rates of clusters and PUE values of data centers are considered based on [55,56], as presented in Table 4. The workload is generated based on the Lublin-Feitelson model [57]. By taking advantage of the arrival rate, gamma, and hyper-gamma Lublin parameters, the bag-of-tasks and web requests are generated, which have long and short holding times, respectively, as compared to the VM types given in Table 3 (shown in Figures 3 and 4). Figure 3 depicts the variation in the numbers of requests in each reservation cycle. Figure 4 presents the total number of CPU utilization requests received from the VMs concerning different reservation cycles.

Results and Discussions
The workload data described above are used to evaluate the proposed VM placement algorithms C-PEF and C-FFF against C-PE, C-FF, and C-FFLE approaches. The C-FFLE algorithm is used to show the impact on power consumption when only resource usage is considered as a parameter in the heuristic approach. The C-FF is the first-fit placement algorithm, which is used for initial placement for all algorithms in this work. The other algorithms improve the placement strategy for power reduction as an extension of C-FF. In this work, along with initial placement, C-FF is used separately to model the worst possible power consumption. Naturally, the C-FFLE and C-FF algorithms will have worse performance than the power management algorithms. The C-PE algorithm is considered as a fair measure to evaluate power management approaches.

Results and Discussions
The workload data described above are used to evaluate the proposed VM placement algorithms C-PEF and C-FFF against C-PE, C-FF, and C-FFLE approaches. The C-FFLE algorithm is used to show the impact on power consumption when only resource usage is considered as a parameter in the heuristic approach. The C-FF is the first-fit placement algorithm, which is used for initial placement for all algorithms in this work. The other algorithms improve the placement strategy for power reduction as an extension of C-FF. In this work, along with initial placement, C-FF is used separately to model the worst possible power consumption. Naturally, the C-FFLE and C-FF algorithms will have worse performance than the power management algorithms. The C-PE algorithm is considered as a fair measure to evaluate power management approaches.

Results and Discussions
The workload data described above are used to evaluate the proposed VM placement algorithms C-PEF and C-FFF against C-PE, C-FF, and C-FFLE approaches. The C-FFLE algorithm is used to show the impact on power consumption when only resource usage is considered as a parameter in the heuristic approach. The C-FF is the first-fit placement algorithm, which is used for initial placement for all algorithms in this work. The other algorithms improve the placement strategy for power reduction as an extension of C-FF. In this work, along with initial placement, C-FF is used separately to model the worst possible power consumption. Naturally, the C-FFLE and C-FF algorithms will have worse performance than the power management algorithms. The C-PE algorithm is considered as a fair measure to evaluate power management approaches.

Reduction in Overall Carbon Footprint
The reduction of grid energy consumption in datacenters is considered as a crucial metric for carbon footprint reduction. Equation (7) formulates the total carbon footprint emission of data centers.

•
The ratio of VM acceptance (RVA) The RVA is considered as a measure of service level agreement (SLA). The RVA is the ratio between the number of VM requests placed and the number of requests submitted.

Scenario-I: Energy-Efficient Mapping of VMs to PMs with StaticPUE
The VM placement algorithms are evaluated based on the reduction of the carbon footprint with static PUE, as shown in Table 4. Figure 5 and Table 5 present the number of active PMs for the same utilization level, with 100% RVA for all algorithms. In order to interpret the number of active PMs shown in Table 5, this has to be compared with the minimum and maximum utilization levels given in Table 6 for each interval. In C-PE, for 10% utilization, the number of active PMs is 36 with a minimum utilization 50% and the number of PMs with 100% utilization is 23. These numbers are far greater than C-FFF and C-PEF, for which the number of active PMs is 17, with minimum utilization ranging between 25% and 40.6%. The C-PE placement strategy utilizes a lower number of PMs with the maximum utilization possible for the current workload, but the C-FFF and C-PEF algorithms utilize the maximum number of PMs with the minimum possible utilization level and a lower number of fully utilized PMs. The results show that distribution of the load among the servers using DVFS with C-FFF and C-PE algorithms limits the percentage of load received at each interval. This approach does not lead to optimal results with very low loads. The minimum load required for best result depends on the machine configuration and power model. According to our specifications, repeated execution with different VM requests shows that20% is the minimum load. For the C-PEF algorithm, the optimal load requirement is less than C-FFF, because in spite of DVFS, it uses the greedy approach, which limits load distribution. It can be noticed that the C-FF algorithm achieves a significant improvement over C-FFLE, displaying a trade-off between effective resource utilization and power consumption. The utilization results presented in Table 6 prove the above algorithm strategies. Figure 6a,b and Table 5 illustrates the power consumption for all algorithms at 100% RVA for the first 8 intervals, the power consumption of the C-PEF algorithm is 3.79% lower than for C-FFF, while the power consumption for C-FFF is 2.26% lower than C-PE, 21.75% lower than C-FFLE, and 12.08% lower than C-FF. Based on the cumulative carbon footprint depicted in Figure 6b, which is equivalent to the power given in Table 5, the C-PEF reduces the carbon footprint to 4.09%, while C-PE, C-FFLE, and C-FF reduce the carbon footprint to 3.35%, 38.8%, and 17.6%, respectively. Based on Figure 7a and Table 7, the total power consumed by the servers using the C-PEF placement algorithm is reduced by 1.61%. The C-FFF algorithm reduces power consumption by2.16%, 13.54%, 2.77% when compared to C-PE, C-FFLE, and C-FF, respectively. Based on Figure 7b and Table 7, the C-PEF placement algorithm's carbon emission is 1.64% less than C-FFF. The C-FFF placement algorithm consumes 2%, 15%, and 2.8% less power than C-PE, C-FFLE, and C-FF, respectively. Table 5. Power consumption for 100% ratio of VM acceptance (RVA).       In Table 5, the carbon footprint values for C-FFF and C-PEF are about 2.30% and 3.18% lower than for C-PE, respectively, with an increased RVA of 1.2%. C-FFLE and C-FF algorithms have 19.89% In Table 5, the carbon footprint values for C-FFF and C-PEF are about 2.30% and 3.18% lower than for C-PE, respectively, with an increased RVA of 1.2%. C-FFLE and C-FF algorithms have 19.89%  In Table 5, the carbon footprint values for C-FFF and C-PEF are about 2.30% and 3.18% lower than for C-PE, respectively, with an increased RVA of 1.2%. C-FFLE and C-FF algorithms have 19.89% and 1.23% greater carbon footprints than C-PE, respectively. Table 8 depicts the substantial improvement in VM request acceptance for C-PEF and C-FFF algorithms compared with other heuristic approaches with different numbers of VM requests. The C-FFF and C-PEF algorithms have approximately 1% higher RVA percentage than other counterparts. The statistical analysis presented in Table 9 supports the fact that the C-FFLE algorithm, which is based on resource utilization, operates with maximum utilization compared to other approaches. The C-FFF, C-PE, and C-PEF algorithms have approximately 1% variation in maximum CPU utilization rates, with similar average utilization rates. With regard to power, the C-PEF placement algorithm's power consumption is 1.64% less than C-FFF. C-FFF consumes 2%, 15%, and 2.8% less power than C-PE, C-FFLE, and C-FF, respectively. The results in Figure 8a,b were obtained by varying the system load with respect to the number of requests in order to measure the power consumption of different algorithms for a single interval with a common initial state. This was done so as to rank the performance of the algorithms from lowest to highest in terms of CPU utilization. Figure 8a displays the power consumption values for all the algorithms, with CPU utilization rates ranging between 40% and 90%. It can be noticed that the C-PEF algorithm shows significant performance improvement between 40%and 85% utilization. Figure 8b presents the carbon footprint values for all of the algorithms for utilization rates above 85%. Above 90% utilization, C-PEF and C-FFF are in close proximity to each other. of requests in order to measure the power consumption of different algorithms for a single interval with a common initial state. This was done so as to rank the performance of the algorithms from lowest to highest in terms of CPU utilization. Figure 8a displays the power consumption values for all the algorithms, with CPU utilization rates ranging between 40% and 90%. It can be noticed that the C-PEF algorithm shows significant performance improvement between 40%and 85% utilization. Figure 8b presents the carbon footprint values for all of the algorithms for utilization rates above 85%. Above 90% utilization, C-PEF and C-FFF are in close proximity to each other. Figures 9 and 10 present the amounts of carbon and power consumed by different powerefficient VM placement algorithms. C-PEF and C-FFF algorithms consume more power initially at lower loads than C-PE, which distributes the loads among all the servers. C-FFLE and C-FF algorithms consume more power as they do not utilize power-efficient allocation. The nonparametric Mann-Whitney U test and Wilcoxon rank sum test are utilized to test whether there is a noteworthy difference in the results obtained. Based on the abovementioned non-parametric tests on two samples for C-PEF with C-PE, C-FFLE, and C-FF, the p-values obtained are less than 0.0001. Therefore, it can be concluded that DVFS-aware scheduling (C-PEF and C-FFF) makes a significant difference compared with standard power-aware scheduling(C-PE) and other heuristic approaches interms of energy consumption. The difference between the two DVFS-aware algorithms, C-PEF and C-FFF, is not substantial (p-value of0.76 > 0.05). Figures 9 and 10 present the amounts of carbon and power consumed by different power-efficient VM placement algorithms. C-PEF and C-FFF algorithms consume more power initially at lower loads than C-PE, which distributes the loads among all the servers. C-FFLE and C-FF algorithms consume more power as they do not utilize power-efficient allocation. The non-parametric Mann-Whitney U test and Wilcoxon rank sum test are utilized to test whether there is a noteworthy difference in the results obtained. Based on the abovementioned non-parametric tests on two samples for C-PEF with C-PE, C-FFLE, and C-FF, the p-values obtained are less than 0.0001. Therefore, it can be concluded that DVFS-aware scheduling (C-PEF and C-FFF) makes a significant difference compared with standard power-aware scheduling(C-PE) and other heuristic approaches interms of energy consumption. The difference between the two DVFS-aware algorithms, C-PEF and C-FFF, is not substantial (p-value of 0.76 > 0.05).

Power Consumption (kW) Total Active PMs Total CPU Utilization % C-PEF C-PE C-FFF C-FFLE C-FF C-PEF C-PE C-FFF C-FFLE C-FF
difference compared with standard power-aware scheduling(C-PE) and other heuristic approaches interms of energy consumption. The difference between the two DVFS-aware algorithms, C-PEF and C-FFF, is not substantial (p-value of0.76 > 0.05).

Scenario-II: Energy-Efficient Mapping of VMs to PMs with DynamicPUE
The power usage effectiveness is the metric used to analyze the efficiency of a datacenter. This is the ratio between the total energy requirements of a data center (total facility power) and the power consumed by IT devices. In total, 60% of the energy consumption is due to cooling device power consumption, which has a direct impact on PUE. The proposed power-aware algorithms C-FFF and C-PEF, along with the standard C-PE algorithm, are considered in scenario II to analyze the impact of dynamic PUE on carbon footprint values, based on Equation (11). Table 10 presents the power consumption and carbon footprint values observed with dynamic PUE under the same workload used for the observed values in Table 5 for fair comparison. Dynamic PUE reduces the carbon footprint by approximately 50%, as shown in Table 10. The RVA percentages for C-FFF and C-PE displayed in Table 11 show a slight dip at the beginning and then a significant increase of 1%. Table  12 shows the overall statistics for CPU utilization and power consumption related to dynamic PUE. The values in Table 10 confirm the impact of dynamic PUE. The power consumption of the C-PE algorithm is reduced by approximately 55% compared to static PUE. The power consumption for the mean CPU utilization presented in Tables 9 and 12 reveals the impact of dynamic PUE on power reduction. C-FFF, C-PE, and C-PEF algorithms achieve approximately 14%, 9%, and 15% greater reductions than static PUE. The results support the approach of energy reduction by dynamically adjusting the cooling device load based on the active power consumption of the server for the current workload.

Scenario-II: Energy-Efficient Mapping of VMs to PMs with DynamicPUE
The power usage effectiveness is the metric used to analyze the efficiency of a datacenter. This is the ratio between the total energy requirements of a data center (total facility power) and the power consumed by IT devices. In total, 60% of the energy consumption is due to cooling device power consumption, which has a direct impact on PUE. The proposed power-aware algorithms C-FFF and C-PEF, along with the standard C-PE algorithm, are considered in scenario II to analyze the impact of dynamic PUE on carbon footprint values, based on Equation (11). Table 10 presents the power consumption and carbon footprint values observed with dynamic PUE under the same workload used for the observed values in Table 5 for fair comparison. Dynamic PUE reduces the carbon footprint by approximately 50%, as shown in Table 10. The RVA percentages for C-FFF and C-PE displayed in Table 11 show a slight dip at the beginning and then a significant increase of 1%. Table 12 shows the overall statistics for CPU utilization and power consumption related to dynamic PUE. The values in Table 10 confirm the impact of dynamic PUE. The power consumption of the C-PE algorithm is reduced by approximately 55% compared to static PUE. The power consumption for the mean CPU utilization presented in Tables 9 and 12 reveals the impact of dynamic PUE on power reduction. C-FFF, C-PE, and C-PEF algorithms achieve approximately 14%, 9%, and 15% greater reductions than static PUE. The results support the approach of energy reduction by dynamically adjusting the cooling device load based on the active power consumption of the server for the current workload. Let n, f, m, and c represent the number of VM requests, the fixed DVFS levels, the number of nodes, and number of clusters, respectively. The complexity of the C-PE algorithm is expressed by O(ncmlog(m)). Its complexity is dominated by mlog(m). The complexity of the proposed C-PEF algorithm is expressed by O(nfcmlog(m)). The C-PEF complexity is f times that of C-PE. The complexity of the C-FFF algorithm O(nf mclog(c)) is dominated by f (fixed frequency level) and clogc. As the number of nodes (m) in the data center increases, the complexity of the C-PE dominates the overhead caused by the constant f in C-FFF. The proposed C-FFF algorithm with complexity O(nf mclog(c)) performs load balancing, while maintaining a better tradeoff between utilization and power consumption than the standard C-PE algorithm with complexity O(ncmlog(m)).

Conclusions
Energy consumption and carbon footprint problems in data centers are handled using different VM placement algorithms with static and dynamic PUE. The data center energy efficiency metric PUE and carbon usage effectiveness are used as important measures for data center selection. The proposed C-FFF and C-PEF placement algorithms perform placement decisions by maintaining the optimal p-state of the servers. In C-PEF, host selection is based on the power-efficient optimal p-state of the servers. In C-FFF, the host selection is based on the optimal p-state of the servers. Both C-FFF and C-PE are compared with a standard power-efficient algorithm (C-PE), where the host selection is based on the highest power-efficient p-state of the servers. Different VM types with varying execution times and arrival rates are used to simulate the system load. The resulting outcomes for scenario I reveal that C-FFF can reduce the carbon footprint by a minimum of 2% more than C-PE, C-FFLE, and C-FF. The experimental results illustrate the importance of considering the DVFS of the servers, along with PUE and carbon release of clusters in data centers. The results for the algorithms in scenario II emphasize the impact of dynamic PUE on the carbon footprints. The C-FF algorithm shows significant improvement over C-FFLE in power reduction, displaying a trade-off between effective resource utilization and power consumption. Among the three power-aware algorithms, C-PEF and C-PE have additional computational overhead due to greedy search function. The results support the fact that C-FFF balances computational overhead and utilization, and stands in between C-PEF and C-PE with some degree of minimum resource request constraint. In conclusion, C-FFF is a power-efficient algorithm for VM placement with reduced computational overhead. The formulations presented in this work open new and challenging areas of further research relating to renewable energy sources.
Author Contributions: Conceptualization, T.R. and K.G.; methodology, T.R. and K.G.; writing-original draft preparation, T.R. and K.G.; supervision, K.G. and P.S.; writing-review and editing, N.P. and P.S. All authors have read and agreed to the published version of the manuscript.