Impact of Fan Airflow of IT Equipment on Thermal Environment and Energy Consumption of a Data Center

Abstract

Energy-saving in regard to heating, ventilation, and air-conditioning (HVAC) in data centers is strongly required. Therefore, to improve the operating efficiency of the cooling equipment and extend the usage time of the economizer used for cooling information-technology equipment (ITE) in a data center, it is often the case that a high air-supply temperature within the range in which the ITE can be sufficiently cooled is selected. In the meantime, it is known that when the ambient temperature of the ITE rises, the speed of the built-in cooling fan increases. Acceleration of the built-in fan is thought to affect the cooling performance and energy consumption of the data center. Therefore, a method for predicting the temperature of a data center—which simply correlates supply-air temperature with ITE inlet temperature by utilizing existing indicators, such as air-segregation efficiency (ASE)—is proposed in this study. Moreover, a method for optimizing the total energy consumption of a data center is proposed. According to the prediction results obtained under the assumption of certain computer-room air-conditioning (CRAC) conditions, by lowering the ITE inlet temperature from 27 °C to 18 °C, the total energy consumption of the machine room is reduced by about 10%.

1. Introduction

As for information-technology equipment (ITE), the higher the inlet temperature of the ITE, the higher the failure rate [1]. Therefore, from the viewpoint of preventing the stoppage of information communication due to the failure of ITE, the ITE is housed in a temperature- and humidity-controlled communication machine room or server room (hereinafter, “machine room”). The facility that operates and manages the machine room—called a “data center”—is positioned as an important social infrastructure. The air-conditioning system in a data center operates 24/7 and must handle the huge sensible heat load of the ITE. As the air-conditioning system consumes more than 30% of the energy consumed by the entire data center [2], it is necessary to implement energy-saving in regard to the air-conditioning system. Accordingly, to date, energy-saving in data centers has been achieved by improving the efficiency of air-conditioning systems, namely, improving the performance of air conditioners [3,4] and comparing and evaluating various air-conditioning air-flow methods [5,6]. In recent years, energy-saving methods that use natural energy, so-called “economizers,” have been researched [7,8,9]. An economizer is defined as “a control system that reduces mechanical heating and cooling requirements” [10]. For example, for a “direct airside economizer,” which directly introduces outside air into the machine room, problems such as particulate and gaseous contamination need to be solved. The American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) TC9.9 considers the surrounding environment, climate, and natural disasters at a site, which are sources of particulates and gases, as considerations when introducing an airside economizer, and it has reported the selection of appropriate filters and provision of real-time monitoring and control [11]. However, a significant effect of implementing a direct airside economizer, namely, a reduction in power consumption of heating, ventilation, and air conditioning (HVAC) by about 50%, has been reported [12].

In the meantime, energy-saving concerning the ITE itself is progressing. The ENERGY STAR^® (a program run by the U.S. Environmental Protection Agency and U.S. Department of Energy that promotes energy efficiency) sets standards for server efficiency during idle time, as well as operating conditions [13]. To save energy during partial loading, such as during the idle state, the rotation speed of the built-in cooling fan of the ITE (hereinafter simply referred to as “built-in fan”) is controlled (i.e., increased or decreased) according to the inlet temperature of the ITE and the temperature of its components. As an example, a system has been introduced that determines the lower limit of the speed of the built-in fan from the inlet temperature of the ITE and controls the built-in fan speed by a PID (proportional integral derivative) controller according to the temperature of the components [14]. In addition, as a more advanced way to control the built-in fan, a “multi-input multi-output (MIMO) fan controller” has been proposed [15]. In both above-mentioned cases, when the ITE inlet temperature rises, the built-in fan speed (i.e., ITE airflow rate) increases by these control schemes. As a result, power consumption of the ITE increases. To improve the operating efficiency of the cooling system and extend the usage time of the economizer, ASHRAE TC 9.9 recommends selecting a high air-supply temperature within the range in which the ITE can be sufficiently cooled [16]. However, when the inlet temperature rises, the rotation speed of the built-in fan increases, and that increase is considered to affect the thermal environment and power consumption of the machine room.

This effect has already been studied. The relationship between the ITE inlet temperature and power consumption of several types of servers was clarified by Moss et al., and these factors were combined in order to suppose a typical complex IT load. The combined IT load was then reproduced by an ITE simulator, and the relationship between the total power consumption of the ITE and HVAC with respect to the supply-air-temperature setting was obtained experimentally. As a result, it was shown that a certain supply-air-temperature setting minimizes the total power consumption. However, in this experiment, an in-row air conditioner was used, and in that case, the supply-air temperature and the ITE inlet temperature are close. In the case of a general machine room fitted with an ambient air-conditioning system, it has been reported that the air-supply temperature and ITE inlet temperature deviate owing to air mixing in the room, which has a negative effect on the efficiency and cooling capacity of the air-conditioner [17]. That is, to correlate and regulate the power consumptions of the ITE and HVAC, it is necessary to clarify the relationship between air supply and ITE air intake. However, clarifying that relationship is technically very difficult. If ITE inlet temperature is changed in an effort to find the optimum point of total power consumption, the ITE airflow rate changes, and the balance between ITE airflow rate and air-supply amount changes. This balance—called “availability of flow” [18]—is defined as the ratio of air supply to ITE airflow rate. In particular, it is known that when the ITE airflow rate relative to the supply-air volume increases and the availability of flow decreases (to less than one in the case of an ideal machine room without air-leakage paths), the recirculation airflow increases, and the indoor thermal environment suddenly changes; namely, it worsens [19]. In other words, as the recirculation-air volume is large in the region where the availability of flow is low, the temperature rise from the air outlet of the air conditioner to the inlet port of the ITE will be underestimated unless the increase in recirculation airflow is considered. This underestimation leads to misunderstandings in regard to power consumption of the air conditioner and power consumption and airflow of the ITE.

Given the fact that the airflow environment in a machine room is complicated, to grasp the rise in temperature from the air-conditioner outlet to the ITE inlet, in addition to actual measurements [20], predictions by ventilation network analysis [19], computational fluid dynamics (CFD) [6], and machine learning [21] are necessary. However, with these solutions, it takes a huge effort to consider changes in recirculation flow due to differences in the availability of flow. Therefore, in the present study, we propose a method for predicting the recirculation flow in an arbitrary machine room by simple numerical calculation by utilizing existing indicators, such as air-segregation efficiency (ASE) proposed by Tozer et al. [18]. As the proposed prediction method can simply correlate supply-air temperature with ITE inlet temperature, by incorporating the characteristics of the ITE and HVAC, the optimum points of the total power consumptions of the ITE and HVAC with respect to supply-air temperature or ITE inlet temperature can be estimated. First, the characteristics of the ITE with variable built-in-fan speed are clarified. Next, the characteristics of the ITE (determined by using a heat generator that can set airflow rate) are reproduced, and whether the thermal environment of the machine room can be predicted by the proposed prediction method is confirmed experimentally. The characteristics of the ITE (which have been determined) are reproduced by using a heat generator that can set the airflow rate, and an experiment is conducted to confirm whether the thermal environment of the machine room can be predicted by the prediction method. Finally, a method for finding the optimum point of total energy consumption of the machine room—by using the clarified ITE characteristics and the power-consumption characteristics of the HVAC (assumed to be computer-room air-conditioning, CRAC)—is proposed.

2. Proposed Prediction Method

2.1. Overview of Airflow Performance

To correlate supply-air temperature with ITE inlet temperature, the metrics for the air performance of data centers and air-segregation efficiency (ASE) proposed by Tozer et al. [18] is further developed in this study. As for these metrics, by using mass-flow balance equations and weighted average temperatures of the air supply and return air of the air conditioner in the machine room, as well as intake and exhaust temperatures of the ITE, it is possible to obtain the average bypass airflow and recirculated airflow, and the efficiencies of air supply and demand can be calculated.

The air-flow model assumed for these metrics are explained as follows. The supply airflow of the CRAC (m_c) merges with the negative-pressure flow (NP) (airflow attracted from the hot aisle into the raised floor of the machine room by the Venturi effect) in the raised floor. The airflow after this merging is divided into the airflow (m_f) that contributes to cooling the ITE and the bypass airflow (BP) that does not. The amount of cooling air (m_f) that reaches the ITE is sucked into the ITE as the ITE airflow rate (mi) together with the circulation flow (R) of the ITE exhaust gas. Tozer et al. excluded NP flow from the evaluation because it does not exist in all data centers and is difficult to measure; however, according to the air-performance metrics, NP is included in R [18].

Bypass flow (BP) is the air supply that returns to the air conditioner without contributing to the cooling of the ITE. It commonly occurs in the gaps between hot-aisle raised-floor opening panels and double-floor fixed panels. It is also caused by the oversupply of cold air to the cold aisle. Recirculation flow (R) is air that returns the exhaust air of the ITE to the air-supply side of the ITE before it is cooled by the air conditioner; as a result, the inlet temperature of the ITE increases. Recirculation flow occurs in the machine room (e.g., in the racks) owing to the local lack of air supply. BP and R are defined as follows:

$$BP=\frac{T_{io}-T_{ci}}{T_{io}-T_{co}} ,$$

(1)

$$R=\frac{T_{ii}-T_{co}}{T_{io}-T_{co}} ,$$

(2)

where T_ci is the CRAC inlet (return) air temperature, T_co is the CRAC outlet (supply) air temperature, T_ii is the ITE inlet-air temperature, and T_io is the ITE outlet-air temperature.

Bypass flow (BP) and recirculation flow (R) affect the amount of cold air reaching the ITE. “Air performance” can therefore be defined by the performance of the supply and demand air streams. “Supply performance” (η_supply) is the ratio of the amount of cooling air (m_f) that reaches the ITE to the amount of air-conditioned air supply (m_c). If η_supply equals 1, it is taken that no bypass flow exists in the room. “Demand performance” (η_demand) is the ratio of the amount of cooling air (m_f) that reaches the ITE to the ITE airflow (m_i). If η_demand equals 1, it is taken that recirculation does not occur in the machine room. η_supply and η_demand are defined as follows:

$${\eta }_{supply}=1-BP=\frac{m_f}{m_c}=\frac{T_{ci}-T_{co}}{T_{io}-T_{co}} ,$$

(3)

$${\eta }_{demand}=1-R=\frac{m_f}{m_i}=\frac{T_{io}-T_{ii}}{T_{io}-T_{co}} ,$$

(4)

“Availability of flow” (A_f) is the ratio of the CRAC air supply (m_c) to airflow required by the ITE (m_i). A_f greater than 1 indicates an oversupply of cold air, and A_f less than 1 indicates an undersupply of cold air. To prevent hot air from flowing back into the cold aisle, A_f should be set slightly higher than 1, and it is calculated as follows:

$$A_f=\frac{m_c}{m_i}=\frac{T_{io}-T_{ii}}{T_{ci}-T_{co}}=\frac{{\eta }_{demand}}{{\eta }_{supply}} ,$$

(5)

Air-segregation efficiency (ASE)—an index related to the efficiency of separating cold air and hot air—is introduced as an index of “air management” that is independent of IT load and fan speed of the air conditioner. ASE is given as follows:

$$ASE=\sqrt{\frac{{\left(1-BP\right)}^2+{\left(1-R\right)}^2}{2}} ,$$

(6)

2.2. Method of Predicting R by Using ASE

Substituting Equations (3) and (4) into Equation (5) and rearranging them gives the following equation:

$$1-BP=\frac{1-R}{A_f} ,$$

(7)

Substituting Equation (7) into Equation (6) and rearranging them gives the following equation:

$$R=1-\sqrt{\frac{2}{1+\frac{1}{A_f{}^2}}}·ASE , R\ge 0 ,$$

(8)

Equation (8) indicates that recirculation flow (R) can be simply obtained from the availability of flow (A_f) and air-segregation efficiency (ASE). It should be noted here that ASE is an index independent of the power consumption of the ITE and air-conditioner fan speed. In other words, unless the air-leakage path that affects BP and R (such as the gap between the raised-floor fixed panels and the installation status of the blanking panels) is changed, ASE takes a constant value even if the power consumption of the ITE, airflow rate of the ITE, and air-supply rate of the HVAC fluctuate. Therefore, if ASE is considered a known value, R for a certain A_f can be obtained. The relationship expressed by Equation (8) is illustrated in Figure 1.

2.3. Prediction Procedure

The prediction procedure, starting from ITE inlet temperature (T_ii), is shown in

Table 1. Prediction procedure.

Step	Input	Output	Calculation Method
1	Tii (given condition)	mi	Empirical formula, etc. (e.g., Equation (9) and Figure 14a)
2	mi	Ei	Empirical formula, etc. (e.g., Equation (10) and Figure 14b)
3	Tii, mi, Ei	Tio	$T_{io}=T_{ii}+\frac{E_i}{c\rho m_i}$
4	mc (given condition), mi	Af	Equation (5)
5	ASE (given condition), Af	R	Equation (8)
6	Tii, Tio, R	Tco	Equation (2)
7	Tco, mc, Ei1	Tci	$T_{ci}=T_{co}+\frac{E_i}{c\rho m_c}$
8	Outside air conditions and Tci 2, Ei 1, etc.	Ec	Empirical formula, etc. (e.g., Section 6.1.3)

Symbol table (including reprinting), c: Specific heat capacity of air (kJ/kg/K); E_c: CRAC power consumption (W); E_i: ITE power consumption (W); T_ci: CRAC inlet-air temperature (°C); T_co: CRAC outlet-air temperature (°C); T_ii: ITE inlet-air temperature (°C); T_io: ITE outlet-air temperature (°C); m_c: CRAC air supply (m³/s); m_i: ITE airflow (m³/s); ρ: Density of air (kg/m³). ¹ In this study, cooling capacity = ITE power consumption. If the actual cooling capacity can be obtained, it can be used. ² Depending on the characteristics of heating, ventilation, and air-conditioning (HVAC), return-air wet-bulb temperature or supply air temperature (dry bulb, wet bulb) may be used.

. For air-supply amount (m_c) and air-segregation efficiency (ASE), which are additional given conditions to ITE inlet temperature, in the case of an existing machine room, actual measurement results are used, and in the case of a new machine room, design values are used. It has been reported that the ASE for a machine room with containment is about 0.65 to 0.8, the ASE for a machine room without containment is about 0.4 to 0.7, and the average ASE is 0.6. [18]. In addition to the given conditions, necessary information is the characteristics of the ITE and the characteristics of the HVAC.

As described later, as the relationship between ITE inlet temperature and ITE airflow varies according to CPU usage rate, ITE outlet temperature (T_io) may be used as a given condition. In this study, to fix CPU usage rate in the prediction, ITE inlet temperature, which is often handled, is used as a given condition.

3. Overview of Experimental Verification Room

With the verification room, which reproduces an actual machine room of a data center, the characteristics of the ITE are revealed through reproduction experiments. The verification room is shown schematically in Figure 2, and the experimental equipment is outlined in

Table 2. Experimental facility.

CRAH	Cooling capacity Return-air conditions Water conditions Airflow, ESP	48.6 kW (sensible) 24.0 °CDB, 50%RH 7–12 °C, 2.87 L/s 4.3 m3/s, 100 Pa
CRAC	Cooling capacity Return-air conditions Outdoor-air conditions Airflow, ESP	45.0 kW (sensible) 27.0 °CDB, 19.0 °CWB 35.0 °CDB 4.0 m3/s, 120 Pa
Heat load	Target rack	1U server (Company A) × 12 or 2 kW heat generator × 1
	Nontarget rack	2 kW heat generator × 1

. The target rack with 12 servers, which will be described later, is shown in the figure. The verification room forms a cold aisle and a hot aisle, and either of the two cooling methods—namely, “raised-floor supply room return” or “raised-floor supply ceiling return”—can be selected. Raised-floor opening panels (with width of 300 mm and aperture ratio of 25% or 50%) are alternately arranged in the cold-aisle section. In the case of ceiling inlets, an inlet port with an aperture ratio of 33% is fitted in the hot-aisle portion of the ceiling. Either “cold-aisle containment” (CAC) or “hot-aisle containment” (HAC), which are airflow management and design methods introduced in the EU Code of Conduct for Data Center Energy Efficiency [22], can be configured. As for the verification room, the CAC is formed so that the cold aisle is covered by horizontal members passed over the top of the racks. HAC is formed so that the hot aisle is surrounded by vertical members that extend from the top of the racks to the ceiling. The inside of the verification room at the center (hot-aisle section) (with width of 9.6 m and depth of 11.0 m), the inside of the raised floor, and the inside of the ceiling are partitioned by temporary partitions (antistatic flameproof film), and the range of the experiment covers a width of 5.0 m and depth of 11.0 m. Of the 22 19-in. (48.26 cm) racks, 16 racks are operated, the rack to be tested is fitted with 12 servers or one heat generator (with a 2 kW heater), and one heat generator is installed in 15 racks. A shielding plate (a blanking panel) for suppressing the recirculation flow generated through these spaces can be installed on the nonmounted portion of the mounting rail or at the side of the mounting rail in the rack. For the air conditioning, a computer-room air-handler (CRAH) (with a rated sensible cooling capacity of 48.6 kW) and a computer-room air-conditioner (CRAC) (with a rated sensible cooling capacity of 45.0 kW) are used. The heat-source system for producing chilled water consists of air-cooled chillers and chilled-water pumps.

4. Understanding Characteristics of ITE

The behavior of the fan of the ITE itself is determined first. As for the ITE, a general-purpose 1U server of a certain vendor is used. To understand the behavior of the built-in fan, the ambient temperature of the server is varied. Moreover, to understand the effect of different airflow environments surrounding the servers, the presence or absence of airflow-separation technology such as CAC and blank panels is taken as a parameter.

4.1. Experimental Conditions (Experiments A and B)

Twelve servers (with specifications as listed in

Table 3. Server specifications.

Server	1U server (Company A) (Model sold in 2012)
Main specifications	CPU: 10-core (3.0 GHz) × 1 Storage: HDD (300 GB) × 1 Memory: 8 GB × 1
Power-supply specifications	Single-phase 100 VAC, 1.91 A (maximum)

) are installed in the target rack. One server is installed in row “6U” (the 6th row from the bottom of the rack, which can mount 42 units) and in every third row above that. The uppermost server is installed in row 39. The characteristics of the server when the ambient temperature of the server is varied are determined by using the temperature setting of the air-conditioning supply air as a parameter.

The specifications of the measurement items and measurement equipment in the experiment are listed in

Table 4. Measured items and measurement instruments (experiments A and B).

Item	Instrument
Server inlet/outlet temperature (3 points, 5 times)	Type-T thermocouple, data logger (GRAPHTEC, GL840)
Power consumption (all servers)	Power meter (HIOKI, 3168)
Server fan speed (each server)	Management tool mounted in server

, and the installation status and measurement points of the servers in the rack to be tested are shown in Figure 3. The mean of the measured values taken over five minutes when the conditions are stable is used. The server inlet temperature and outlet temperature are average values of the five measured temperatures, and the server airflow is calculated from the average value of the difference between server outlet temperature and server inlet temperature and the total power consumption of the 12 servers. The temperature-measurement points are arranged as evenly as possible, including the top and bottom ones, so that the average values for 12 servers can be covered. Under the conditions listed in

Table 5. Experimental conditions (experiments A and B).

Experimental Conditions	CAC	Blanking Panel	CPU Utilization %	Supply-Air-Temperature Set Point °C
A-1	No	No	0	15, 19, 23, 27, 31
A-2	No	No	50	15, 19, 23, 27, 31
A-3	No	No	100	15, 19, 23, 27, 31
B-1	No	No	100	15, 19, 23, 27, 31
B-2	No	Yes	100	15, 19, 23, 27, 31
B-3	Yes	No	100	15, 19, 23, 27, 31
B-4	Yes	Yes	100	15, 19, 23, 27, 31

, an experiment using the CRAH and CAC presence or absence, BP presence or absence, CPU utilization rate, and setting of air-conditioning supply-air temperature as parameters was conducted. The CPU usage rate is defined as (real time–idle-state time/real time), and to generate a CPU load, a stress-test tool based on multiple technical calculations was used. Checking the CPU load factor (i.e., percentage of actual computing power to maximum computing power) with respect to the CPU usage rate with the management tool installed in the server revealed that when the CPU usage rate was 100%, the CPU load rate was about 100%; when the CPU usage rate was 50%, the CPU load rate was about 51%; and when the CPU usage rate was 0%, the CPU load rate was about 1%.

As mentioned above, it is known that the speed of the internal fan of the server fluctuates with varying ambient temperature. Therefore, as shown in Table 5, changing the air-conditioning supply-air temperature changes the ITE airflow; however, it is a concern that changing the balance between ITE airflow and supply-air volume will affect the measurement results. Therefore, a preliminary experiment under certain conditions, namely, with CAC, blanking panels, a CPU usage rate of 50%, and a constant CRAH air volume, was conducted, and the server airflow (i.e., total airflow of 12 servers)—corresponding to the temperature setting of the CRAH air supply (15, 19, 23, 27, and 31 °C)—was obtained. At each supply-air-temperature setting, the airflow of the heat generator was set so that the differences between the inlet and outlet temperatures of the server and the heat generator determined in a preliminary experiment matched. The CRAH supply-air volume was also set so that the ratio of the air-conditioning air-supply volume (m_c) to ITE air volume (m_i) (i.e., availability of flow (A_f)) was close to 1.4. The airflow setting for each determined supply-air temperature setting is shown in Figure 4.

4.2. Results of Experiments A and B

4.2.1. Air-Flow Characteristics of Server

The results of experiment A are plotted in Figure 5, which shows the relationship between server inlet and outlet temperatures and server fan speed (airflow) with server CPU usage as a parameter. As shown in Figure 5a, when the CPU usage rate is 0%, the fan speed (airflow) corresponding to the server inlet temperature is lower than those values for the other CPU usage rates. It can be said that the relationship between server inlet temperature and server airflow differs according to CPU usage rate. On the contrary, it can be seen from Figure 5b that server outlet temperature and server-fan speed (airflow) have the same correlation regardless of CPU usage rate. As mentioned above, ITE controls the built-in fan speed to keep the temperature of the components at the target value [14]. As the CPU usage rate increases, the CPU load increases, so the server-fan speed (airflow) is increased to maintain the target temperature of the CPU. Therefore, it is considered that there is no difference in the server-outlet temperature after heat exchange depending on the CPU usage rate.

The results of experiment B are plotted in Figure 6, which shows the correlation between server inlet and outlet temperatures and server-fan speed (airflow) obtained with the use (or no use) of airflow-separation technology (i.e., with or without CAC and with or without blanking panels) as a parameter. Although the relationship between server inlet temperature and server-fan speed, shown in Figure 6a-1, varies slightly, the server airflow in Figure 6a-2 does not vary at all. In Figure 6b, server-fan speed and sever airflow also do not vary, and it can be said that the relationship between server ambient temperature and server-fan speed (airflow) is not significantly affected by the usage (or not) of the airflow-separation technology. The maximum fan speed confirmed in the experiment was 94%, and the minimum fan speed was 29%.

Moreover, to confirm whether the measurement-target server can represent all the servers, the average value of fan rotation speeds of the measurement target server was compared with those values of which all the servers were compared. As shown in Figure 7, the fan speeds of the server to be measured and all the servers are about equal, so it is reasonable to represent the temperatures of all twelve servers by the average value of the measured temperatures of five servers.

4.2.2. Server-Power-Consumption Characteristics

Plotting the results of experiments A and B, Figure 8 shows the correlation between server airflow rate and server power consumption of all 12 units. Clearly, as server airflow rate increases, server power consumption increases. In addition, server power consumption increases with CPU usage rate. In Figure 8b, the server power consumption at the highest and lowest server airflow rates for each CPU usage rate is compared as follows: When the CPU usage rate is 100%, the server power consumption is about 18% lower at the latter airflow rate; when the CPU usage rate is 50%, it is about 22% lower; and when the CPU usage rate is 0%, it is about 48% lower. On the other hand, it can be seen from Figure 8b that the relationship between server airflow rate and server power consumption is not significantly affected by the usage (or not) of the airflow-separation technology. As a means such as lowering the supply-air temperature or suppressing recirculation by applying airflow-separation technology, lowering the ambient temperature of the server and keeping the server airflow rate low not only reduce the power consumption of the server itself but also reduce the energy consumed by the air-conditioning system that cools the server. On the contrary, it should be noted that even a server with a CPU usage of 0% (that is, an idle server) will still consume about half the power consumption of that with a CPU usage of 100%, so it cannot be ignored as a cooling target.

5. Validity of Prediction Method

A heat generator was installed in each rack, and the characteristics of a general-purpose 1U server in the presence of the heat generator were reproduced. It was then confirmed whether the thermal environment of the machine room can be predicted by the proposed prediction method.

5.1. Method to Reproduce ITE

The relationship between Figure 6a-2,b-2 is reproduced in an experimental environment in order to clarify the effect of changes in the ITE airflow rate on cooling. The server is substituted by a heat generator that can adjust the fan airflow rate, and the heat generator reproduces the slope of the plot of server airflow rate versus server ambient temperature. As the inlet temperature of the heat generator depends on the rack position, it is difficult to reproduce the slope with an individual heat generator. In this experiment, therefore, the airflow rates of all the heat generators were set to the same setting, and the supply-air temperature was adjusted so that the average ambient temperature and average airflow rate of all the heat generators accorded with the slopes of the plots in Figure 6a-2,b-2. Moreover, it was determined that the temperature zone did not affect the prediction of air performance to which the temperature zone was set so that the experiment was easy.

5.2. Experimental Conditions (Experiments C and D)

All 16 racks are equipped with heat generators. The CRAC is used as the air conditioner. The simulated heating element has a height of 6U, and it is mounted in the target rack at 23U to 28U from the bottom. As shown in

Table 6. Experimental conditions (experiments C and D).

Experimental Conditions	CAC	HAC	Blanking Panel	Airflow Scheme
C-1 C-2	Yes Yes	- -	No Yes	Raised-floor supply room return
D-1 D-2	- -	Yes Yes	No Yes	Raised-floor supply ceiling return

, the experiment was conducted under two conditions: (i) Raised-floor outlet/horizontal inlet method with CAC and BP and (ii) raised-floor outlet/ceiling inlet method with HAC and BP. As mentioned in the previous section, the ambient temperature and airflow rate of the heat generator are matched with the slopes of the plots in Figure 6a-2,b-2. The airflow rate of the heat generator at five points is set for each experimental condition ((i) and (ii) above), and the CRAC supply-air temperature is adjusted to match the slopes of those plots while the inlet air temperature and outlet temperature are monitored. Note that the setting of the CRAC supply-air amount is constant. The fan of the heat generator is voltage-controlled, and the voltage settings at the five points and the assumed airflow rate (measured separately) are listed in

Table 7. Voltage setting and assumed airflow rate of heat generator.

Additional Number	Voltage (V)	Assumed Airflow Rate (m3/s)
-1	26.4	0.24
-2	23.0	0.22
-3	19.5	0.21
-4	17.2	0.19
-5	15.5	0.17

. In addition, the measured items and measurement equipment used in this experiment are detailed in

Table 8. Measured items and measurement instruments (experiments C and D).

Item	Instrument
Supply-air temperature (3 points) Return-air temperature (6 points)	Type-T thermocouple, data logger (GRAPHTEC, GL840)
Heat-generator inlet/outlet temperature (1 point/server)	Type-T thermocouple, data logger (GRAPHTEC, GL840 and GL820)
Power consumption (all servers)	Power meter (HIOKI, PW3365 and 3168) 1 Power meter (Panasonic, T3720N) 2

¹ used in Experiment C; ² used in Experiment D.

. Mean values (averaged over five minutes under stable conditions) were used. Note that in this experiment, the “side blanking panels” shown in Figure 3b were installed even under the “no blanking panel” condition shown in Table 6.

5.3. Comparison of Experimental Results and Prediction Results (Experiments C and D)

5.3.1. Confirmation of Reproduction Status

To confirm whether the heat generator can reproduce the relationship between temperature and airflow rate plotted in Figure 6a-2,b-2, the ambient temperature of the equipment and airflow rate per rack are plotted in Figure 9. As mentioned above, 12 servers were installed in each rack, and one heat generator was installed in each rack. When the results were confirmed, although the variation range of airflow was narrow, because the variable speed range of the built-in fan of the heat generator was narrow, the heat generator was able to roughly reproduce the “tilt” of the 12 servers in terms of both inlet temperature and outlet temperature.

The relationship between airflow rate and power consumption of the heat generator is shown in Figure 10. The results of experiment B were used as the numerical values of the servers to be compared. As mentioned above, although fluctuation in the airflow rate of the heat generator was in the low airflow-rate range of the server, it can be seen that the power consumption tended to increase as the built-in fan of the heat generator accelerated.

5.3.2. Confirmation of Given Conditions

To obtain the values of the given conditions used for prediction, ITE inlet temperature (T_ii), CRAC supply-air amount (m_c), ASE, and ITE-related empirical equations are confirmed hereafter. The experimental results for T_ii, m_c, and ASE under the respective experimental conditions are shown in Figure 11. As given conditions, T_ii uses the respective measured values, and m_c and ASE use the average values of C-1, C-2, D-1, and D-2. As the airflow setting of the CRAC is fixed, m_c is almost constant under each airflow scheme. However, m_c of the ceiling-inlet method was smaller than that of the horizontal-inlet method. It is probable that m_c of the ceiling-inlet method was small due to increased pressure losses such as those due to the ceiling inlet port, the inside of the ceiling, and the return-air duct (from the ceiling to the CRAC inlet port).

The ITE-related empirical equations are obtained by approximating the results for each heat generator fitted per rack shown in Figure 9a and Figure 10. The approximate equation for each rack is given as follows:

$$\frac{m_i}{16}=0.009T_{ii}-0.092 ,$$

(9)

$$\frac{E_i}{16}=\mathrm{30,745}{\left(\frac{m_i}{16}\right)}^3+2191 ,$$

(10)

5.3.3. Comparison of Measurement Results with Prediction Results

Using the given conditions and empirical equations determined as described in the previous section, recirculation flow (R) (step 5 in Table 1), supply-air temperature (T_co) of the CRAC, and return-air temperature (T_ci) (steps 6 and 7 in Table 1) are predicted. Experimental measurement results and prediction results for the recirculation flow and temperature of each measurement point are shown in Figure 12 and Figure 13, respectively. The experimental measurement results are considered first. The experimental measurement results shown in Figure 12 reveal that as the availability of flow (A_f) decreases, recirculation flow (R) tends to increase. In other words, it can be said that the amount of air supplied from the CRAC was insufficient to meet the required airflow rate of the ITE, so the recirculation flow increased. This result is explained as follows. As the supply-air amount is roughly fixed, the ITE inlet temperature (T_ii) becomes high, and the ITE airflow rate increases. In Figure 13, the predicted recirculation flow (R) values are used to predict supply-air temperature (T_co) and return-air temperature (T_ci) with respect to ITE intake air temperature (T_ii). As can be seen from these graphs, as T_ii increases, the temperature difference from T_co to T_ii increases. For example, in the case of D-1, in which R is largest, the temperature rise is 1.5 °C when the inlet temperature is 27 °C; however, when the inlet temperature is 30 °C, the temperature rise is bigger, i.e., 2.1 °C. In each case, the prediction results reproduce the experimental measurement results well, so the proposed temperature-prediction procedure is considered appropriate.

6. Proposed Method for Optimization of Total Energy of a Machine Room

In the previous section, it was confirmed that the thermal environment of the machine room can be predicted by the proposed prediction procedure; in particular, the temperatures of the environments around the CRAC and ITE can be linked. It is therefore possible to predict the power consumption of the HVAC with respect to ITE inlet temperature. In this section, a method for optimizing the power consumption of the machine room, under the assumption that ITE characteristics are understood, is proposed.

6.1. Settlement of Conditions

6.1.1. Predetermined Conditions

It is assumed that 12 ITEs are mounted on each of the 16 racks. The cooling device is a single CRAC. ASE is set from 0.5 to 1.0 in 0.1 increments with indoor airflow performance as a parameter. Further, ITE inlet temperature (T_ii), which is taken as the starting point of the prediction, is predicted in 1 °C increments.

6.1.2. Characteristics of ITE

The characteristics of the ITE (servers) were determined by the least-squares method from the values measured in experiment B. The results of experiment B and the characteristics of the assumed ITE are shown in Figure 14. The ITE airflow rate shown in Figure 14a was set to the upper limit at 34 °C or higher and the lower limit at 13 °C or lower. It was determined by estimating the ITE inlet temperature (T_ii) for which server-fan speed has maximum and minimum values, as shown in Section 4.2.1.

6.1.3. Characteristics of HVAC

Various methods for assuming the characteristics of the HVAC are available; however, as the main focus of this section is to propose a total-energy optimization method, a simple method is used. We interviewed Japanese air-conditioning manufacturers about two characteristics, namely, the (i) maximum cooling capacity with respect to the indoor wet-bulb temperature of a CRAC (rated sensible cooling capacity of 63 kW) used for data centers and the (ii) power consumption under partial load, and organized those characteristics.

The specifications of the CRAC are listed in

Table 9. Specifications of computer-room air-conditioning (CRAC).

Rated condition	Cooling capacity Energy consumption Return-air conditions Outdoor-air condition Airflow, external static pressure (ESP)	63.0 kW (sensible) 21.6 kW 27.0 °C DB, 19.0 °C WB 35.0 °C DB 5.3 m3/s, 120 Pa
Control setting	Capacity control Airflow control	Inverter speed control Fixed airflow rate

, and example characteristics of the CRAC are shown in Figure 15. The wet-bulb-temperature range of inlet air under which the CRAC is operable is 12 to 24 °C WB. Table 1 exemplifies the fact that the dry-bulb temperature of the return air is used for calculating the power consumption of the CRAC. However, in this section, the wet-bulb temperature of the return air is used in consideration of the characteristics of the CRAC. For the simulation, the absolute humidity was fixed at 20 °C and 50% RH, and the wet-bulb temperature was calculated from the dry-bulb temperature of the return air.

To estimate total energy consumption of the CRAC, annual power consumption is calculated. The power consumption in 5 °C increments is calculated, and duration times of outside-air temperature in increments of 5 °C are added up. The duration time used is meteorological data for Tokyo [23] published on Energy Plus. It is plotted in Figure 16.

6.2. Prediction Results

The results of predicting the total energy consumption of the machine room by using the prediction procedure described in Section 2.3 are shown hereafter.

6.2.1. Prediction Results for Inside Environment

The results of calculating the availability of flow (A_f) and circulation flow (R) in regard to ITE inlet temperature (T_ii) for each ASE by performing steps 1 to 5 shown in Table 1 are shown in Figure 17. As shown in Figure 17a, A_f decreases with T_ii. In this prediction, A_f ranges from 1.2 to 3.4. Moreover, when A_f decreases, the amount of air supplied to meet the required airflow rate to the ITE shifts to the insufficient side, so the recirculation flow tends to increase. Actually, as shown in Figure 17b, R increases as T_ii increases. When ASE is 1, recirculation does not occur (R = 0), but when ASE is 0.5, R is 0.46 at maximum, and nearly half of the exhaust-air volume of the ITE is recirculated. In addition, as shown in Figure 14a, the airflow rate of the ITE does not increase nor decrease in the ranges in which the ITE inlet temperature (T_ii) is respectively 13 °C or lower and 34 °C or higher. Consequently, R does not fluctuate in those ranges.

The results obtained by executing steps 6 and 7 and determining the temperature of each part around the ITE and around the CRAC are shown in Figure 18. In the machine room with an ASE in the range of 0.5 to 0.7, the return-air wet-bulb temperature in the low-temperature range fell below the operable range of the CRAC; therefore, only the results within the operable range are shown. It can be seen from the graph that the higher ITE inlet temperature (T_ii), the larger the temperature rise from supply-air temperature (T_co) to T_ii, except in the case of the machine room with almost no recirculation (i.e., ASE of 0.9 or 1.0). At the upper limit of the ITE airflow rate, the temperature-rise range was 7.3 °C at an ASE of 0.5, 4.7 °C at an ASE of 0.6, 2.8 °C at an ASE of 0.7, and 1.4 °C at an ASE of 0.8. Furthermore, although the amount of air supplied to the CRAC was constant, the power consumption of the ITE (E_i) (i.e., cooling load) increased with T_ii; thus, the temperature difference between return-air temperature (T_ci) and supply-air temperature (T_co) of the CRAC tended to increase.

6.2.2. Prediction Results for Power Consumption of CRAC

The wet-bulb temperature of the return air is calculated from the return-air (dry-bulb) temperature (T_ci) of the CRAC (shown in Figure 18), which is plotted in Figure 19, and the power consumption (E_c) of the CRAC is calculated for each outside air temperature (step 8 in Table 1). Then, E_c is time-accumulated to calculate the annual power consumption of the CRAC (ΣE_c) in Tokyo. The cooling load of the CRAC is equivalent to the power consumption of the ITE (E_i) (for 16 racks) obtained from Figure 14b. As described in the previous section, Figure 19 only shows the results that fall within the operable range.

The annual power consumption (ΣE_c) of the CRAC with respect to ITE inlet temperature (T_ii) is plotted in Figure 20. In the temperature range below 34 °C, where ITE airflow rate varies, when the curve is such that T_ii is from 22 to 24 °C, ΣE_c reaches its lowest point in the cases of all the machine-room airflow performances. On the contrary, in the range above 34 °C, where ITE airflow rate is constantly at its maximum, as T_ii increases, ΣE_c decreases. In the range of this prediction, ΣE_c becomes minimum in the range in which the ITE airflow rate is maximum (T_ii = 40 °C) only when the ASE equals 0.5. As shown in Figure 21, and as generally known, as T_ii increases, the coefficient of performance (COP) of the CRAC increases, and its operation becomes more efficient. However, as ITE power consumption (which is equivalent to cooling load) also increases, ΣE_c tends to increase in the ITE-inlet-temperature range from the lowest point to 34 °C.

6.2.3. Optimization of Total Energy

The predicted annual total power consumption (ΣE_total), which comprises total CRAC annual power consumption (ΣE_c) and ITE annual power consumption (ΣE_i), is shown in Figure 22. As ΣE_i is larger than ΣE_c, the range in which CRAC power consumption varies is relatively small, and the effect of a decrease in ITE power consumption due to a decrease in ITE inlet temperature (T_ii) is greater. Therefore, in the range of T_ii from 14 °C to 33 °C (in which ITE airflow rate varies), as T_ii drops, ΣE_total decreases. As ΣE_i does not fluctuate below 14 °C, as T_ii drops, ΣE_total increases due to the influence of ΣE_c. As a result, except for the machine room with an ASE of 0.5, which is outside the operable range of the CRAC, ΣE_total reaches its optimum point at 14 °C. As for the machine room with an ASE of 0.5, the optimum point is to operate at the lower limit of the operable range. Moreover, if T_ii is 33 °C or higher, ΣE_total starts to decrease due to the influence of ΣE_c.

In this study, ITE power consumption was used as the cooling load of the CRAC; however, other loads (such as the thermal conduction) also affect the cooling load. In particular, when the ITE inlet temperature is lowered, as the case in this study, the inside temperature is lowered, and the difference between inside and outside temperatures is increased. The inflow of heat from outside therefore becomes larger than that when the ITE inlet temperature is high. As a result, the lowest point of annual total energy consumption might shift to the higher ITE-inlet-temperature side. In this study, the ITE was assumed to have the characteristics revealed in Section 4; however, some ITE used around the world have airflow rates that do not change with ambient temperature, and other ITE have airflow rates that change in steps. In an actual machine room, it is considered that different kinds of ITE are combined at the rack level. That is also one of the factors that change the optimum point of the total energy of the machine room.

In reality, the data-center operator and the customer often conclude a “service level agreement” (SLA) to define the temperature conditions inside the machine room. Commonly quoted is the “recommended environmental range” in ASHRAE’s thermal guidelines [1]. This envelope requires the dry-bulb temperature of the ITE inlet port to be kept at 18 to 27 °C. The energy-saving effect when ITE inlet temperature (T_ii) is changed within that range is verified hereafter. That is, the annual power consumption is calculated under T_ii within that dry-bulb-temperature range for minimizing ΣE_c and under T_ii within that range for minimizing ΣE_total with T_ii of 27 °C as a reference, and the energy-saving effect is verified as follows.

The results of the calculation and verification are shown in Figure 23. First, if T_ii for minimizing ΣE_c is focused on, it becomes clear that ΣE_c hardly changes (99% to 100%) compared to the standard. This is because the standard (T_ii = 27 °C) is close to the bottom of the curves shown in Figure 20. On the contrary, ΣE_i decreased by 3 to 7% for each T_ii. Although the reduction effect of ΣE_total depends on the ASE, it ranged from 3 to 6%. All Tii values for minimum ΣE_total were at the lower limit of the recommended range (T_ii = 18 °C). However, as T_co was low in regard to the CRAC, ΣE_c increased compared to the standard. In particular, ΣE_c increased by 4% in the case of the machine room with an ASE of 0.5 (i.e., a large recirculation flow). However, ΣE_i was greatly reduced (by 11%), and ΣE_total was reduced by 8 to 9%.

Note that care must be taken when evaluating the optimization results by using power-usage effectiveness (PUE). Partial PUE (pPUE) is defined by the following equation, and pPUE for ITE inlet temperature (Tii) is plotted in Figure 24.

$$pPUE=\frac{\Sigma E_{total}}{\Sigma E_i}=\frac{\Sigma E_i+\Sigma E_c}{\Sigma E_i} ,$$

(11)

According to Equation (11), as Tii rises, the ITE annual power consumption (ΣE_i) increases, so pPUE decreases. The original desirable state is to minimize the annual total power consumption (ΣE_total); however, if pPUE is evaluated and the policy is set, ΣE_total may increase.

7. Concluding Remarks

In this study, existing “air performance” indicators, such as air segregation efficiency (ASE), were expanded, and a method for predicting a thermal environment by simply correlating supply-air temperature with ITE inlet temperature was proposed. Experiments were conducted in a verification room simulating a data center, and the experimental results were compared with the predicted results. These results demonstrate that the indoor thermal environment can be predicted accurately by the proposed method. In this manner, the supply-air and return-air temperatures of HVAC can be predicted from the ITE inlet temperature, and from those predicted temperatures, the power consumption of HVAC can be predicted. Based on this prediction method, a method for optimizing the total energy consumption of the data center—by using ITE inlet temperature as a parameter—was devised.

The findings from the predictions are as follows:

• The results of the trial calculation assuming a machine room with an ASE in the range of 0.5 to 1.0 and using the characteristics of a certain CRAC as an example revealed that (i) to minimize the annual power consumption of the CRAC, the ITE inlet temperature should be selected in the range of 22 to 24 °C, and (ii) to minimize the annual total energy consumption of the machine room, it is better to select the highest ITE inlet temperature among those that give the lowest rotation speed of the ITE’s built-in fan.
• Within the recommended environmental range of the ASHRAE thermal guidelines (ITE inlet temperature of 18 °C to 27 °C), the total energy consumption was lowest when the ITE inlet temperature was 18 °C. It was shown that by lowering the ITE inlet temperature from 27 °C to 18 °C, the annual power consumption of the CRAC will increase; however, annual power consumption of the ITE will be significantly reduced, so the total energy consumption will be reduced by 8 to 9%.

Various methods for assuming the characteristics of the HVAC are available; however, as the main focus of this section is to propose a total-energy optimization method, a simple method is used. We believe that the proposed method can also be applied for predicting the power consumption of complicated heat sources such as economizers.

In this study, it was assumed that ITE power consumption and cooling load are equal, and the annual power consumption of the CRAC was calculated; however, in reality, other loads such as the thermal conduction also have an effect. In particular, it should be noted that when the inside temperature is lowered, the temperature difference between the inside and outside widens and the heat inflow increases. Moreover, although the temperature and power consumption were predicted by unifying the ITE (servers) with the same characteristics, some ITE in the world have airflow rates that do not change with ambient temperature and other ITE have airflow rates that change in steps. The presence of these kinds of ITE also contributes to changing the optimum point of total power consumption, etc. Therefore, it is also important to correctly understand the ITE group implemented in the data center that is targeted for prediction. When these factors are taken into consideration, it becomes clear that the lowest point of annual total energy consumption of the CRAC may shift to the side where ITE inlet temperature is higher.