Improving Prediction Accuracy Concerning the Thermal Environment of a Data Center by Using Design of Experiments

: In data centers, heating, ventilation, and air-conditioning (HVAC) consumes 30–40% of total energy consumption. Of that portion, 26% is attributed to fan power, the ventilation e ﬃ ciency of which should thus be improved. As an alternative method for experimentations, computational ﬂuid dynamics (CFD) is used. In this study, “parameter tuning”—which aims to improve the prediction accuracy of CFD simulation—is implemented by using the method known as “design of experiments”. Moreover, it is attempted to improve the thermal environment by using a CFD model after parameter tuning. As a result of the parameter tuning, the di ﬀ erence between the result of experimental-measurement results and simulation results for average inlet temperature of information-technology equipment (ITE) installed in the ventilation room of a test data center was within 0.2 ◦ C at maximum. After tuning, the CFD model was used to verify the e ﬀ ect of advanced insulation such as raised-ﬂoor ﬁxed panels and show the possibility of reducing fan power by 26% while keeping the recirculation ratio constant. Improving heat-insulation performance is a di ﬀ erent approach from the conventional approach (namely, segregating cold / hot airﬂow) to improving ventilation e ﬃ ciency, and it is a possible solution to deal with excessive heat generated in data centers.


Introduction
In data centers, information-technology equipment (ITE) expends an enormous amount of electricity on information processing. Data centers are thus known as facilities that consume huge amounts of electricity. In fact, it is estimated they account for 1% of global electricity consumption [1]. Although, almost all the power consumption of the ITE in a data center is converted into sensible heat, the failure rate of the ITE increases as its inlet temperature increases [2]. From the standpoint of preventing "information-communication stagnation" due to failure of ITE, the ITE is therefore cooled 24 h a day, 365 days a year. As an operational target for cooling, the thermal guidelines [2] presented by the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) TC 9.9 are often cited. As for a data center, heating, ventilation, and air-conditioning (HVAC) consumes 30-40% of the energy of the whole facility [3]. Accordingly, better energy conservation is strongly demanded. Especially in the case of a data center, fans consume a large amount of power because the sensible-heat load of the housed ITE is processed with a large volume of air, which accounts for about 26% of the power consumption of the HVAC [4]. For that reason, improving the efficiency of air-conditioning airflow has been actively researched.

Parameter-Tuning Method
The design of experiments (DOE)-developed by R. A. Fisher of the United Kingdom for streamlining agricultural experiments-is a method for assignment and analysis of experiments [19]. Compared with the conventional one-factor-at-a-time approach, this DOE can obtain sufficient information in a small number of experiments, so it makes it possible to study the effects of multiple factors on a certain process [20].
The aim of this study is to improve prediction accuracy of CFD simulations by using this DOE. The parameter-tuning method for multiple characteristics proposed by Nishina et al. is used so that the same CFD model can be accurately reproduced even if the air volume of the air-conditioner varies [21]. As for this multi-character parameter-tuning method, before the two steps that respectively reduce deviation and bring values closer, a step for reducing the difference in deviation patterns with respect to the signal factor for each characteristic is added.
The DOE for parameter tuning when K types of characteristics are given as P k (k = 1, 2, . . . , K) is shown in Table 1. This is a direct-product experiment in which the parameter-tuning factor is the inner factor, and the signal factor (G) is the outer factor. Mixed-level orthogonal array L 18 is used for the inner orthogonal array. Here, x jk is taken as the experimental measurement of signal factor G j at the j-th level in regard to characteristic P k , and c ijk is taken as the simulation result of characteristic P k and signal factor G j in regard to the i-th analysis of the inner orthogonal array. The difference between the simulation results and the experimental measurements is taken as y ijk , given as The condition regarding the reproducibility of the deviation pattern of the signal factor for each characteristic is set. The factorial effect of the tuning factor is calculated with S i , calculated by using the following formula, as the analysis characteristic. For each factor, a level with a small average S i value is selected.

Second Step
The analysis characteristic for the variation of signal factors is defined as the sum of squared deviation S i ', and the condition to minimize the value of S i ' is given as: Mean value y i·· for each analysis is obtained, and the factorial effect of the tuning factor on the mean value is calculated. A level with an average value close to zero is selected.
The handling of the interaction between factors when using the orthogonal array is described hereafter. Generally, in an orthogonal array, the main effect of a factor appears as the effect of the column to which the factor is assigned, and the interaction between factors appears in a different column from the column to which the factor is assigned. When another factor is assigned to the column in which the interaction appears, the effect of that factor and this interaction appear to be confounding [22]. In this study, the interaction is not considered so as to select the optimum level with high reproducibility, even if the interaction has a certain influence. Therefore, in regard to the orthogonal array, mixed-system orthogonal array L 18 (in which interaction does not appear in a specific column; i.e., interactions appear almost uniformly in all columns) is used, and the superiority or inferiority between the levels is evaluated by using the average value of the analysis characteristic for each level. Moreover, prediction accuracy is confirmed at the change of level in each step, and unexpected deterioration of accuracy due to interaction is eliminated.

Overview of Verification Room
The verification room simulating a data center is shown in Figure 1, and the specifications of the experimental equipment are detailed in Table 2. The rack layout in the verification room is a cold-aisle/hot-aisle type, and the cold aisle is fitted with cold-aisle containment (CAC) that surrounds with horizontal components that pass over the racks. At the center of the verification room (hot-aisle part), the room, raised floor, and ceiling are divided by temporary partitions (antistatic flameproof films), and the target range of verification room is 11.0 m long and 5.0 m deep. Of the 22 19-inch racks, 20 racks are equipped with one heat generator with a 2-kW heater, and the other two (unused) racks are blocking the door side of the racks. The heat generators (with height of 6U) are installed at heights of 23U to 28U from the rack bottom. As for the CFD model, the following conditions are set: airflow of the CRAC, air-supply temperature of the CRAC, airflow at the heat generator, and calorific value of the heat generator. They are values measured at the points listed in Table 3 or calculated values based on the measured values. The air volume of the CRAC was calculated from average air velocity and effective area of the inlet of the CRAC, and the air volume of the heat generator was calculated from the power consumption and the inlet/outlet temperature difference. Inlet-air temperature of the heat generator is a prediction target of the CFD simulation.
Energies 2020, 13, 4595 6 of 21 horizontal components that pass over the racks. At the center of the verification room (hot-aisle part), the room, raised floor, and ceiling are divided by temporary partitions (antistatic flameproof films), and the target range of verification room is 11.0 m long and 5.0 m deep. Of the 22 19-inch racks, 20 racks are equipped with one heat generator with a 2-kW heater, and the other two (unused) racks are blocking the door side of the racks. The heat generators (with height of 6U) are installed at heights of 23U to 28U from the rack bottom.

Overview of CFD Model
Commercial software was used for CFD analysis. The simulation set-up is shown in Table 4. The discretization algorithm used is SIMPLEC (Semi-Implicit Method for Pressure-Linked Equations Consistent) based on the orthogonal finite-volume method [23]. As the turbulence model, the standard k-ε model, which is commonly chosen in CFD analysis of data centers [7,8,[13][14][15][16]18,24], was selected. Fundamental governing equations are the continuity equation, the Navier-Stokes equation and the energy equation. The buoyancy is considered in the Navier-Stokes equation using the Boussinesq approximation. This CFD software does not consider the surface roughness of the materials. The walls enclosing the verification room are modeled as adiabatic. Convergence is considered when calculation error in the entire analysis area is 10 −3.5 . VanGilder et al. reported that grid size used for CFD analysis significantly affects prediction accuracy in a rough range [24]. In this study, to eliminate the effect of different grid sizes as much as possible, the same number of partitions (23,400,000 grids) was used for all models.

Type of Analysis
Steady State

SIMPLEC algorithm
Turbulence Model standard k-ε turbulence model

Continuity equation Navier-Stokes equation for incompressible
Newtonian fluids Energy equation

Computational Grid
Non-uniform staggered Cartesian grid

Boundary Condition
Inlet domain: CRAC outlet port Outlet domain: CRAC inlet port Initial temperature: 25 • C See Table 5 below for other setting values. A model of the verification room is shown in Figure 2. Aperture ratios are assigned to the raised-floor perforated panels of the cold aisle and rack door, respectively. The raised-floor perforated panel has alternately arranged aperture ratios of 25% and 50%. In the CFD model, the aperture ratio is uniform (37.5%). A ratio of 76% (catalog value) is assigned to aperture of the rack door.

Signal Factors and Characteristics
To improve the accuracy of the CFD model, the goal of this study is to be able to reproduce the simulation results with the same model even if the airflow supplied by the air conditioner varies. Therefore, for the experiments conducted in the verification room, fan frequency of the CRAC was set to 45 Hz (G 1 ), 54 Hz (G 2 ), and 63 Hz (G 3 ) with air-supply volume of the CRAC as a signal factor. The conditions assigned in the experiments are listed in Table 5. In addition, the inlet temperature of the heat generator, which is the prediction target, is measured 10 times (namely, 10 racks), as shown in Table 3. Since 10 temperature measurements are targeted, a multi-characteristic DOE is applied as a combination of 10 characteristics.

Energy equation Computational Grid
Non-uniform staggered Cartesian grid

Boundary Condition
Inlet domain: CRAC outlet port Outlet domain: CRAC inlet port Initial temperature: 25 °C See Table 5 below for other setting values.

Signal Factors and Characteristics
To improve the accuracy of the CFD model, the goal of this study is to be able to reproduce the simulation results with the same model even if the airflow supplied by the air conditioner varies. Therefore, for the experiments conducted in the verification room, fan frequency of the CRAC was set to 45 Hz (G1), 54 Hz (G2), and 63 Hz (G3) with air-supply volume of the CRAC as a signal factor. The conditions assigned in the experiments are listed in Table 5. In addition, the inlet temperature of the heat generator, which is the prediction target, is measured 10 times (namely, 10 racks), as shown in Table 3. Since 10 temperature measurements are targeted, a multi-characteristic DOE is applied as a combination of 10 characteristics.

Prameter-Tuning Factors
Factors that are considered to affect the results of CFD analysis are extracted as listed in Table 6. As for the level indicating the presence or absence of a model, the first level has a model, and the second level has no model. As for the levels related to the numerical values of the model, the first level is a theoretical value or a measured value, the second level is half the value of the first level, and the third level is not considered in the model. However, the second level of thermal resistance (factor C), calculated by thermal conductivity and thickness of the materials, is twice the thickness of the first level. Column H, to which no factor is assigned, is the error column (e). Each parameter-tuning factor is outlined as follows.

A. Presence or Absence of Obstructions
The raised floor frame, airflow guide, piping and cables, and air-conditioner inlet structure (shown in Figure 3) are modeled as obstacles. Here, the airflow guide is a steel plate-obliquely fixed immediately below the air-conditioner outlet-for smoothly introducing air-conditioned air into the raised-floor space.
Energies 2020, 13, 4595 9 of 21 6 mm. For the third level, the dummy-level technique is used, and a standard first level is substituted as the dummy level.
G. Aperture Ratio of CAC Although no large gaps exist in the CAC itself, a gap of about 2 mm exists in the 80-mm fixing member between the upper part of the rack and the CAC horizontal members. This measured value is used as the aperture ratio of the CAC.

Implementation of Parameter Tuning
The signal factor and parameter-tuning factor are assigned as shown in Table 1 and analyzed on the basis of the DOE. From the analysis results shown in Appendix A, analytical characteristics Si, Si', and y •• were calculated and averaged at each level of the fitting factor to create the factor-effect diagram shown in Figure 4. As for analysis characteristic Si, the level reflecting any of the parametertuning factors, including the dummy level of factor F, in the CFD model is smaller, and it can be said that the reproducibility of the results when the air-supply capacity of the CRAC is varied is good. On the other hand, if theoretical values and measured values (first level) of aperture ratio of the CRAH (B), thermal resistance (C), and aperture ratio of the blank panels (E) are taken as conditions for the CFD model, the reproducibility becomes worse than that of the second level.

B. Aperture Ratio of CRAH
The unused computer-room air handler (CRAH) adjacent to the CRAC is tuned off. However, a pressure difference between the inlet and outlet ports occurs, so it is possible that the inside of the CRAH becomes a leakage path and affect the CFD results. Static pressure inside the CRAH during rated operation were obtained from the P-Q characteristic curve of the internal fans as well as rated air capacity and rated external static pressure of the CRAH, and resistance coefficient was calculated from internal static pressure of the CRAH and airflow velocity on the inlet surface. An aperture ratio corresponding to the resistance coefficient is assigned to the inlet port of the CRAH.

C. Thermal Resistance
Heat transmission of the raised-floor fixed panel, rack case, heat generating case, CAC, and blank panel are considered. Thermal-resistance value was obtained from thermal conductivity and thickness of the materials, and the heat-transfer coefficient was calculated by using the automatic calculation function of the CFD software.

D. Aperture Ratio of Fixed Panels
Measured values were used for the aperture ratio of raised-floor fixed panels. The air-gap size between panels was about 10 mm per 26 fixed panels (with width of 300 mm).

E. Aperture Ratio of Blank Panels
The aperture ratio is separately assigned to the following blank panels: (i) a blank panel to be installed on the mounting rail in the rack in which ITE is not installed; (ii) a side blank panel that blocks the gap between the rack side and mount rail; and (iii) a lower blank panel that blocks gap at the bottom of the rack and the bottom edge of the blank panels and side blank panels. Measured values are used for each aperture ratio.

F. Presence or Absence of Gap between Racks
The gap between the racks, namely, 6 mm, is considered in the model. In the case that no gap exists, the space between the racks is closed with a rectangular parallelepiped model with a width of 6 mm. For the third level, the dummy-level technique is used, and a standard first level is substituted as the dummy level.
G. Aperture Ratio of CAC Although no large gaps exist in the CAC itself, a gap of about 2 mm exists in the 80-mm fixing member between the upper part of the rack and the CAC horizontal members. This measured value is used as the aperture ratio of the CAC.

Implementation of Parameter Tuning
The signal factor and parameter-tuning factor are assigned as shown in Table 1 and analyzed on the basis of the DOE. From the analysis results shown in Appendix A, analytical characteristics S i , S i ', and y i·· were calculated and averaged at each level of the fitting factor to create the factor-effect diagram shown in Figure 4. As for analysis characteristic S i , the level reflecting any of the parameter-tuning factors, including the dummy level of factor F, in the CFD model is smaller, and it can be said that the reproducibility of the results when the air-supply capacity of the CRAC is varied is good. On the other hand, if theoretical values and measured values (first level) of aperture ratio of the CRAH (B), thermal resistance (C), and aperture ratio of the blank panels (E) are taken as conditions for the CFD model, the reproducibility becomes worse than that of the second level.  In the first step of parameter tuning, a level with a small analysis characteristic Si is selected. In the second step, the level at which analysis characteristic Si' is small is selected, and in the third step, the level at which analysis characteristic y •• is close to zero is selected. Basically, the level selected in the first step is given the highest priority, and the level selected the second step is given second priority. When the level is changed in a lower-order step, the optimum level selected in a higherorder step will be changed. Consequently, the factors that change the level in each step are limited to a single factor. Prediction accuracy is then confirmed after the change. The indexes taken to confirm prediction accuracy are: (i) average deviation ( y ) between the simulated heat-generator inlet temperatures of 10 racks and the corresponding measurement results and (ii) residual sum of squares, Se.
The levels to be selected by parameter tuning in each step are listed in Table 7, and the confirmation results concerning prediction accuracy are shown in Figure 5. As described above, in the first step, the level at which analysis characteristic Si was small for each factor was selected. In the second step, parameter-tuning factor D with the largest decrease in analysis characteristic Si' was changed from the first level to the second level. Prediction accuracy is confirmed after the second step. Residual sum of squares Se and average deviation y remained almost unchanged. In the third step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  In the first step of parameter tuning, a level with a small analysis characteristic S i is selected. In the second step, the level at which analysis characteristic S i ' is small is selected, and in the third step, the level at which analysis characteristic y i·· is close to zero is selected. Basically, the level selected in the first step is given the highest priority, and the level selected the second step is given second priority. When the level is changed in a lower-order step, the optimum level selected in a higher-order step will be changed. Consequently, the factors that change the level in each step are limited to a single factor. Prediction accuracy is then confirmed after the change. The indexes taken to confirm prediction accuracy are: (i) average deviation (y e ) between the simulated heat-generator inlet temperatures of 10 racks and the corresponding measurement results and (ii) residual sum of squares, S e .
The levels to be selected by parameter tuning in each step are listed in Table 7, and the confirmation results concerning prediction accuracy are shown in Figure 5. As described above, in the first step, the level at which analysis characteristic S i was small for each factor was selected. In the second step, parameter-tuning factor D with the largest decrease in analysis characteristic S i ' was changed from the first level to the second level. Prediction accuracy is confirmed after the second step. Residual sum of squares S e and average deviation y e remained almost unchanged. In the third step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares S e ; however, average deviation y e was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L 18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  Level  1  2  1  2  3  1  2  3  1  2  3  1  2  3  1  2  1  1  2  3 First Step residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved. Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.

2 3 2 3 2 3 2 1′ 2 3 First
Step residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials. Step residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials. Step residual sum of squares Se; however, average deviation y was close to zero, and prediction accurac was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tunin factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulation With the proposed parameter-tuning method, by performing the analysis assigned to mixed-leve orthogonal array L18 in terms of only the number of signal factors and performing all three steps fo confirming the tuning result in terms of only the number of signal factors, an optimum combinatio was derived after 63 trials. Step residual sum of squares Se; however, average deviation y was close to zero, and was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) an factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requir With the proposed parameter-tuning method, by performing the analysis assi orthogonal array L18 in terms of only the number of signal factors and performi confirming the tuning result in terms of only the number of signal factors, an op was derived after 63 trials. Step residual sum of squares Se; however, average deviation y was close to was significantly improved.
Originally, to derive the optimal combination of signal factors (3 factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force meth With the proposed parameter-tuning method, by performing the ana orthogonal array L18 in terms of only the number of signal factors and confirming the tuning result in terms of only the number of signal fact was derived after 63 trials.

2 3 2 3 2 First
Step residual sum of squares Se; however, average deviation y was significantly improved.
Originally, to derive the optimal combination of si factors (2 levels × 2 factors and 3 levels × 5 factors), the br With the proposed parameter-tuning method, by perfo orthogonal array L18 in terms of only the number of sign confirming the tuning result in terms of only the numbe was derived after 63 trials. Step Step shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.
shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 shown in the factor-effect diagram, change in prediction accuracy after the third step incr residual sum of squares Se; however, average deviation y was close to zero, and prediction acc was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-t factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simula With the proposed parameter-tuning method, by performing the analysis assigned to mixed orthogonal array L18 in terms of only the number of signal factors and performing all three ste confirming the tuning result in terms of only the number of signal factors, an optimum combin was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 shown in the factor-effect diagram, change in prediction accuracy after the residual sum of squares Se; however, average deviation y was close to zero, and was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) an factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requir With the proposed parameter-tuning method, by performing the analysis assi orthogonal array L18 in terms of only the number of signal factors and performi confirming the tuning result in terms of only the number of signal factors, an op was derived after 63 trials.  Step shown in the factor-effect diagram, change in prediction accuracy a residual sum of squares Se; however, average deviation y was close to was significantly improved.
Originally, to derive the optimal combination of signal factors (3 factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force meth With the proposed parameter-tuning method, by performing the ana orthogonal array L18 in terms of only the number of signal factors and confirming the tuning result in terms of only the number of signal fact was derived after 63 trials. Step shown in the factor-effect diagram, change in predict residual sum of squares Se; however, average deviation y was significantly improved.
Originally, to derive the optimal combination of si factors (2 levels × 2 factors and 3 levels × 5 factors), the br With the proposed parameter-tuning method, by perfo orthogonal array L18 in terms of only the number of sign confirming the tuning result in terms of only the numbe was derived after 63 trials. step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 step, the third level of factor D, at which the analysis characteristic is close to zero, was selected. As shown in the factor-effect diagram, change in prediction accuracy after the third step increased residual sum of squares Se; however, average deviation y was close to zero, and prediction accuracy was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parameter-tuning factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 simulations. With the proposed parameter-tuning method, by performing the analysis assigned to mixed-level orthogonal array L18 in terms of only the number of signal factors and performing all three steps for confirming the tuning result in terms of only the number of signal factors, an optimum combination was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 2 1′ 1 step, the third level of factor D, at which the analysis characteristic is close to zero, was s shown in the factor-effect diagram, change in prediction accuracy after the third step residual sum of squares Se; however, average deviation y was close to zero, and predictio was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) and parame factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requires 2916 si With the proposed parameter-tuning method, by performing the analysis assigned to m orthogonal array L18 in terms of only the number of signal factors and performing all thre confirming the tuning result in terms of only the number of signal factors, an optimum co was derived after 63 trials.  1 2 1 2 3 1 2 3 1 2 3 1 2 3 1 step, the third level of factor D, at which the analysis characteristic is close to ze shown in the factor-effect diagram, change in prediction accuracy after the residual sum of squares Se; however, average deviation y was close to zero, and was significantly improved.
Originally, to derive the optimal combination of signal factors (3 levels) an factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force method requir With the proposed parameter-tuning method, by performing the analysis assi orthogonal array L18 in terms of only the number of signal factors and performi confirming the tuning result in terms of only the number of signal factors, an op was derived after 63 trials.  Step step, the third level of factor D, at which the analysis characteristic is c shown in the factor-effect diagram, change in prediction accuracy a residual sum of squares Se; however, average deviation y was close to was significantly improved.
Originally, to derive the optimal combination of signal factors (3 factors (2 levels × 2 factors and 3 levels × 5 factors), the brute-force meth With the proposed parameter-tuning method, by performing the ana orthogonal array L18 in terms of only the number of signal factors and confirming the tuning result in terms of only the number of signal fact was derived after 63 trials.

Third
Step ✓ ✓ ✓ ✓ ✓ step, the third level of factor D, at which the analysis ch shown in the factor-effect diagram, change in predict residual sum of squares Se; however, average deviation y was significantly improved.
Originally, to derive the optimal combination of si factors (2 levels × 2 factors and 3 levels × 5 factors), the br With the proposed parameter-tuning method, by perfo orthogonal array L18 in terms of only the number of sign confirming the tuning result in terms of only the numbe was derived after 63 trials.  Reproducibility of the actual inlet temperature of a heat generator is confirmed as follows. The measurements and simulation results of heat-generator inlet temperature for each rack when CRAC fan frequency was varied [45 Hz (G1), 54 Hz (G2), and 63 Hz (G3)] are plotted in Figure 6. Each step of the simulation is considered first. The simulation results of the first step and those of the second step show no significant difference under any CRAC airflow conditions. Next, compared to the first and second steps, the third step reveals that the heat-generator inlet temperature decreased, especially in the racks far from the CRAC, namely, racks 1-5 to 11 and 2-5 to 11. It is considered that the reasons for this result are twofold: (i) there is no bypass air flowing from inside of the raised floor to the hot aisle via the gap in the fixed panels and (ii) sufficient air-conditioning air supply passed through the perforated panel in the raised floor.
Next, the simulation results of the third step after parameter tuning are compared with the measurements. As for the first row of racks, although deviations in temperature of 1 to 1.5 °C and 3 °C exist for rack numbers 1-1 and 1-11, as for the other racks, the experimental measurements could be roughly reproduced. Also, as for the second row, the experimental measurements could be predicted accurately. Rack number 1-11 shows large deviations, which suggest that a factor that not reflected in the CFD model is increasing the heat-generator inlet temperature. Under the assumption that there is a gap in the part where the vertical member of the CAC and the rack number 1-11 contact, the results that reflect this gap in the CFD model for CRAC fan frequency of 63 Hz (G3) is shown in Figure 7. As for the result of changing the gap in intervals of 10 mm, when the gap was 30 mm, the simulated heat-generator inlet temperature of rack number 1-11 was close to the measured value. Also, the assumed gap did not adversely affect the prediction results of other inlet temperatures.
As mentioned above, in the existing literature [17,18], the influence of the air gap between the raised-floor fixed panels is mentioned. However, prediction accuracy of this model was higher if the gap was not reflected in the model. In this CFD model, the bypass route that passes through the stopped CRAH is modeled, and as a bypass route, it exists in parallel with the gap between the raisedfloor fixed panels. Since this opening is larger than the gap between the raised-floor fixed panels, the influence of the presence or absence of the model of the fixed-panel gap may be reduced. Reproducibility of the actual inlet temperature of a heat generator is confirmed as follows. The measurements and simulation results of heat-generator inlet temperature for each rack when CRAC fan frequency was varied [45 Hz (G 1 ), 54 Hz (G 2 ), and 63 Hz (G 3 )] are plotted in Figure 6. Each step of the simulation is considered first. The simulation results of the first step and those of the second step show no significant difference under any CRAC airflow conditions. Next, compared to the first and second steps, the third step reveals that the heat-generator inlet temperature decreased, especially in the racks far from the CRAC, namely, racks 1-5 to 11 and 2-5 to 11. It is considered that the reasons for this result are twofold: (i) there is no bypass air flowing from inside of the raised floor to the hot aisle via the gap in the fixed panels and (ii) sufficient air-conditioning air supply passed through the perforated panel in the raised floor.
Next, the simulation results of the third step after parameter tuning are compared with the measurements. As for the first row of racks, although deviations in temperature of 1 to 1.5 • C and 3 • C exist for rack numbers 1-1 and 1-11, as for the other racks, the experimental measurements could be roughly reproduced. Also, as for the second row, the experimental measurements could be predicted accurately. Rack number 1-11 shows large deviations, which suggest that a factor that not reflected in the CFD model is increasing the heat-generator inlet temperature. Under the assumption that there is a gap in the part where the vertical member of the CAC and the rack number 1-11 contact, the results that reflect this gap in the CFD model for CRAC fan frequency of 63 Hz (G 3 ) is shown in Figure 7. As for the result of changing the gap in intervals of 10 mm, when the gap was 30 mm, the simulated heat-generator inlet temperature of rack number 1-11 was close to the measured value. Also, the assumed gap did not adversely affect the prediction results of other inlet temperatures.
As mentioned above, in the existing literature [17,18], the influence of the air gap between the raised-floor fixed panels is mentioned. However, prediction accuracy of this model was higher if the gap was not reflected in the model. In this CFD model, the bypass route that passes through the stopped CRAH is modeled, and as a bypass route, it exists in parallel with the gap between the raised-floor fixed panels. Since this opening is larger than the gap between the raised-floor fixed panels, the influence of the presence or absence of the model of the fixed-panel gap may be reduced.

Improvement Measures
According to the result of the parameter tuning described in the previous section, heat transmission by the components was reflected in the proposed model. However, it is considered that the heat transfer due to air flow accounts for the majority of the heat transfer in the data center, and the heat transfer due to the heat transmission of the structural components has not received much attention. Accordingly, in this section, the effect of improving the indoor thermal environment when the heat-insulation performance of the structural components is improved is verified.
As for the CFD model implementing the third step of tuning, in consideration of heat flow, a heat-insulation material (glass wool; thickness: 10 mm) is uniformly attached to the structural components to improve the thermal environment. Changes in the thermal resistance of each component are listed in Table 8. Thermal resistances of the original components are sufficiently small compared to that of the attached heat-insulation material, so they are ignored. In other words, in the CFD model after improvement, the geometry and grids of computational domain remain the same, and just the thermal-resistance values of the original components are changed to the value equivalent to the glass wool with 10 mm thickness. However, the heat generator is a model of ITE, and it is not realistic to attach a heat-insulation material to ITE. Accordingly, heat resistance of the heat generator's case is taken as the same value before and after the improvement of thermal environment.

Improvement Measures
According to the result of the parameter tuning described in the previous section, heat transmission by the components was reflected in the proposed model. However, it is considered that the heat transfer due to air flow accounts for the majority of the heat transfer in the data center, and the heat transfer due to the heat transmission of the structural components has not received much attention. Accordingly, in this section, the effect of improving the indoor thermal environment when the heat-insulation performance of the structural components is improved is verified.
As for the CFD model implementing the third step of tuning, in consideration of heat flow, a heat-insulation material (glass wool; thickness: 10 mm) is uniformly attached to the structural components to improve the thermal environment. Changes in the thermal resistance of each component are listed in Table 8. Thermal resistances of the original components are sufficiently small compared to that of the attached heat-insulation material, so they are ignored. In other words, in the CFD model after improvement, the geometry and grids of computational domain remain the same, and just the thermal-resistance values of the original components are changed to the value equivalent to the glass wool with 10 mm thickness. However, the heat generator is a model of ITE, and it is not realistic to attach a heat-insulation material to ITE. Accordingly, heat resistance of the heat generator's case is taken as the same value before and after the improvement of thermal environment.

Evaluation Criteria
As an index to measure the effect of improving the thermal environment, the air-management metrics [25] proposed by Tozer et al. are used.
As for air-management metrics, when modeling the air flow path in the data center, in addition to CRAC air volume (m c ) and ITE air volume (m s ): (i) negative-pressure airflow (m n ) (which is the airflow that is induced into the raised floor by the Venturi effect), (ii) bypass-air volume (m b ) (supplied from the raised floor and returns to the CRAC without cooling the ITE), and (iii) recirculation-air volume (m r ) (i.e., that of outlet air from the ITE that returns to the ITE inlet) are incorporated in the model.
By solving the mass and heat-balance equations that hold for each part of these airflow paths, the air-volume ratio can be obtained from the temperature of each part. The air-volume ratio is composed of four elements: (i) balance ratio (BAL) (i.e., ratio of CRAC air flow (m c ) to ITE air flow (m s ); (ii) negative-pressure ratio (NP) (i.e., ratio of negative-pressure air flow (m n ) to CRAC air flow (m c ); (iii) bypass ratio (BP) (i.e., ratio of bypass airflow (m b ) to raised-floor airflow (m f ); and (iv) recirculation ratio (R) (i.e., ratio of recirculation air flow (m r ) to ITE air flow (m s )). The equations for obtaining these respective airflow ratios from temperature are given below: Here, T r is CRAC return-air temperature, T c is CRAC supply-air temperature, T f is raised-floor temperature, T s is ITE inlet temperature, and T h is ITE outlet temperature. As described above, the air-management metrics evaluate the efficiency of air-conditioning from air-volume ratio, and the equation holds under the assumption that heat is only transferred by air flow. This assumption also holds for data-center-evaluation indicators such as "supply heat index" (SHI) and "return heat index" (RHI) by Sharma et al. [26] and "air segregation efficiency" (ASE) by Tozer et al. [27] However, heat transfer in the proposed CFD model is a combination of air flow and heat transmission. Accordingly, in this study, as in the past, it is assumed that heat transfer is due to air flow, and the thermal environment is evaluated on the basis of air-flow ratio. Hereafter, change in the evaluation index due to improvement in heat-insulation performance of the structural components is considered, and the possibility that heat transmission by overall heat transfer affects the thermal environment is suggested.

Result of Improvement
Temperature and air-flow ratio of each part before and after improvement of the thermal environment are shown in Figures 8-10. Even under the condition in which the CAC and blank panels are installed, before improvement, the temperature of the air supplied from the CRAC is raised by 0.4-1.2 • C in the raised floor and by a further 1.3-2.4 • C in the ITE inlet port. At CRAC fan frequency of 45 Hz (G 1 ) with BAL of 0.85, temperature of the air supplied from the CRAC rises by 3.6 • C on reaching the ITE inlet. Under any of the CRAC airflow (fan frequency) conditions, compared with the respective temperatures before the improvement, temperature inside the raised floor decreased by about 0.3 • C and ITE inlet temperature decreased by about 0.8 • C after the improvement. Regarding the air volume ratios obtained from the temperature of each part, the value after improvement was 0.02 lower than that before improvement, 0.02 for NP, 0.04-0.05 for R, and 0.04 for R. It is clear from the change of each index of the air-management metrics that the improvement of the thermal environment has the effect of improving heat-insulation performance. It is also clear that heat-insulation performance affects all three airflow paths, namely, the negative-pressure flow path, bypass flow path, and recirculation flow path.
R. It is clear from the change of each index of the air-management metrics that the improvement of the thermal environment has the effect of improving heat-insulation performance. It is also clear that heat-insulation performance affects all three airflow paths, namely, the negative-pressure flow path, bypass flow path, and recirculation flow path.
The extent of the effect of heat transmission on the evaluation index based on air-volume ratio is considered hereafter. As explained in the previous section, in the proposed CFD model, heat is transferred by air flow and heat transmission. Therefore, by checking the difference between each index (i.e., NP, BP, and R) before and after the thermal-environment improvement under the same CRAC fan frequency, it can be confirmed how much the low thermal resistance of the structural components deteriorates the indexes. CRAC fan frequency of 45 Hz (G1) is described as an example hereafter. Airflow equivalent to 10% of the CRAC airflow (mc) is induced into the raised floor from the room interior as negative-pressure airflow (mn), and 2% of that value is due to heat transmission. Similarly, 11% of the raised-floor air volume (mf) returns to the CRAC without cooling the ITE as bypass air volume (mb), and 4% of that value is due to heat transmission. Recirculation-air volume (mr) (namely, volume of outlet of the ITE that returns to the ITE inlet port) accounts for 17% of the ITE air volume (ms), of which 4% is due to heat transmission. Since it is assumed that 10-mm-thick glass wool will be installed as the thermal-environment improvement measure, it is considered that the effect on the heat environment will improve if the data center is completely insulated. Originally, the CAC and blank panels are installed in an indoor environment that requires high airflowsegregation efficiency; however, as the high-density installation of ITE progresses, even higher levels of cold-and hot-air segregation are required. Improving ventilation efficiency by reducing heat transmission is one possible solution to this problem.  Air volume ratio The possibility of energy reduction by improving heat-insulation performance is considered as follows. To make it easier to see changes in each index due to varied CRAC fan frequency, Figures 8-10 are rearranged as Figure 11. BAL on the horizontal axis of Figure 11 is the ratio of CRAC air volume (mc) to ITE air volume (ms), as given by Equation (5), and BAL increases as CRAC air volume increases. As BAL increases, NP decreases, BP increases, and R decreases. It is considered that NP decreases and BP increases because the larger the BAL, the more air is supplied to the raised floor. Similarly, it is considered that R decreases as a sufficient amount of air is supplied to the cold-aisle containment (as BAL increases). In both cases, each index is lower after the improvement than that before the improvement within the range of the CRAC-airflow conditions used in this study.
Hereafter, the index to be improved is taken R, the target value of R is set to maximum CRAC supply-air volume, and the energy-saving effect due to the improvement of heat-insulation performance is calculated. The calculation results are listed in Table 9. The relationship between BAL and R is approximated with a linear function, and when BAL (which is equivalent to R = 0.12 (BAL = 1.00) before improvement) after the improvement is calculated, BAL was determined as 0.90. Under the assumption that ITE air volume (ms) is constant, CRAC air volume (mc) can be reduced to 90%. This effect translates into the effect of reducing power consumption. Theoretically, power consumption of the fan is proportional to the cube of the ratio of pre-change air volume and postchange air volume. Therefore, if fan air volume is reduced by 10%, power consumption of the fan is reduced by 26%. In other words, it can be said that by insulating the structural components with 10mm-thick glass wool, it is possible to reduce power consumption of the CRAC fan by 26% and obtain  The extent of the effect of heat transmission on the evaluation index based on air-volume ratio is considered hereafter. As explained in the previous section, in the proposed CFD model, heat is transferred by air flow and heat transmission. Therefore, by checking the difference between each index (i.e., NP, BP, and R) before and after the thermal-environment improvement under the same CRAC fan frequency, it can be confirmed how much the low thermal resistance of the structural components deteriorates the indexes. CRAC fan frequency of 45 Hz (G 1 ) is described as an example hereafter. Airflow equivalent to 10% of the CRAC airflow (m c ) is induced into the raised floor from the room interior as negative-pressure airflow (m n ), and 2% of that value is due to heat transmission. Similarly, 11% of the raised-floor air volume (m f ) returns to the CRAC without cooling the ITE as bypass air volume (m b ), and 4% of that value is due to heat transmission. Recirculation-air volume (m r ) (namely, volume of outlet of the ITE that returns to the ITE inlet port) accounts for 17% of the ITE air volume (m s ), of which 4% is due to heat transmission. Since it is assumed that 10-mm-thick glass wool will be installed as the thermal-environment improvement measure, it is considered that the effect on the heat environment will improve if the data center is completely insulated. Originally, the CAC and blank panels are installed in an indoor environment that requires high airflow-segregation efficiency; however, as the high-density installation of ITE progresses, even higher levels of cold-and hot-air segregation are required. Improving ventilation efficiency by reducing heat transmission is one possible solution to this problem.
The possibility of energy reduction by improving heat-insulation performance is considered as follows. To make it easier to see changes in each index due to varied CRAC fan frequency, Figures 8-10 are rearranged as Figure 11. BAL on the horizontal axis of Figure 11 is the ratio of CRAC air volume (m c ) to ITE air volume (m s ), as given by Equation (5), and BAL increases as CRAC air volume increases. As BAL increases, NP decreases, BP increases, and R decreases. It is considered that NP decreases and BP increases because the larger the BAL, the more air is supplied to the raised floor. Similarly, it is considered that R decreases as a sufficient amount of air is supplied to the cold-aisle containment (as BAL increases). In both cases, each index is lower after the improvement than that before the improvement within the range of the CRAC-airflow conditions used in this study.
Hereafter, the index to be improved is taken R, the target value of R is set to maximum CRAC supply-air volume, and the energy-saving effect due to the improvement of heat-insulation performance is calculated. The calculation results are listed in Table 9. The relationship between BAL and R is approximated with a linear function, and when BAL (which is equivalent to R = 0.12 (BAL = 1.00) before improvement) after the improvement is calculated, BAL was determined as 0.90. Under the assumption that ITE air volume (m s ) is constant, CRAC air volume (m c ) can be reduced to 90%. This effect translates into the effect of reducing power consumption. Theoretically, power consumption of the fan is proportional to the cube of the ratio of pre-change air volume and post-change air volume. Therefore, if fan air volume is reduced by 10%, power consumption of the fan is reduced by 26%. In other words, it can be said that by insulating the structural components with 10-mm-thick glass wool, it is possible to reduce power consumption of the CRAC fan by 26% and obtain a thermal environment equivalent to that corresponding to CRAC fan frequency of 63 Hz (G 3 ). In practice, it is desirable to: (i) calculate, on a trial basis, the reduction of running cost when CRAC fan frequency is reduced and the initial cost of installing the heat-insulation material and (ii) judge whether to adopt these improvement measures on the basis of the life-cycle cost of the data-center room.
Energies 2020, 13, x FOR PEER REVIEW 17 of 21 a thermal environment equivalent to that corresponding to CRAC fan frequency of 63 Hz (G3). In practice, it is desirable to: (i) calculate, on a trial basis, the reduction of running cost when CRAC fan frequency is reduced and the initial cost of installing the heat-insulation material and (ii) judge whether to adopt these improvement measures on the basis of the life-cycle cost of the data-center room.

Concluding Remarks
In this study, the aim was to improve the prediction accuracy of ITE inlet temperature in CFD simulation of a verification room simulating a raised-floor supply/transverse-return data center utilizing cold-aisle containment and blank panels. In the verification room, obstructions, gaps between components, and heat transmission of components are considered as factors that may affect the results of CFD analysis. In addition, the factors were tuned so that prediction accuracy could be maintained with the same model even if the CRAC supply-air volume was varied. To ensure the parameter tuning is efficient, the "design of experiments" (DOE) method was used. To tune the parameter-tuning factor and its level under the three conditions of CRAC supply-air volume, it is necessary to perform 2916 simulations by brute force; however, the proposed method can derive the optimum level of the tuning factor in 63 simulations. Comparing the average ITE inlet temperature of 10 racks with the experimental measurements and CFD simulation results after parameter tuning showed that the former was within ±0.2 • C under all CRAC airflow conditions. Although the factors and levels adopted depend on the target room, the proposed parameter-tuning method is considered to be versatile in terms of improving the accuracy of CFD simulations.
Moreover, the fact that the CFD model after tuning incorporates heat flow through the structural components was focused on, and improving the thermal environment was investigated. As a result of improving the insulation performance of the raised-floor fixed panels, rack cases, aisle containment, and blank panels by insulating with 10-mm-thick glass wool, average ITE inlet temperature was decreased by about 1 • C. This result was rearranged by air-management metrics, and a method of evaluating the influence of CRAC supply-air volume and thermal-insulation performance on the thermal environment was demonstrated. It was shown that improving the heat-insulation performance of the structural members makes it possible to reduce power consumption of the CRAC fan by 26%, while keeping the recirculation ratio constant. For the target thermal environment in question, it is desirable to calculate reduction of running cost by narrowing the CRAC supply-air volume and initial cost increase due to installing advanced heat insulation on a trial basis and compare them in regard to life-cycle cost.
High-density mounting of ITE is progressing, and air-segregation methods such as aisle containment and blank panels have been implemented; however, in the future, higher ventilation efficiency (cold/hot air segregation) will be required. Reduction of heat transmission by improving thermal-insulation performance is an approach for cold/hot air segregation that differs from airflow segregation because it suppresses heat transfer in the partitioned space. Improving thermal-insulation performance of structural components is likely to be one solution to the problem of high heat generation in data centers.