Thermal Management and Energy Consumption in Air, Liquid, and Free Cooling Systems for Data Centers: A Review

Abstract

The thermal management and reduction of energy consumption in cooling systems have become major trends with the continued growth of high heat dissipation data centers and the challenging energy situation. However, the existing studies have been limited to studying the influences of individual factors on energy saving and thermal management and have not been systematically summarized. Thus, this paper reviews the key factors in achieving thermal management and reducing energy consumption in each cooling system, the corresponding research, and optimization methods. To achieve these goals, in this paper, literature surveys on data center cooling systems are investigated. For data center air cooling, thermal management is mainly related to the uniform distribution of hot and cold air. Adjusting the porosity of perforated tiles can reduce energy consumption. For liquid cooling and free cooling systems, climate conditions, cooling system structural design, coolant type, and flow rate are key factors in achieving thermal management and reducing energy consumption. This paper provides the power usage effectiveness (PUE) values of the cooling systems in some cases. A summary of the key factors can provide directions for research on thermal management and energy reduction, and a summary of previous research can provide a basis for future optimization.

1. Introduction

In the last few decades, IT industries such as mobile telecoms, big data, and the Internet of Things have undergone rapid development and are needed in various fields [1]. As a result, the consumption of energy in data centers has experienced continuous and substantial growth due to data centers acting as the infrastructure for IT services. Despite the global financial crisis in 2008, the energy consumption of data centers increased by 56% between 2005 and 2010. Among them, the energy consumption of data centers in the United States has increased by nearly 20 times. The increase in energy consumption is driving the global energy crisis. It is important to note that the energy consumption of cooling in data center systems accounts for 30% to 50% of the total energy consumption [2]. The energy consumption share of each device is shown in Figure 1. For this circumstance, reducing cooling system energy consumption is more critical. Moreover, the safe operation of a data center depends on whether data center thermal management can be achieved, i.e., whether it can dissipate heat efficiently and maintain a uniform temperature distribution under safe conditions, which means that the cooling system needs to ensure efficient heat transfer capacity.

There are three major forms of data center cooling systems, namely, air cooling systems, liquid cooling systems, and free cooling systems. For air cooling systems, achieving thermal management is to obtain good airflow organization. In order to achieve good airflow organization, the size and location of barriers under the ventilation floor were adjusted by Sorell et al. [4] to increase the effective usable area, resulting in achieving a uniform distribution of airflow and enhancing the cooling efficiency. Fakhim et al. [5] closed cold aisles to restrict cold air in the server inlet, reducing hot and cold air mixing, which could reduce the maximum server temperature from 28 °C to 21 °C. Studies by Schmidt et al. [6,7,8] found that optimizing perforated tile structures and reducing tile opening areas helped to allow uniform airflow through the server. In terms of liquid cooling systems and free cooling systems, Ding et al. [9] compared the cooling capacity of annular thermosyphon under R22, R134a, and R410A by using an enthalpy difference laboratory. The results revealed that the heat transfer capacity of R410A was 8.7% and 4.8% higher than that of R134a and R22, respectively. In order to reduce the possibility of chilled water leakage, an in-line cooling component was designed by Cho et al. [10], which included water–refrigerant heat transfer and refrigerant–air-in-room heat transfer, ensuring safety, improving cooling efficiency, and reducing the PUE of the improved cooling system from 1.563 to 1.361 (a reduction of 43.5%). The existing research has just paid attention to the impacts of individual elements on achieving thermal management and low energy consumption, and, even in reviews, there are few summaries of the key factors affecting the cooling capacity and energy consumption of all cooling systems. The review by Zhang et al. [2] summarized the principles of various cooling systems as well as the design and innovation of cooling systems for reducing energy consumption in existing research. Methods for liquid cooling systems to improve cooling efficiency and for free cooling systems to improve energy efficiency were analyzed by Habibi et al. [11].

The purpose of this paper is to provide key factors, related literature studies, and optimization methods for cooling systems. In this paper, firstly, the key factors regarding how to keep the uniformity of temperature distribution in data centers and raise the heat transfer coefficient and energy efficiency under the conditions of using air cooling systems are offered. Moreover, the correlational and concrete studies of these factors are summarized in Section 2. Then, in Section 3 and Section 4, for free and liquid cooling systems in data centers, the primary elements affecting thermal management performance and energy consumption are discussed by collecting previous studies, and the specific research on key factors and optimization methods for cooling systems in the previous literature are discussed.

2. Air Cooling for Data Centers

This kind of technique can be achieved via the way that the chilled air is sent from the underfloor air supply system into the cold aisle. Then, the cold air, after passing through the servers and absorbing the heat dissipated from the servers, is, in turn, discharged into the distributed hot aisle as well as the computer room air conditioning (CRAC). The heated air is finally cooled by the water in the CRAC unit. The cycle is repeated again. For guaranteeing the speed of heat transfer, new data centers do not share CRAC with the building they are located in [12]. Separating hot and cold channels instead of sharing a single channel is conducted to avoid mixing hot and cold air and ensure the cooling efficiency of the air cooling system. The air cooling system in a data center is divided into a room-level cooling system and a row-level cooling system. Figure 2 illustrates the structural designs of both cooling systems. Unlike room-based cooling systems, the air conditioners in row-based cooling systems are usually installed in a row between rack servers or above the cold aisles, with the air from under the floor being fed directly into the cold aisles between the racks rather than being mixed with the original air throughout the data center [13]. Hence, row-based cooling systems, reducing the possibility of mixing cold air with hot air, are widely used in data centers due to the shorter airflow paths and the more sufficient cooling of the cooling target than room-based cooling systems. The combination of hot and cold aisles includes hot-cool-hot and cold-hot-cool aisle layouts. The cold-hot-cool aisle layout can improve the heat balance by at least 9% in row-based cooling systems, which makes it applicable.

2.1. Thermal Management Performance

2.1.1. Uniformity of Temperature Distribution

The uniformity of temperature distribution within a data center and the number of localized hot spots are critical factors for determining the success of thermal management. Hot and cold air mixing and localized hot spots are associated with a non-uniform plenum airflow distribution, which is closely linked to the pressure field of the plenum airflow. There are several dominant elements that determine the pressure field, such as the height of the raised floor, the degree of openness of the perforated tiles, and the deployment of obstacles in the plenum ventilation system [15]. A higher floor height results in a uniform server temperature distribution and efficient cooling, which represents the reduced likelihood of server overheating. The recommended floor height is 1.0–1.2 m, 0.6–0.8 m, and 0.4–0.6 m in open air cooling, a closed cold aisle structure, and a closed hot aisle structure, respectively, in [16]. A design of variable perforated tiles is proposed by Wang et al. [17]. Increasing the opening of the perforated tiles at the CRAC outlet and decreasing the opening before entering the cold aisle can availably alleviate the non-uniform distribution of plenum airflow without increasing pump power.

2.1.2. Cooling Efficiency

Reducing server temperatures to keep them at a safe operating temperature is the primary goal to be achieved by a cooling system. The factors influencing air cooling efficiency are analyzed by Oró et al. [18] based on fluid dynamics software. The results show that hot air mixing, a cold air bypass, and negative pressure at the air inlets of the split floor in a data room are the main factors affecting the efficiency. Hot and cold air mixing can be solved by closing the cold aisle, optimizing the air inlet arrangement, blocking the gap between racks, adding floor dividers, and adding ceiling vents [19]. Schmidt et al. [6] observed that the hot and cold air mixability is minimum and the optimal air cooling is obtained when the distance between the air inlet and the ceiling is 1 m by changing the position of the air inlet based on numerical simulations. The supply/return heat index (SHI/RHI), rack cooling index (RCI), and return temperature index (RTI) are commonly used as measures in today’s research, with SHI and RHI defined as follows:

$$\mathrm{SHI}=\left(\frac{\mathsf{\delta }\mathrm{Q}}{\mathrm{Q}+\mathsf{\delta }\mathrm{Q}}\right)=\frac{\mathrm{Enthalpy} \mathrm{rise} \mathrm{due} \mathrm{to} \mathrm{iniltration} \mathrm{in} \mathrm{cold} \mathrm{aisle}}{\mathrm{Total} \mathrm{enthalpy} \mathrm{rise} \mathrm{at} \mathrm{the} \mathrm{rack} \mathrm{exhaust} }$$

(1)

$$\mathrm{RHI}=\left(\frac{\mathrm{Q}}{\mathrm{Q}+\mathsf{\delta }\mathrm{Q}}\right)=\frac{\mathrm{Total} \mathrm{heat} \mathrm{extraction} \mathrm{by} \mathrm{the} \mathrm{CARA} \mathrm{units}}{\mathrm{Total} \mathrm{enthalpy} \mathrm{rise} \mathrm{at} \mathrm{the} \mathrm{rack} \mathrm{exhaust} }$$

(2)

$$\mathrm{Q}={\displaystyle \sum _{\mathrm{j}}{\displaystyle \sum _{\mathrm{i}}{{\displaystyle {{\displaystyle \mathrm{m}}}^{\mathrm{r}}}}_{\mathrm{i},\mathrm{j}}{{\displaystyle \mathrm{C}}}_{\mathrm{P}}}}\left[{{\displaystyle \left({{\displaystyle {{\displaystyle \mathrm{T}}}^{\mathrm{r}}}}_{\mathrm{out}}\right)}}_{\mathrm{i},\mathrm{j}}-{{\displaystyle \left({{\displaystyle {{\displaystyle \mathrm{T}}}^{\mathrm{r}}}}_{\mathrm{in}}\right)}}_{\mathrm{i},\mathrm{j}}\right]$$

(3)

where Q is the total amount of heat dissipated by all racks in the data center. δ Q is the enthalpy rise of the cold air before it enters the rack.

The supply heat index parameter SHI is the ratio of the heat gained by the cold aisle air before it enters the rack to the total heat gained by the air leaving the rack, while the rack heat index parameter RHI is the ratio of the total heat dissipated by all racks to the total heat gained by the air leaving the rack. SHI and RHI indicate the degree of mixing of hot and cold air.

The relationship between SHI and RHI can be expressed by (4). A better separation of hot and cold air is achieved when SHI tends toward 0 and RHI tends toward 1 [20].

$$\mathrm{SHI}+\mathrm{RHI}=1$$

(4)

The RCI is used to measure the degree of overheating and overcooling of a rack, including RCI_HI and RCI_LO, and is defined as follows:

$$\begin{array}{l}{{\displaystyle \mathrm{RCI}}}_{\mathrm{HI}}=\left(1-\frac{\mathrm{Total} \mathrm{over} \mathrm{temperature}}{\mathrm{Max} \mathrm{allowable} \mathrm{over} \mathrm{temperature}}\right)\times 100\%\\ =\left(1-\frac{{\displaystyle \sum {{\displaystyle \left({{\displaystyle \mathrm{T}}}_{\mathrm{i}}-{{\displaystyle \mathrm{T}}}_{\max -\mathrm{rec}}\right)}}_{{{\displaystyle \mathrm{T}}}_{\mathrm{i}}\succ {{\displaystyle \mathrm{T}}}_{\max -\mathrm{rec}}}}}{\mathrm{n}\times \left({{\displaystyle \mathrm{T}}}_{\max -\mathrm{all}}-{{\displaystyle \mathrm{T}}}_{\max -\mathrm{rec}}\right)}\right)\times 100\%\end{array}$$

(5)

$$\begin{array}{l}{{\displaystyle \mathrm{RCI}}}_{\mathrm{LO}}=\left(1-\frac{\mathrm{Total} \mathrm{under} \mathrm{temperature}}{\mathrm{Max} \mathrm{allowable} \mathrm{under} \mathrm{temperature}}\right)\times 100\%\\ =\left(1-\frac{{\displaystyle \sum {{\displaystyle \left({{\displaystyle \mathrm{T}}}_{\min -\mathrm{rec}}-{{\displaystyle \mathrm{T}}}_{\mathrm{i}}\right)}}_{{{\displaystyle \mathrm{T}}}_{\mathrm{i}}\prec {{\displaystyle \mathrm{T}}}_{\min -\mathrm{rec}}}}}{\mathrm{n}\times \left({{\displaystyle \mathrm{T}}}_{\min -\mathrm{rec}}-{{\displaystyle \mathrm{T}}}_{\min -\mathrm{all}}\right)}\right)\times 100\%\end{array}$$

(6)

where T_i is the average temperature of each rack inlet; n is the total number of rack inlets; T_max−rec is the maximum recommended temperature according to some guidelines or standards; T_min−rec is the minimum recommended temperature according to some guidelines or standards; T_max−all is the maximum allowed temperature according to some guidelines or standards; and T_min−all is the minimum allowed temperature according to some guidelines or standards. Theoretically, the RCI_HI should be equal to 1 to avoid overheating the rack.

The RTI is used to characterize the extent of cold air bypass and hot air return calculated by Equation (7). In order to minimize the extent of cold air bypass and hot air return, RTI should approach 1.

$$\mathrm{RTI}=\frac{{{\displaystyle \mathrm{T}}}_{\mathrm{r}}-{{\displaystyle \mathrm{T}}}_{\mathrm{s}}}{\Delta {{\displaystyle \mathrm{T}}}_{\mathrm{e}}}\times 100\%$$

(7)

where T_r is the hot air return temperature, K; T_s is the cold air supply temperature, K; and ΔT_e is the temperature difference before and after the air enters the rack [17].

Nada, S.A. et al. [21] experimentally tested the effects of three cold aisle arrangements on the degree of hot and cold air mixing in a data center.

The authors observed that compared with the free open aisle, the air intake temperature in the fully enclosed aisle dropped by 11%. When the rack power was 250 W, the SHI of the fully enclosed aisle was 0.03, and the RHI was 0.97, while the SHI of the free open aisles was 0.13, and the RHI was 0.84. Therefore, the SHI of the fully enclosed aisle increased and the RHI decreased significantly, which means the degree of mixing of cold and hot air in the fully enclosed structure decreased. Although the semi-enclosed cold aisle reduced the average intake temperature and average surface temperature, the mixing tendency of hot and cold air increased compared with the fully enclosed aisle. Furthermore, Fakhim et al. [5] experimented and numerically simulated to analyze temperature distribution and observed that the number of local hot spots is reduced and the cooling effect is improved by containing a cold aisle, blocking the blank spaces between the racks, making the CRAC unit’s location perpendicular to and aligned with the hot aisles, and adding ceiling vents and ceiling ducts. The combination of 100 cm ducts and cold aisle containment can fulfill the smallest SHI of 0.16 and the maximum temperature of up to 33.1 °C.

2.2. Energy Saving and Consumption Reduction of System

In the face of increasing energy scarcity, changes to the structural designs of cooling systems and the timely recovery of waste heat can help to improve energy efficiency and reduce energy consumption. At present, power usage effectiveness (PUE) is used as a metric for efficiently measuring and comparing data center energy efficiency and is defined as the ratio of the total power drawn by a data center facility to the power used by the IT equipment in that facility:

$$\mathrm{PUE}=\frac{\mathrm{Total} \mathrm{facility} \mathrm{power} \mathrm{consumption}}{\mathrm{IT} \mathrm{equipment} \mathrm{power} \mathrm{consumption} }$$

(8)

The total power drawn by a data center facility includes the energy consumed by the cooling system. Therefore, in order to yield higher data center energy efficiency, there is a need for all the energy consumed by the facility to be used to power the IT equipment and the energy consumption of the cooling system to be minimized. When the consumption of the cooling system decreases, the energy efficiency rises, and the PUE approaches one.

Researchers have achieved cooling system consumption reduction by adjusting structural parameters. Fulpagare et al. [22] built an air cooling system for a small data center to study the cooling effect using perforated tile porosity, air conditioning fan speed in the server room, cooler return water temperature, and thermal load of the partially closed and closed cold aisle containment as variables. The results showed that the lowest PUE of 1.38 was achieved at full thermal load using 50% perforated tiles for local sealing.

3. Free Cooling System

Free cooling systems are among the most effective energy-saving systems, which refers to a system using natural resources such as air and natural water for data center cooling.

3.1. Air-Side Economizers

Air is widely used to cool data centers because of its features, such as a low acquisition cost, a stable nature, such that air does not react at high temperatures, and high mobility, which means air-side economizers are useful for achieving heat dissipation. Air-side economizers can be implemented in both direct and indirect ways according to whether outdoor air is directly introduced into the data center for cooling. Air cooling can generally be achieved for a 30 kW single rack.

3.1.1. Direct Air-Side Economizers

Direct air cooling is the idea of drawing outside air into a data center for cooling equipment when the outside air temperature is lower than the indoor air temperature. The circulating air in the room is cooled by a CRAC unit until the supply of cold air can last for a long period of time [23]. The cooling system principle is plotted in Figure 3.

Applicability of System

Direct air cooling has strict air quality requirements. Humid air, particulate matter in the gas, and contaminants limit the use of direct air cooling [24]. The presence of high humidity and contaminants increases the risk of equipment operation. Increased concentrations of particulate matter can damage the electronic equipment in data centers and reduce their usability; hence, filtration or deposition of particulate matter is often used to reduce the risk of damage to electronic equipment. Nevertheless, this can also lead to increased investment costs [25]. The suitability of air cooling varies from region to region. Coles et al. [26] conducted a test on local outdoor air that was introduced into five data centers in the USA and India for cooling, and it was shown that the local air met the requirements for direct air cooling and data centers could be safely cooled without damaging the equipment. In addition, Hong et al. [27] determined that Kunming, Guangzhou, and Shanghai have an average of over 3000 h per year available for direct air cooling, whereas Beijing and Harbin are not suitable for cooling with direct air cooling due to severe haze and high levels of dust in the air.

Energy Saving and Consumption Reduction of System

Compared with conventional air conditioning cooling units, a direct air-side economizer does not require a refrigeration unit to produce low-temperature refrigerants but directly introduces external low-temperature air, which reduces power consumption and simplifies the system. Facebook’s data center in Oregon took advantage of this cooling system, with a PUE of 1.08. Geng et al. [28] introduced air directly into the Kunming data center and optimized the energy efficiency, resulting in the PUE being reduced from 1.9 to 1.56. The energy-saving effects of different cooling systems in data centers were analyzed based on the reference model set by the Seoul Municipality by Cho et al. [29]. The results showed that a direct cooling cycle can save about 68% of energy, whereas an indirect air cooling cycle can save 50% of energy, and a water-side cooling cycle can save 15%, which demonstrates that direct air cooling has good energy-saving properties.

The local climatic condition is one of the decisive factors in energy saving because direct air cooling is not suitable when the humidity is extremely high or low. The longer the cooling method is used with the shorter cooling time using vapor compression, the better the energy-saving effect. Siriwardana et al. [23] adopted weather data from 20 weather stations in Australia to analyze the potential for data center direct air cooling. The study showed that direct air cooling can only be used for long periods of time in dry and cold areas and not in hot and humid tropical areas such as Harwin and Townsville. Oró et al. [30] analyzed the energy-saving potential of using air cooling in data centers in five climate zones. By measuring the expenditure, a conclusion can be made that the cooler climates of areas where direct air-side economizers can be utilized for a longer period of time will decrease energy consumption by 15–22%; however, overly dry and humid areas have less applicability for this cooling method due to the increased power consumption and cost caused by the need for humidification or dehumidification. The Tokyo Data Center was cooled via a direct air cooling system in combination with evaporative cooling technology and IT equipment waste heat recovery technology [31]. Evaporative cooling technology uses water evaporation to absorb heat dissipated from the air and to increase the air humidity to an appropriate value, while IT equipment waste heat recovery transfers the waste heat generated by the equipment to the air, heats the air, increases the temperature, and reduces the relative humidity to a suitable level. The combined system offers annual energy savings of 20.8% compared with conventional air-conditioning and refrigeration systems. In addition, a drop in indoor temperature reduces the effect of energy saving, and for every 2 °C drop in temperature, energy savings decrease by 2.8–8.5%.

3.1.2. Indirect Air-Side Economizers

Indirect air-side economizers refer to the exchange of outdoor air with indoor hot air through a heat exchanger, bringing cold energy to the indoor air in order to cool the servers. The principle of the most typical rotary wheel indirect air cooling system is shown in Figure 4.

Outdoor air does not enter the data center directly, helping to prevent harmful substances such as dust and smoke from entering the data center [33]. However, the heat transfer resistance between air and air will increase, resulting in the heat transfer area needing to be increased. An indirect evaporative cooling air conditioning unit was built in the Guang Ming District of Shenzhen to cool the data center, with a PUE value of 1.249 and a single cabinet power of 5.3 KW [34]. Efficient heat transfer between indoor and outdoor air is the key link to determining the energy-saving effect of a system. Ham et al. [32] compared the cooling effects of nine kinds of air-side heat exchangers in data centers. Compared with the traditional cooling system in data centers, the total cooling energy of the energy-saving device can be commonly saved by 47.5% to 62%. Among the nine types of cooling systems, indirect air-side economizers using an efficient heat exchanger can save 63.6% of the energy, while indirect air energy-saving devices using an inefficient heat exchanger can save the least energy. There are many kinds of heat exchangers. In previous studies, it was found that plate heat exchangers are more suitable for indirect air cooling. In the communication base station in Beijing, when the outdoor temperature is low enough, a plate heat exchanger is used to discharge the excess heat of the communication equipment. Compared with traditional air conditioning refrigeration, this cooling method can save 29% of the electricity consumption [35].

3.2. Water-Side Economizers

3.2.1. Direct Water-Cooled System

To provide sufficient cooling capacity for a data center, a direct water-cooled system uses natural cold water such as seawater and lake water to achieve direct cooling through a heat transfer piping facility. The system principle of the seawater cooling module is proposed as shown in Figure 5, whereby the cooling water absorbs the heat from the module and transfers it to the seawater after entering the heat exchanger. The kinetic energy of the seawater flow is captured and used to drive the cooling pump. Nonetheless, the system takes into account the effects of waves and weather, so it is not suitable for use in developing countries.

Thermal Management Performance

Each server in a data center operates differently and requires different amounts of cooling. Therefore, Chen et al. [36] used seawater as a cooling source to exchange heat with chilled water and return water through plate heat exchangers on the seabed to achieve cooling of the data center. Then, the fixed water supply pump was changed to a mobile water supply pump to provide the corresponding cooling water flow to the server with different heat dissipation, and a mixed integer linear model was also established to obtain the optimal cooling water flow so as to improve the heat dissipation performance and reduce the occurrence of local hot spots. The improved system could make the surface temperature distribution of the data compartment more uniform.

Energy Saving and Consumption Reduction of System

The Dong Jiang Lake Data Center in Zi Xing City, Hunan Province, utilizing lake water for cooling, does not require mechanical refrigeration for 90% of the year, whereby the annual average PUE value is 1.1 [37]. Reducing the energy consumption of a cooling system is the main purpose of cooling with natural water sources. An energy optimization method based on a genetic algorithm was developed by Ling et al. [38] to optimize the temperature difference between the cooling water and the temperature of the supplied cooling water, which reduced the final energy consumption by up to 33.1%. However, due to the limitation of geographical factors and the requirements of environmental protection, this kind of cooling method is not universally applicable [39].

3.2.2. Air-Cooled System

An air-cooled system utilizes outdoor air in the dry cooler to absorb the heat of the circulating water in the computer room air handler (CRAH) device when the outdoor wet bulb temperature is low enough, resulting in the circulating water being cooled and re-entering the CRAHdevice to provide cooling capacity for the data center, which reduces or even eliminates the use of mechanical cooling and achieves the purpose of energy saving. The schematic of this system is depicted in Figure 6. In contrast, another conventional air-cooled system integrates a dry cooler with an air-cooled chiller, as shown in Figure 6. The air-cooled chiller is based on the principle of duplicating coolers to achieve redundancy, thereby increasing the heat exchange area and improving the cooling capacity, and the integration of the two coolers can reduce the footprint of the system.

3.2.3. Cooling Tower System

After cooling via a cooling tower, the cooling water enters a plate heat exchanger or chiller to cool the cold water flowing through the CRAHunit, and then the cold water is cooled down and re-enters the CRAHunit to cool the air in the data center to realize circulating refrigeration. The schematic of this system is depicted in Figure 7.

Depending on the outdoor wet bulb temperature, the system switches to full free cooling, partial free cooling, or mechanical cooling. The plate heat exchanger is turned on and the chiller is turned off in full free cooling mode, whereas the plate heat exchanger is turned off and the chiller is turned on in mechanical cooling mode. The system is divided into a series system and a parallel system according to whether the cold and cooling water pass through the plate exchanger and the chiller at the same time under partial free cooling conditions. At that time, the cold and cooling water in the tandem system pass sequentially through the plate heat exchanger and chiller, while the cold and cooling water in the parallel system pass only through the chiller or plate heat exchanger [40]. The system schematic and valve control are articulated in Figure 8 and Figure 9 and

Table 1. Equipment and valve operating status in series system [40].

	Cooling Tower	Chiller	Plate Heat Exchanger	V1	V2	V3	V4
Mechanical cooling	On	On	Off	Off	On	On	Off
Partial free cooling	On	On	On	Off	On	Off	On
Full free cooling	On	Off	On	On	Off	Off	On
	Cooling Tower	Chiller	Plate Heat Exchanger	V5	V6	V7	V8
Mechanical cooling	On	On	Off	On	Off	On	Off
Partial free cooling	On	On	On	On	Off	Off	On
Full free cooling	On	Off	On	Off	On	Off	On

and

Table 2. Equipment and valve operating status in parallel system [40].

	Cooling Tower	Chiller	Plate Heat Exchanger	V1	V2	V3	V4
Mechanical cooling	On	On	Off	On	Off	On	Off
Partial free cooling	On/off	On/off	On/off	On/off	On/off	On/off	On/off
Full free cooling	On	Off	On	Off	On	Off	On

.

Ling et al. [41] developed an annual energy consumption model for a cooling tower system using lake water as the cooling source and evaluated the energy consumption of this cooling unit under different loads. Aiming to ensure the accuracy of the measurements, this model took into account the effects of the dynamic heat dissipation characteristics of servers, lake water temperature, outdoor weather conditions, and cooling plant thermal performance. The results showed that the average energy efficiency ratio of the cooling equipment ranged from 16.8 to 49.3, and the average PUE value ranged from 1.11 to 1.15 under 4 types of load factors, indicating that the cooling tower system was effective for saving energy.

Improvement of Cooling Capacity

The spray density and packing distribution of a cooling tower can be optimized to enhance the heat exchange between air and cooling water for providing lower-temperature cooling water to the plate heat exchangers and chillers. Huang et al. [42] used the NSWCT97 calculation program to optimize spray density and packing distribution. The cooling effect (temperature difference between before and after water cooling) temperature difference between before and after cooling of water cooling was increased by 0.7 °C compared with a uniform filler and uniform drench equipment and by approximately 0.3 °C compared with drench-only equipment.

Energy Saving and Consumption Reduction of System

Energy saving in cooling towers depends on extending the time of complete full free cooling and reducing the use of compressors in chillers. Li et al. [43] developed a model-based optimization method based on a cooling tower system to obtain the optimal free cooling switching point for different cooling load ratios to operate the system in full natural cooling mode as far as possible, and to obtain the optimal cooling tower approach temperature for the ambient wet bulb temperature. The results showed that via optimization, more than 100,000 kWh of energy can be saved practically throughout the year in different climate zones.

In addition to the cooling water, the chilled water flow rate and inlet temperature of CRAC units also have optimal values in terms of energy saving. Taking the Tianjin data center as an example, the optimal values for the above parameters could be obtained through building a heat dissipation model, a hybrid thermal management model, and an energy consumption model, which could achieve 21.3% energy savings and a 4.43% PUE reduction [44]. In addition to this, Alt et al. [45] combined a cooling tower system with a solar absorption cooling system, whereby solar collectors absorbed heat to power absorption chillers. Cooling water for the internal cooling cycle was supplied by the absorption chillers, and the cooling tower system restarted to supply cooling water at night when there was no solar supply. This system has been widely studied due to its good energy saving properties.

4. Liquid Cooling Systems

The cooling capacity of air in data centers is only about 37 W/cm², representing air as having a low heat dissipation rate [11]. With the advent of high-performance processors, air cooling systems cannot meet the demand for processor cooling, so liquid cooling systems are gradually being implemented in data center cooling by virtue of the fact that liquid has a higher heat transfer coefficient.

4.1. Indirect Liquid Cooling

Indirect liquid cooling is the exchange of heat between a heat source and the coolant without contact. The cold water cooled by the external chiller is supplied by the coolant distribution unit (CDU) to the internal closed-loop cooling installed in the electronic device for cooling down electronic equipment, as shown in Figure 10. Traditional indirect cooling methods include cold plates and water blocks. The main chips of the server are cooled by cold plates, while other devices are still cooled by air cooling, enhancing heat dissipation. Although cold plates can only cool single racks of less than 45 k W, they are energy efficient and quiet. In terms of cost, this method does not require expensive water-cooling units and thus has some advantages over pure liquid cooling [46]. At present, the main focus of the enhancement of heat transfer characteristics of cold plates is on the properties of the porous medium of heat exchangers and the design of microchannel heat sinks. The PUE of a data center was decreased by 1.17–1.30 with enhancing heat dissipation in [47].

4.1.1. Single-Phase Cooling

Single-phase liquid cooling means that the coolant does not undergo a phase change as it absorbs heat. Water can be used as a common coolant. Studies of dissipation enhancement on single-phase water-cooling methods have focused on the porous media properties of cold plates and the design of microchannel radiators. Singh et al. [48] designed a sintered porous heat sink for compact microprocessors that consists of rectangular channels made of 40% porosity and 1.44 × 1011 m² sintered porous copper. The cooling medium is water. The heat sink can achieve a heat removal of 2.9 W/cm², and the effective thermal conductivity can be up to 32 W/(m-K). An enhanced microchannel heat sink with sectional oblique fins was also designed for thermal performance studies, which showed that the new heat sink had an enhanced heat transfer capability compared with conventional microchannel heat sinks. In particular, the enhanced system had a better heat transfer ability with smaller oblique angles and a smaller oblique fin pitch, and the best heat transfer performance was achieved at a fin pitch of 100 μm and an angle of 27° [49].

4.1.2. Two-Phase Cooling

Two-phase fluid cooling refers to the refrigeration of a cooling fluid by means of latent heat of phase change. Dielectric fluids and refrigerants with low boiling points can be used as cooling fluids [50]. In the current research, improving the structure of microchannel heat exchangers to achieve improved heat transfer performance has become the main development direction for two-phase boiling heat transfer. In addition, the presence of porous media is beneficial for strengthening the heat transfer on the boiling surface and for improving the thermal management performance of electronic devices. Over the years, many studies have been carried out on the application of porous materials in two-phase flow cooling systems [51]. Agostini et al. [52] developed a pump-driven two-phase cooling system containing a multi-microchannel silicon heat sink with R236fa as the cooling medium and studied the cooling system’s thermal characteristics. As shown in the study, the chip could be kept below 52 °C after cooling without overcooling, and the heat flux of the heat sink ranged from 24.2 to 222 W/cm². Additionally, Zhao et al. [53] utilized a horizontal foam metal-filled tube with R134a as the cooling medium to study the heat transfer characteristic. As a result, by reducing the metal foam cell size, the heat transfer coefficient increased substantially, which not only dealt with more surface areas and boiling sites but also related to strong flow mixing. Moreover, the heat transfer coefficient of the copper foam tube was three times that of the ordinary copper tube.

4.1.3. Heat Pipe Cooling

Heat pipe cooling is a passive two-phase cooling method in which the coolant evaporates at the evaporating end of the heat pipe, flows to the condensing end via differential pressure, condenses, and exothermically generates liquid coolant again at the condensing end, and, eventually, the coolant returns to the evaporating end of the heat pipe via gravity (thermosiphon) or capillary action (wicked heat pipe) [54]. Common coolants include water, methanol, acetone, ammonia, R141b, NF, or SiO₂-H₂O, while water and ethanol are generally used in condensers to cool the condensing end of the heat pipe [55]. Heat pipe cooling offers efficient heat transfer with thermal conductivity several orders of magnitude higher than solid materials and reduces the possibility of leakage within the data centers at the same time. Apart from these advantages, heat pipes are highly efficient passive devices with simple constructions and have a high reliability and implementation ability for long-distance cooling, with no need to provide additional pumps, and moving parts [56,57,58,59]. In terms of cooling capacity, heat pipes are not significantly improved compared with cold plates [60].

Thermal Management Performance

Ding et al. [61] summarized the important influencing factors for loop thermosyphon (the filling ratio (FR) and coolant type). Scholars observed that a suitable FR should keep a system from overheating or subcooling, and CO₂, whose heat transfer ability is 22–25% higher than that of R22, allows efficient heat transfer. Ahmadou et al. [62] designed a heat pipe with an inclined condenser for cooling the evaporated coolant to study the cooling performance of this heat pipe, using pentane as the coolant. The results showed that a heat pipe cooling system can carry twice the heat load of a conventional cooling system, enabling it to cool data centers with higher heat flow densities. The principle is illustrated in Figure 11.

Energy Saving and Consumption Reduction of System

The energy-saving capability of a heat pipe is mainly related to the heat pipe structure as well as the coolant. A looped thermosiphon cooling system was constructed to charge CO₂ and R410A, respectively, by Ding et al. [63] to study the operating characteristics of the system. The results showed that the PUE of CO₂ was reduced by 0.013 compared with R410A throughout the year. A looped thermosiphon with a coupled compressor was proposed by Han et al. [64,65]. This heat pipe has two modes of operation, VC and TS. In this study, a new mode-switch valve and the design methods of the connection pipe and evaporator were used for the heat pipe, which increased the circulating flow of the two modes and especially the cooling capacity of the TS mode. The optimized system can reduce energy consumption by 34.3–36.94%. The system principle is shown in Figure 12.

4.2. Direct Liquid Cooling

In direct liquid cooling, the coolant is in direct contact with the electronics for heat transfer, so it is necessary to ensure that the coolant is a dielectric fluid. Compared with indirect cooling systems, direct cooling has two advantages: (1) the coolant has electrical insulation, which can ensure the safety of electronic equipment; (2) indirect cooling needs to determine a specific liquid pipelines layout size according to the cooling load, but direct cooling overcomes these shortcomings and can be flexibly modified according to different cooling loads. Indirect liquid cooling mainly includes the following three forms.

4.2.1. Liquid Immersion Cooling

Liquid immersion cooling is characterized by high heat dissipation efficiency, uniform temperature, low noise, no pollution, high reliability, energy saving, high system safety, and a significant reduction in floor space compared with air cooling. Liquid immersion cooling is split into single-phase liquid immersion cooling and two-phase liquid immersion cooling. In single-phase liquid immersion cooling, the server is immersed in a closed container with a coolant that absorbs heat from the surface of the server and then flows into the heat exchanger where the heat is discharged. Finally, the server achieves cooling in the closed container. The temperatures of liquid coolant in front of CPU and under the outlet location are commonly measured to analyze the cooling capacity of the coolant under different power of CPU. The measuring position are shown in Figure 13a,b. Unlike single-phase liquid immersion cooling, the coolant of two-phase liquid immersion cooling absorbs heat from the surface of the server and then evaporates. Due to the existence of latent heat of phase change, two-phase liquid immersion cooling has a better heat dissipation capacity [66]. The schematic diagrams of single-phase and two-phase liquid immersion cooling are presented in Figure 13 and Figure 14.

Single-Phase Liquid Immersion Cooling

Single-phase immersion cooling can be applied to a single server rack with up to 100 k W. Commonly used coolants are mineral oil, synthetic oil, e-fluorinated fluids with higher boiling points, etc. According to Google, oil cooling technology can decrease the PUE by up to 1.02~1.03. Influencing factors of single-phase immersion cooling include coolant flow rate [69,70], coolant type, coolant inlet and outlet position, circulation temperature, heat sink structure [71,72], etc. The effects of these factors on the cooling effect are shown in

Table 3. Influences of factors on the refrigeration effect of single-phase immersion cooling.

Factor	Literature	Influence
Flow rate	Gandhi et al., 2019 [73]	The temperature of all components immersed in the EC-100 appeared to decrease as the volume flow rate increased and the inlet temperature decreased, while the server temperature remained essentially constant as the volume flow rate increased above 2 L/min.
	Li et al., 2016 [74]	Although the flow rate was reduced to 1.5 lpm at high temperatures, the server immersed in mineral oil for cooling could remain under the safety temperature (74 °C) with a PUE of 1.03. Therefore, there was no need to increase the flow rate excessively to achieve server cooling.
Coolant type	Sun et al., 2014 [75]	The dimethyl silicon oil had good cooling capacity and could control the cell temperature to 20–31 °C, and there was no significant drop in cell efficiency after 270 days of immersion.
	Li et al., 2021 [76]	The greater the viscosity, multiplication of density and specific heat capacity, and volume expansion coefficient of the coolant, the stronger the convective heat transfer capability. Fluorocarbon compounds have a better cooling effect, followed by aliphatic compounds, and silicone substances are poor.
	Jithin et al., 2022 [77]	The control of deionized water, mineral oil, and engineering fluid on the temperature of lithium-ion batteries was analyzed, and the results showed that with a 3-C discharge rate and a mass flow rate of 0.05 kg/s, the temperature rise of deionized water, mineral oil, and engineering fluid was controlled at 2.2 °C, 8.6 °C, and 7.2 °C, respectively.
Coolant inlet and outlet arrangement	Li et al., 2016 [74]	The inlet and outlet were arranged in a U-shape and a Z-shape. The temperature of the servers further away from the outlet was effectively reduced in the U-shape arrangement, but there was no significant improvement in temperature uniformity in either arrangement.
	Kuncoro et al., 2020 [70]	The effect of the inlet-outlet position on the temperature of CPU was up to 8.582%, which is low in contrast with all factors of the mineral oil immersion cooling system.
Cycle temperature	Chhetri et al., 2022 [66]	The server temperature increased as the cycle temperature decreased.
Heat sink structure	Li et al., 2016 [74]	The main factor affecting the maximum temperature of the substrate was the thickness of the fins, and the main factor affecting the uniformity of the substrate was the thickness of the substrate.

.

Two-Phase Liquid Immersion Cooling

Two-phase liquid immersion cooling is available for a single server rack with up to 110 k W. Commonly used coolants can be selected from refrigerants and dielectric fluids with boiling points below the maximum tolerable temperature of the working parts, such as fluorocarbon fluids (FC-72, FC-87, and PF-5060) and hydroflurane (HFE7000, HFE-7100, and HFE-7300) [50,78]. It is worth noting that the unit cost of these fluorocarbon coolants is 10–20 times higher than that of mineral oils.

Thermal Management Performance

The influencing factors for two-phase liquid immersion cooling are mainly fin structure design, surface roughness, and coolant type. The studies on these parameters are illustrated in

Table 4. Influences of factors on the refrigeration effect of two-phase immersion cooling.

Factor	Literature	Influence
Fin structure design	Yu et al., 2007 [79]	Decreasing the fin spacing or increasing the fin length increases the total heat transfer coefficient. A fin spacing of 0.5 mm and a fin length of 4 mm have the maximum heat transfer coefficient.
	Singh et al., 2015 [80]	The surface heat transfer performance is enhanced by pressing the pattern on the surface of the fin, and the heat transfer rate obtains the maximum value when the spacing and angle of the embossed pattern are 12 mm and 45°, respectively.
	Zhang et al., 2020 [81]	The heat dissipation effect of a W-shaped fin radiator is significantly better than that of a parallel plate fin radiator, and the average heat transfer coefficient shows a trend of increasing first and then decreasing with the increase in fin spacing, inclination angle, and gap.
Surface roughness	Manetti et al., 2020 [82]	Different heat fluxes have different optimal copper foam thicknesses, and copper foam can increase the heat transfer coefficient by 145%.
	Habibi et al., [11]	The heat transfer coefficient can be improved by adding nanoparticle coatings, porous, microporous, and roughening the surface.
Coolant type	Li et al., 2021 [83]	The heat transfer performances of different cooling media (ethanol, FC-72, and R113) were studied under the conditions that media flew compulsively and naturally. The heat transfer resistance of ethanol was the minimum whether in the case of forced convection or natural convection, which were 0.073 °C/W and 0.2 °C/W, respectively.

.

Energy Saving and Consumption Reduction of System

The PUE of two-phase liquid immersion cooling is closely related to server power. The higher the server power, the lower the PUE. A two-phase immersion liquid cooling system was established by Kanbur et al. [84], and the thermodynamic properties of the system were analyzed under 6 different thermal loads within a range of 3.43–9.17 KW. The experimental results showed that the coefficient of performance (COP) was the highest, reaching 6.67, and the PUE was the lowest, reaching 1.15 at the highest operating load. The COP was the lowest, reaching 2.5, and the PUE was the highest, reaching 1.4 at the lowest operating load. In addition, the use of seawater to absorb the heat of an evaporative coolant to extract salt can not only condensate the vapor but also realize the waste heat utilization of data centers and reduce energy consumption [85]. In 2019, Dawning Information Industry Co., Ltd. released the “Silicon Cube” high-performance computer, which used the immersion phase change liquid cooling technology method to approach the energy efficiency level of a single computer to the limit, and the PUE reached 1.04.

4.2.2. Spray Cooling

Spray cooling refers to the coolant being dispersed or atomized into fine droplets before making contact with the heated surface, and the droplets directly absorb surface heat to achieve the purpose of cooling. This cooling system offers a high heat flux removal capability and the ability to maintain safe temperatures of electronic components [86]. Yu et al. designed a closed-loop spray cooling system, as represented in Figure 15.

Nonetheless, the heat transfer mechanism of spray cooling is complex and is still being studied [88]. The spray cooling coolant can be R134a, PF-5060, R22, water, or methanol. Spray cooling has a strong cooling capacity, and can realize the cooling of a single cabinet with a power of 140 KW, but the nozzle is easy to block and difficult to maintain, and the cooling system requires a sealed steam chamber and a steam recovery device [60]. Water and R-134a were chosen as the cooling mediums by Hsieh et al. in [89]. The heat transfer performance test of spray cooling was conducted with spray mass flow and subcooling as variables. As represented in the results, for both water and R-134a, cooling performance increases with mass flow. However, because of the tiny degree of subcooling for R134-a, the effect of subcooling seems not significant. Nevertheless, this still shows that an increase in supercooling leads to the weakening of the heat dissipation effect. The water is more subcooled and therefore has a more pronounced effect on the cooling performance. According to the data, the cooling ability of water is better than that of R134-a.

The PUE of the improved plug-in spray-cooling-coupled waste-heat-driven absorption refrigeration system is still 1.44, as tested by Chen et al. [90]. A spray cooling system composed of a server and heater was developed by Kandasamy et al. [91] to study the effects of nozzle flow, condenser fan power, condenser cooling water flow, and heat load on the thermal performance of the system and spray chamber pressure. It should be noticed that the heater surface temperature could be reduced by 6.5 °C with the nozzle flow increasing and the pressure and temperature in the cavity decreasing, while the fan power, cooling water flow, and heat load had little effect on the thermal performance of the system. Compared with an air cooling system, the total energy consumption of this new spray-cooling system can be reduced by 25.8%.

4.2.3. Jet Impingement

Jet impingement refers to a high velocity coolant from a convergent nozzle being directly injected onto the heated surface for heat absorption and cooling [92], as shown in Figure 16. FC-32 and water are commonly used as coolants, but, recently, hybrid nanofluids increasing heat transfer efficiency have attracted the attention of researchers [93]. Jet impingement is currently rarely used in data center cooling. Nonetheless, the heat transfer coefficients for jet impingement and spray cooling are 16 to 275 W/cm² and 3.97 to 250 W/cm², respectively; thus, jet impingement cooling systems have better heat transfer performance than spray cooling systems [55]. Researchers have focused on researching jet velocity [94,95], nozzle structure [96,97], the structural design of the heating surface [98,99], and the distance [100,101] between the nozzle and the jet surface regarding the enhancement of the heat transfer coefficient. Baghel et al. [102] investigated heat transfer efficiency under three kinds of jet Reynolds numbers and four different ratios of surface-to-jet velocity. The increase in velocity favors fast heat removal. The flow boiling heat transfer mechanism of jet impinging on a smooth surface and a reinforced surface was experimentally investigated by Ndao et al. [103], and the conclusion can be drawn that supercooled boiling was the main heat transfer mechanism. The heat transfer coefficient of the reinforced surface with round microneedle fins (D = 125 μm) exceeded 150,000 W/(m²·K) when the flow rate was up to 2.2 m/s and R134a was used as a coolant, and nucleated boiling of the reinforced surface occurred at a lower wall superheat compared with that of smooth walls refraining from a too high wall temperature in order to dissipate heat in time.

5. Conclusions and Further Recommendations

The main objective of this paper was to summarize the key factors for air, liquid, and free cooling systems to achieve low energy consumption and accomplish data center thermal management, and to describe the corresponding research as well as the optimization methods. The specific analysis is as follows:

(1)

With regard to air cooling data centers, where thermal management is mainly achieved by controlling the uniform distribution of airflow and the non-mixing of hot and cold air, researchers have further investigated the impacts of the height of the raised floor, the degree of openness of the perforated tiles, and the deployment of obstacles in the plenum ventilation system on the organization of airflow and have obtained the optimal values of these parameters through simulations and experiments. The system achieves energy consumption reduction mainly through changes in structural parameters, particularly the porosity of the porous tiles.

(2)

Other cooling systems include free cooling systems and liquid cooling systems. The cooling principles of each system were described in this paper. The corresponding PUE values and the cooling loads that can be achieved were provided at the same time.

(3)

The thermal management and energy saving of free cooling systems are mainly focused on climatic conditions, the structural design of the cooling systems, the free cooling switching point, and flow rates. The previous literature mainly focused on obtaining the optimal value of various factors through algorithms and improving the structural performance to save energy consumption and achieve uniform heat dissipation.

(4)

The thermal management and energy saving of liquid cooling systems are mainly focused on thermal loads, structural parameters, coolant flow rate, and coolant type. Researchers not only analyzed the relationships between these factors and cooling performance as well as energy consumption but also optimized the heat-exchange facility structures.

Although researchers have conducted a lot of research on energy conservation and efficient heat dissipation in cooling systems, there are still some gaps that entail future research. Water-side economizers and liquid cooling systems are home to PUEs approaching 1; therefore, both of them have a good trend of development. Water-side economizers are applied to areas where water resources are rich. However, the energy consumption of this system is mainly affected by the free cooling switching point, which is related to the cold water supply temperature and cooling load. The height of the free cooling switching point decides the time of complete free cooling. Therefore, a control system needs to be established to adjust the switching point according to the situation to extend the time of complete free cooling in order to save energy. A free cooling system, a system that has few requirements for the geographical location of a data center, is suitable for the cooling of high-power data centers. At present, no accurate conclusions have been reached on the heat transfer mechanisms of spray cooling and jet impingement, and there is still little research on energy saving and heat transfer enhancement by improving nozzle structure design in spray cooling and jet impingement. As two cooling systems with high efficiencies in heat transfer, they still need to be studied in more depth in the future.