- Email: [email protected]
Experimental and numerical study of water-cooled datacom equipment
Applied Thermal Engineering 84 (2015) 350e359
Contents lists available at ScienceDirect
Applied Thermal Engineering journal homepage: www.elsevier.com/locate/apthermeng
Research paper
Experimental and numerical study of water-cooled datacom equipment pahne Le Masson a, Patrick Glouannec b Fabien Douchet a, b, *, David Nortershauser a, Ste a b
Orange Labs, 2 Avenue Pierre Marzin, 22300 Lannion, France LIMATB, Rue de Saint Maud e, Universit e de Bretagne Sud, 56321 Lorient Cedex, France
h i g h l i g h t s A study of an electronic rack cooled by air to water heat exchanger is carried out. Experimental study with a substantial instrumentation at different scales is performed. The energy efficiency of the cooling system is highlighted. Numerical model of the system by using nodal approach is defined and validated.
a r t i c l e i n f o
a b s t r a c t
Article history: Received 20 January 2015 Accepted 14 March 2015 Available online 1 April 2015
This paper presents an experimental and numerical study of an electronic rack cooled by a finned tube heat exchanger. The objectives are to evaluate cooling and energy performance of this system and formulate a numerical model based on measurements. Experimentation has been carried out using commercial servers. A substantial instrumentation has been conducted at different scales (servers, rack and exchanger). Several tests have been made with different sets of parameters like water inlet temperature or power dissipated by servers. In each case, the heat exchanger has removed more than 90% of electrical power consumed by the rack (with no chiller use). Furthermore, promising results are obtained with this system, especially the reduction of energy consumption of cooling part compared to traditional air cooling. Finally, the rack and heat exchanger are simplified into two numerical models which can predict temperature outputs as a function of defined inputs (water and air flow rate and temperature, power dissipation). Validation tests have been carried out with different sequences of measure and numerical models have given satisfactory results. They will be duplicated to compute the impact of this cooling system at a data center room scale. © 2015 Elsevier Ltd. All rights reserved.
Keywords: Data center Cooling Instrumentation Heat exchanger Energy efficiency Numerical model
1. Introduction With the expansion of Internet services (cloud, social networking, etc …), thermal management in telecommunication systems and data centers has become a major issue for operators. In 2005, data centers represented 1% of the worldwide electricity consumption [1]. According to Van Heddeghem [2], worldwide data centers electricity consumption has grown from 2007 to 2012 by 4% per year. Typically, the power load per rack starts from 5 kW for low density equipment and can reach 40 kW for high density system like supercomputers [3,4]. * Corresponding author. Orange Labs, 2 Avenue Pierre Marzin, 22300 Lannion, France. Tel.: þ33 296072334. E-mail address: [email protected] (F. Douchet). http://dx.doi.org/10.1016/j.applthermaleng.2015.03.030 1359-4311/© 2015 Elsevier Ltd. All rights reserved.
The Power Usage Effectiveness (PUE) is energy efficiency metric for data centers defined as the ratio of total energy consumption of the data center to energy consumption used by the IT equipment [5]. Typical values of PUE are around 1.6, and can decrease to 1.2 in case of best cooling practices application [6]. These values confirm the fact that cooling architecture represents 25% or more of the total power consumption of data centers [7]. To maintain high reliability in datacenters, it is recommended to maintain datacom equipment into specific climatic range according to European Telecommunications Standards Institute (ETSI) [8]. As power consumption has increased, the power dissipation of telecommunication equipment has dramatically raised too. With high density servers and power supplies, electronic equipment is fitted with high power fans to remove heat. It will become more and more difficult to cool IT equipment by air cooling with Computer
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Nomenclature C Cp h
thermal capacity, J K1 specific heat capacity, J kg1 K1 thermal conductance, W K1
m P Q t T
mass flow rate, kg s1 power, W volumetric flow, m3 s1 time, s temperature, K
·
Greek symbols ε effectiveness F thermal heat flux, W
Room Air Conditioning (CRAC) systems usually used in data centers. In the most of them, racks are arranged in rows with cold and hot aisles configurations [9]. Row containments can be added to eliminate hot spots and air recirculation [10]. Fulpagare et al. [11] performed numerical studies to quantify the effect of plenum obstructions on total CRAC flow rates to increase data center performances. The authors indicated that obstructions in plenum chambers lead to decreased airflow rates by as much as 80%, with an increase in the occurrence of hot spots. Nevertheless, the use of chillers is very expensive in terms of energy consumption. Free cooling [12,13], and container data centers [14] are innovations that have reduced the part of cooling consumption by using direct freshair from outside. Zhang et al. [15] made an experimental study for free cooling of data centers by combining refrigeration and thermosyphon loops independently of each other. They also show that annual energy-saving rates are between 5.4 and 47.3% with an indoor temperature is set on 27 C. Moreover, Ham et al. [16] analyzed the effect of supply air temperature ranges for different configurations of modular data center. Energy consumption reductions appear when air inlet temperature is set on 18e23 C. Higher supply air temperatures are not recommended due to a rise of CRAC energy fans which increase cooling energy consumption. However, the efficiency depends on the location of the data centers because of annual humidity and temperature conditions. Switching to liquid cooling seems to be an alternative option to make energy efficiency cooling systems. Several studies deal with Computational Fluid Dynamics (CFD) simulations of water-cooled rear-door heat exchanger. Udakeri et al. [17] have investigated hybrid cooling solutions which consist in associated air and liquid cooling for 32 kW racks. They concluded that underfloor supply configuration has better performance than overhead supply configuration. In this case, heat exchangers have removed 55% of the total 32 kW rack with a temperature of air chiller entering the room fixed at 13 C. Some additional studies have shown that the use of liquid cooling with rear heat exchanger have drastically reduced the power consumption of CRAC systems [18,19]. Furthermore, Karki et al. [20] studied the impact of using rear-door heat exchangers as the sole source of cooling. Simulations have been made with moderate rack heat loads (3e6 kW) and the water inlet temperature and flow rate have been fixed at 12.8 C and 0.44 L/s. For these loads and water temperature and flow rate conditions, it has not been necessary to install heat exchanger on all of racks, 52%e70% of racks were enough. In addition, this technology could be transferred to other technological areas as well, as for waste heat recovery in air conditioning (in general) [21] or from diesel engine exhaust [22].
Indices a c HE R w
351
air component heat exchanger rack water
Exponents amb ambient exp experimental in inlet out outlet PS power supplies S servers sim simulated
The most of these research publications have used commercial CFD tools without any experimental studies. However, a recent study has characterized heat exchanger effectiveness by monitoring air and water temperatures [23]. The feature of this experiment is the horizontal position of exchanger which is installed at the bottom of the rack, below servers. An air duct with additional fans allows air flow circulation through heat exchanger. Possibility of heat reuse is one of the advantages of liquid cooling with hot water data center cooling. Several studies have shown the benefits of waste heat recovery from data centers [24,25]. Ebrahimi et al. [26] have listed some waste energy recovery techniques like district heating/hot water production, power plant co-location, absorption cooling or biomass colocation. Zimmermann et al. [27,28], have also reported exergy analysis of electronics cooling and have introduced a new metric, the economic value of heat. They have concluded that waste heat from data centers (with water temperatures above 60 C) could be used directly for space heating. According to these studies, it seems to be interesting to carry out an experimental study in order to use measurement data to model cooling system with non-commercial software. For instance, Song et al. [29] have used zonal method to describe the airflow and temperature patterns in a data center. This paper focuses on the cooling of a rear-door heat exchanger mounted on an electronic rack. No chiller has been used during the tests so inlet water temperature has been dependent on outside conditions. The objective is to estimate the energy efficiency of such system and to predict its functioning for any sets of input parameters. The paper presents an experimental and numerical study of the use of air to water finned tube heat exchanger to cool a rack. The first part is dedicated to the experimental setup to measure transient thermal response and energy efficiency of the system. In the second part numerical models of the rack and the exchanger are implemented and described. Results of measurements and simulations are presented in the third part. 2. Experimental set-up Fourteen IBM BladeCenter HS21 servers and eight power supplies were placed in a 1900 rack (Fig. 1). In total, the rack was constituted with four blowers to remove heat from servers and power supplies. Power supplies had additional smaller fans. The airflow rate was taken at the front side of the rack and vented warmer at the rear side due to the heat dissipation of electronic components. The airflow rate provided by the fans was a function of
352
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 1. Picture of half of the power supplies and servers.
the rack air inlet temperature. A cross-flow heat exchanger is mounted at the rear side of the rack to remove heat dissipated by the servers. An air duct was added between the rack and the heat exchanger to prevent air recirculation so all of the airflow from rack went through the heat exchanger. The rack and the heat exchanger were placed in a climatic chamber with a volume of 27 m3. The heat exchanger was connected to a water cooling loop which was completed with a pump and an outside heat exchanger to remove heat from the water loop. To evaluate the temperature evolution of the different equipment, a full array of instruments was installed. Several K type thermocouples were located at different positions (Fig. 2). One thermocouple was placed on front side of each of fourteen servers to measure the air inlet temperature (Tin R;a ). Furthermore, sixteen calibrated thermocouples were used to measure the air temperature of the two sides of the heat exchanger out (Tout R;a and THE;a ). Thermocouples were inserted to measure inlet (Tin ) and outlet water temperature (Tout w w ) of the heat exchanger. Moreover, eight thermocouples measured the temperature of the climatic chamber (Tamb ). a In addition, several thermocouples were placed to measure air temperature through the servers and some surface temperature components (Fig. 3). For these measurements, homemade thermocouples with wire diameter of 0.1 mm were used. Thus they
were placed between the surface component and heat sink. For the HDD component, ambient temperature was measured. Four servers on the fourteen were totally instrumented to evaluate the reproducibility of temperatures. In order to measure the airflow rate through the rack, four hotwire anemometer (Kimo CTV 210) were placed on the rear side of
Fig. 3. Picture of server and components.
Fig. 2. Schematic overview of experimental bench.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
the heat exchanger. A convergent duct was used to reduce the outlet surface from 750 mm 400 mm to 220 mm 220 mm in order to measure air velocity through a smaller surface. So each anemometer measured air velocity through a surface of 110 mm 110 mm. Further air velocity measurements on the front side of servers were carried out with the same sensors. Moreover, an electromagnetic flow meter (IFM Electronic SM8050) was installed in the water cooling loop to measure the water flow rate through the heat exchanger. Finally, electrical power consumption measurements of servers, power supplies, blowers, pump and external heat exchanger were taken with an energy analyzer (Fluke 434). All the sensors were connected to an Agilent 34972A or Fluke Hydra Series II data logger. Thermocouples were calibrated with a thermostat (Lauda RE104) and PT-100 sensor (Pico SE012þPT-104 data logger) for more accuracy on temperature measurements. The evaluation of the uncertainties on the sensors and the measurement chain is summarized in Table 1. During these experiments, several operating parameters were modified independently. The first parameter was the air inlet temperature of the rack. However, this temperature was directly related to the heat exchanger water inlet temperature. For these tests, the water target temperature was between 17 and 27 C, which correspond to an air temperature between 20 and 30 C. The water flow rate provided by the pump was the second parameter. Due to the pressure drop in the water cooling loop, the flow rate covered by the pump with variable speed ranged from 10 to 16 L/min. Finally, the power dissipated by the servers was the third parameter. Linux distribution was installed on each server and stress software was used to increase electrical power consumption of components (especially CPUs). Two modes of power dissipation were defined. The first was Pmin, whose only operating system was running, and the second mode Pmax whose components are stressed. 3. Numerical model In this part, we will present in a first time experimental measurements in transient state to see the thermal response of rack and heat exchanger in relation to operating parameter variations. Based on these results, the second part will focus on numerical model which were taken on. 3.1. Measurement series In order to predict temperature evolutions of rack and heat exchanger, tests were carried out with operating parameter variations (Fig. 4). During this measurement series, the inlet air temperatures varied in increments of 5 C. The water flow rate remained constant during 42 h at 14.6 L/min whereas the rack power consumption was set at 2,750 W and then at 4,150 W for the last hours. Fig. 5 shows the experimental results for the rack and heat exchanger temperatures. We notice that surface component temperatures are not homogeneous. Indeed, lowest temperatures are
processor temperatures (TCPU) whereas highest temperatures are RAM temperatures (TRAM). The power consumption variation is clearly seen at the 42nd hour with a quickly increase of the most of temperatures. We observe that the rack outlet temperature (Tout R;a ) is lower than the server outlet temperature (T42cm). This is due to the air mixing between hot air from the servers and colder air from the power supplies. Moreover, air temperature at x ¼ 14 cm (T14cm) is close to processor temperatures at Pmin and lower than it at Pmax with a difference of 7 C. It can be explained by the fact that thermocouple which measures air temperature at x ¼ 14 cm is not at the good place to measure hot air after processors. Temperatures at x ¼ 29 and x ¼ 42 cm are more revealing to evaluate air temperature evolution through servers. The temperatures of the heat exchanger were also measured during the same time. We note that the air outlet temperature out (Tout HE;a ) and water outlet temperature (Tw ) were very close all the time. The maximal difference between these two temperatures was about 0.5 C. 3.2. Nodal models Fig. 6 shows the nodal representation of the rack by electrical analogy. Nodes and conductances correspond to temperatures and heat transfers by conduction and convection, respectively. Moreover, heat capacitances are represented with capacitances [30]. The airflow is modeled by directional fluid conductances. The value of airflow rate for a given inlet temperature is gotten thanks to the experimental results described in the next part. Input variables are the rack inlet temperature (TiR;a ), the climatic chamber temperature (Tamb ), and the power consumption of the a rack (Pmin or Pmax) as function of time. The total airflow through the rack is divided into two non-equal parts in order to model the airflow through servers (Q Sa ) and power supplies (Q PS a ). According to experimental results shown in paragraph 3.1, some physical points are selected to be the nodes of numerical models. Node 1 (TSa;1 ) and node 2 (TSa;2 ) correspond to the air temperature at x ¼ 29 cm and x ¼ 42 cm, respectively (Fig. 3). They represent the equivalent temperatures in the first part (from 0 to 29 cm) and the second part (from 29 to 42 cm) of servers. In addition, node 3 (TSc;1 ) and node 4 (TSc;2 ) represent the equivalent component temperatures in each part. The total power consumption of servers is divided into two fractions P and P’ and associated to node 3 and node 4. The convection/conduction heat transfers between nodes 1, 3 and 2, 4 are represented by heat conductances h13 and h24. Heat transfer between ambient air (Tamb ) and servers are also taken into a account.
Table 1 Sensors and evaluation of their measure uncertainties. Sensor
Accuracy (MV ¼ Measured Value)
Thermocouple type K Anemometer Water flow meter Electrical power
±0.1 C ±3% MV ±0.1 m/s ±2% MV ±0.5 L/min ±1.5% MV ±10 W
353
Fig. 4. Operating parameter variations.
354
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 5. Experimental temperature profiles as function of time a) inside servers, b) for heat exchanger.
Fig. 6. Nodal representation of rack model.
The heat transfer by convection through the power supplies is also modeled. The node TPS a corresponds to the outlet air temperature of the power supplies. The power dissipation P00 of power supplies represents the thermal power loses during AC/DC conversion. Power supply efficiency is defined thanks to power measurements and is evaluated around 90%. Node 6 corresponds to the air outlet temperature of the rack (Tout R;a ) where airflow from servers and power supplies join. In order to solve the model, the heat equation was solved for each node. Then this system is discretized using an implicit order scheme and the derivative was approximate by finite differences. Major equations of the rack model are given below:
. · S S a S in S dt ¼ h00 1 Tamb þ ma Cap TR;a Ca;1 $dTa;1 Ta;1 Ta;1 S S Ta;1 þ h31 Tc;1
consumption P of the first part of servers is also injected into equation (2).
. · S · PS out S out out dt ¼ ma Cap Ta;2 þ ma Cap TaPS TR;a Ca;4 $dTR;a TR;a
A nodal representation of the cross flow heat exchanger was also made (Fig. 7). Input parameters are the water inlet temperature · · (Tiw ), the water mass flow rate (m w ), the airflow rate (m a ) and the out air inlet temperature (TR;a ), which is calculated from the rack model (node 6). Two intermediary nodes were introduced, corresponding to the arithmetic mean of the inlet and the outlet measured temm perature of air (Tm HE;a ) and water (Tw ). Finally, node 10 and node 11 represent the air outlet temperature (ToHE;a ) and the water outlet temperature (Tow ).
(1)
with Ca,1 the heat capacitance of the air, ṁSa the airflow rate through servers. h00 1 and h31 are heat conductances to model heat transfers with ambient air and surface components.
. S S S dt¼ h13 Ta;1 þP C1 $dTc;1 Tc;1
(2)
with C1 the heat capacitance of components in the first part of servers. h13 corresponds to heat conductance h31 described previously and represents heat transfers with air. The equivalent power
(3)
Fig. 7. Nodal representation of heat exchanger.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
355
Heat balance on each allows defining a system of differential equations. The equation for the node 9 is given below:
· m m m in m þ mw C w Cw;1 $dTw Tw dt¼ h89 THE;a p Tw Tw
(4)
3.3. Identification procedure The objective was to estimate the value of the different heat conductances and heat capacity thanks to experimental measurements. The method of estimation known as the least squares estimation method was used [31]. The normalized cost function introduced is given by the following equation:
f ðC; hÞ ¼
1 Nmes
m N i2 mes h X X exp Tj ðpÞ Tjsim ðpÞ
(5)
j¼1 p¼1
where Nmes is the total number of time steps and m is the number of nodes. C ¼ [C1, C2, …] and h ¼ [h100 , h13, …] are the values of heat capacitance and heat conductances and are the unknown parameters to identify. The algorithm used consists in minimizing, for several sets of values x¼(C, h), the normalized cost function. This is a gradientbased method to find minimum of constrained nonlinear multivariable function [32]. 4. Results and discussion In the first part, cooling efficiency of the solution is discussed. Steady state measurements are described and studied. According to measurements, reproducibility was observed in spite of little temperature differences. To compare the results with traditional air cooling, additional measurements were also made with the rack without the heat exchanger by using the air conditioning system of the climatic chamber. In the second part, transient state measurements are presented. The results of the parameter estimation are given and the numerical models are validated using several sets of measurements. 4.1. Steady state experimental results 4.1.1. Airflow rate measurement As airflow rate depends on rack air inlet temperature, measurements have been done for air inlet temperature varying from 15 C to 45 C (Fig. 8). The uncertainty was evaluated at 5%. From 15 C to 25 C, the airflow rate is constant at 1,066 m3/h and increases from 25 C to 33 C to reach a value of 1,894 m3/h. The airflow rate is also constant from 33 C to 39 C, and from 39 C the airflow rate is maximal with a value near 2,020 m3/h. Airflow rate measurements have also shown that only 32% of the total airflow passes through the servers. We assume that the remaining airflow rate is equal to the airflow rate through the power supplies. Nevertheless, heat exchanger added a pressure drop resulting in 20% airflow rate decrease. At the end, these measurements allow to calculate the airflow rate through the rack and the heat exchanger whatever the rack inlet temperature. 4.1.2. Temperature measurements and effectiveness Fig. 9 shows the temperatures of components at Pmin and Pmax for different values of rack inlet temperature. The measurements were taken in steady state for 1 h. In each case, temperatures do not reach the maximal recommended operating temperatures.
Fig. 8. Evolution of airflow rate with air inlet temperature.
An increase of the temperatures between the two power modes was observed. On average, the surface temperatures of components rose 13.8 C for CPUs, 8.7 C for the chipset and the SAS, 12.6 C for the RAM and finally 7.7 C for the HDD. Compared with air cooling results, temperatures of components were 2.5 C higher on average with a maximum of 8.1 C for the SAS. This disparity can be explained by the airflow rate reduction described previously. In few cases, the temperatures of components when air inlet temperature was near 30 C, were lower than cases that had air inlet temperature near 25 C (for example, temperatures of SAS at Pmin). This phenomenon was due to the raise of the airflow rate. Furthermore, temperature variations through the servers show the power distribution along them (Fig. 10). Logically, we observe that power dissipation is more important at the beginning of the servers because of CPUs’ localization. Between x ¼ 0 and x ¼ 29 cm, the temperatures increased by 10 and 16.4 C on average at Pmin and Pmax, respectively. Then, the temperature evolution is lower between x ¼ 29 cm and x ¼ 42 cm with a raise of 3.1 and 4.3 C for each power dissipation. At x ¼ 42 cm, the temperatures values at Pmax are closest than at Pmin with a maximal difference of 2.5 C against 8.1 C at Pmin. Finally, fluid temperatures and effectiveness of the heat exchanger are summarized in Table 2. The inlet water temperatures are close to the water temperature set points fixed at 17, 22 and 27 C. The inlet air temperature of the heat exchanger is the same as the outlet temperature of the rack. These values increased normally at the higher power dissipation. Moreover, the outlet water temperatures and outlet air temperatures of the heat exchanger were very close. The average difference between these temperatures was 0.2 C and 0.4 C at Pmin and Pmax, respectively. The effectiveness of the heat exchanger is given by the following relationship:
. i o i i THE;a ε ¼ THE;a THE;a Tw
(6)
The average effectiveness decreased at the higher airflow rate and water inlet temperature. Note that for all experiments the water flow rate is unchanged and equal to 15.7 L/min. The variation in the effectiveness for close airflow rate is due to the measurement accuracy.
4.1.3. Power measurements and energy efficiency In order to evaluate the ratio of heat removed by the heat exchanger, heat balances were calculated at the rack and heat exchanger levels (Fig. 11). In addition, electrical power measurements of the blowers and the servers (with power supplies) were measured to compare electrical and heat balances. In steady state the heat gain in air (Fa) through the rack and the heat gain in water
356
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 9. Component temperatures a) at Pmin, b) at Pmax.
Fig. 10. Evolution of T(x) for different values of Tin R;a and power dissipation.
Fig. 11. Electrical (dashed lines) and thermal (dotted lines) balance.
(Fw) through the heat exchanger are given by the following equations:
· out in Fa ¼ ma Cap TR;a TR;a
(7)
· out in Fw ¼ mw Cw p Tw Tw
(8)
Uncertainties were evaluated at 6% and 12% for the air-energy balance and the water-energy balance, respectively. Logically, the power consumption of the blowers rose with the rack inlet temperature like the airflow rate. At 20 C, the blowers used roughly 280 W against 880 W at 32.2 C. This represented an increase of 214% from the minimal value of power consumption. The power consumption of the servers and the power supplies was around 2,755 W at Pmin and reached to 4,150 W at Pmax. For each power dissipation, these values were stable whatever the rack inlet temperature. For rack inlet temperatures between 20.1 and 26.6 C, the air thermal balance was about 2771 and 4,103 W for Pmin and Pmax, Table 2 Fluid temperatures and effectiveness of heat exchanger. Pmin Tin ( C) ·R;a ma (kg/s) Tin w ( C) Tout w ( C) Tin HE;a ( C) Tout HE;a ( C) ε
20.1 0.356 17.6 20.0 27.8 20.0 0.76
Pmax 25.1 0.351 22.8 25.1 32.9 25.0 0.78
30.9 0.522 28.2 30.8 37.3 30.8 0.71
21.7 0.355 17.7 21.2 33.0 21.6 0.75
26.6 0.392 22.9 26.4 37.1 26.6 0.74
32.2 0.571 27.9 31.8 40.7 32.2 0.66
respectively. This represents roughly 100% of the electrical power consumption of the servers and the power supplies. The water thermal balance was around 2561 W at Pmin and 3809 W at Pmax for the same temperature range. These thermal balances corresponded to an average ratio of 92% of heat removed from the rack. However, at temperatures close to 30 C or more, this ratio was higher (more than 115% for air thermal balance and 105% for water thermal balance) but still within uncertainty ranges. The energy efficiency of the cooling system is also presented. We recall that the power consumption of the rack and the cooling system represents the total power of the system (PIT þ PCooling) and the power consumption of the cooling system is only given by those of pump and outdoor water to air heat exchanger. All of these values for the different cases presented in this part are summarized in Table 3. As the power consumption of the pump was always constant at 21 W, the power variations of the cooling system were due to the outdoor heat exchanger. Outside temperatures and the amount of heat removed explained these power variations. Finally, to evaluate the energy efficiency of the cooling system, we used a variant of the PUE, named pPUE (partial Power Usage Effectiveness) [33]. The pPUE has a close definition as PUE and in our case it focuses on the cooling energy consumption:
Table 3 Power consumption and energy efficiency of experimental bench. Pmin Tin R;a ( C) PIT (W) PCooling (W) pPUE
20.1 3,001 73 1.02
Pmax 25.1 3,047 65 1.02
30.9 3,510 56 1.02
21.7 4,365 101 1.02
26.6 4,497 84 1.02
32.2 5,094 69 1.01
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
pPUE ¼ ðrack energy þ cooling energyÞ=rack energy . PIT ¼ PIT þ Pcooling
(9)
The energy efficiency indicator of the cooling system is roughly 1.02. Compared with literature, this value is very low and satisfying despite low power load used. In addition, this study has given great results and a substantial database of measurements.
357
The values of the other conductances are assumed constant; one observes a short variation for high airflow rate. Fig. 12 gives the calculated heat conductances for the heat exchanger. These conductances were function of the total volumetric airflow rate and the water flow rate through the heat exchanger. A first order regression was made to express heat conductance linear dependency in term of these two operating parameters. For example, the relationship for the heat conductance h6,11 is given by the equation:
4.2. Parameter identification and validation of numerical model
h6;11 ðQa ; Qw Þz 4:27$105 Qa Qw þ 435Qa 86:7$103 Qw þ 81 4.2.1. Parameter identification In a first time, the parameter identification was done in steady state to find the heat conductances. Indeed, in steady state, the left part of equations (1)e(4) is equal to zero. The measurement data presented previously is used (Fig. 4). Fig. 12 shows the results for the server model. We observe that the heat conductance h13 depends on airflow rate through the rack and power dissipation. At Pmin, the value is near 150 W/K whereas at Pmax the value is roughly 250 W/K. Then, to model these dependencies, the following relations are retained:
h13 QaS z
(
185 QaS þ 134 ðPmin modeÞ 287 QaS þ 211 ðPmax modeÞ
(10)
(11) Once the heat conductances identified, the entire measurement series are exploited to estimate thermal capacities. After optimization, the calculated temperatures are close to the experimental data (Fig. 13). The corresponding heat capacitance values of each node for the rack and heat exchanger models are summarized in Table 4. For the rack, we observe that heat capacitances of nodes 1 and 3 are higher than those of other nodes (Table 5). These nodes corresponded to the physical points which were at the beginning of the servers. It can be explained by the fact that most of power dissipation occurs here. The rest of the heat capacitances were close.
Fig. 12. Calculation of heat conductances as function of airflow rate, water flow and power dissipation a) of the rack, b) of the heat exchanger.
Fig. 13. Model simulation on identification measurement series a) of rack model, b) of heat exchanger model.
358
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
capacitances from air side (Ca,5, Ca,6) were lower than those from water side (Cw,1, Cw,2).
Table 4 Heat capacitances of the rack model.
Thermal capacity (kJ K1)
Ca,1
Ca,2
C1
C2
Ca,3
Ca,4
Total
16.8
3.5
14.6
4.1
5.9
5.6
50.5
Table 5 Heat capacitances of the heat exchanger model.
Thermal capacity (kJ K1)
Ca,5
Cw,1
Ca,6
Cw,2
Total
43.5
47.1
56.4
61.9
208.9
Considering the total mass of the rack, the sum of all capacitances gave a value which was constant. Indeed, taking the heat capacity of steel (465 J/kg/K) and the supposed mass of the IT equipment (130 kg, from the equipment datasheet), we obtain a value near 60 kJ/K. All of the heat capacitances of the heat exchanger model had the same order of magnitude. Indeed, the values of the capacitances Ca,5 and Cw,1 were around 43.5 and 47.1 kJ/K whereas for the outlet temperature nodes, the values of Ca,6 and Cw,2 were 56.4 and 61.9 kJ/K. This disparity can be explained by the choice of the intermediate nodes defined as the arithmetic average of inlet and outlet temperatures which not correspond to the temperatures in the middle of the heat exchanger. We noticed that the heat
Fig. 14. Operating parameter variations for model validation.
4.2.2. Model validation The numerical models of rack and heat exchanger defined previously were tested on additional measurement data with variable variations of the operating input (Fig. 14). This step is important to valid numerical models of each system with other values of operating parameters. Air inlet temperature went from 24.7 to 27.8 C, the inlet water temperature was near 22 C whereas water flow rate was ranged from 10 to 14.7 L/min. Server power consumption varied during the tests. Fig. 15 shows the results for these operation parameters. We notice that the average deviation of temperatures between experiment and simulation is less than 7% for the rack model and around 4% for the heat exchanger model. The simulated temperatures are close to the experimental data but the rack outlet temperature (ToR,a) is overestimated (þ0.7 C) at Pmax and underestimated (0.6 C) at Pmin. Despite these disparities, the temperature predictions were acceptable throughout the tests. 5. Conclusion This paper presented an experimental and numerical study of air to water heat exchanger mounted on an electronic rack. The experimental part of the study was carried out with a substantial instrumentation at different scales of the system. In order to characterize thermal response of the installation, a monitoring of temperatures, electrical power, air and water flow were realized. The temperature of electronic component is maintained under their maximum operating temperatures. Air and water thermal balances showed that more 90% of the heat dissipated by the rack was removed by the heat exchanger. Promising results were also obtained with the estimation of the energy efficiency metric of the cooling system which had very low values. This was mainly due to the low power consumption of the pump and the outside heat exchanger. The partial energy efficiency indicator pPUE had very promising values close to 1.02, which is less than traditional air cooling. In parallel, thermal numerical models were defined at the scale of servers/rack and heat exchanger by using nodal approach. System of linear differential equations was written by heat balance equation for each node of the two proposed models. Thus, by using least squares estimation method and experimental data, the heat capacitances and heat transfer coefficients were evaluated for rack
Fig. 15. Model validations on additional measurement series a) of rack model, b) of heat exchanger model.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
and heat exchanger. Furthermore, complementary experimental measurements were used to validate the numerical models. The models were tested on two measurements series. The results were very satisfactory, with an error on temperature predictions less than 10%. These numerical models will be able to predict the temperatures of each system as function of fluid inlet temperatures, flow rates and power consumption of the electronic equipment. To estimate the impact of such cooling solution for a data center, they will be duplicated at a room scale. References [1] J.G. Koomey, Worldwide electricity used in data centers, Environ. Res. Lett. 3 (3) (Jul. 2008) 034008. [2] W. Van Heddeghem, S. Lambert, B. Lannoo, D. Colle, M. Pickavet, P. Demeester, Trends in worldwide ICT electricity consumption from 2007 to 2012, Comput. Commun. 50 (Sep. 2014) 64e76. [3] J.F. Karlsson, B. Moshfegh, Investigation of indoor climate and power usage in a data center, Energy Build. 37 (10) (Oct. 2005) 1075e1083. [4] S.V. Garimella, L.-T. Yeh, T. Persoons, Thermal management challenges in telecommunication systems and data centers, IEEE Trans. Compon. Packag. Manuf. Technol. 2 (8) (Aug. 2012) 1307e1316. [5] G.A. Brady, N. Kapur, J.L. Summers, H.M. Thompson, A case study and critical assessment in calculating power usage effectiveness for a data centre, Energy Convers. Manag. 76 (Dec. 2013) 155e161. [6] J. Cho, T. Lim, B.S. Kim, Viability of datacenter cooling systems for energy efficiency in temperate or subtropical regions: case study, Energy Build. 55 (Dec. 2012) 189e197. [7] K. Kant, Data center evolution: a tutorial on state of the art, issues, and challenges, Comput. Netw. 53 (17) (Dec. 2009) 2939e2965. [8] ASHRAE TC 9.9, 2011 Thermal Guidelines for Data Processing Environments e Expanded Data Center Classes and Usage Guidance, ASHRAE, 2011. [9] S.-Y. Jing, S. Ali, K. She, Y. Zhong, State-of-the-art research study for green cloud computing, J. Supercomput 65 (1) (Jul. 2013) 445e468. [10] B. Fakhim, M. Behnia, S.W. Armfield, N. Srinarayana, Cooling solutions in an operational data centre: a case study, Appl. Therm. Eng. 31 (14e15) (Oct. 2011) 2279e2291. [11] Y. Fulpagare, G. Mahamuni, A. Bhargav, Effect of plenum chamber obstructions on data center performance, Appl. Therm. Eng. 80 (Apr. 2015) 187e195. [12] K.-P. Lee, H.-L. Chen, Analysis of energy saving potential of air-side free cooling for data centers in worldwide climate zones, Energy Build. 64 (Sep. 2013) 103e112. [13] T. Lu, X. Lü, M. Remes, M. Viljanen, Investigation of air management and energy performance in a data center in Finland: case study, Energy Build. 43 (12) (Dec. 2011) 3360e3372. [14] H. Endo, H. Kodama, H. Fukuda, T. Sugimoto, T. Horie, M. Kondo, Effect of climatic conditions on energy consumption in direct fresh-air container data centers, in: Green Computing Conference (IGCC), 2013 International, 2013, pp. 1e10. [15] H. Zhang, S. Shao, H. Xu, H. Zou, C. Tian, Integrated system of mechanical refrigeration and thermosyphon for free cooling of data centers, Appl. Therm. Eng. 75 (Jan. 2015) 185e192.
359
[16] S.-W. Ham, J.-S. Park, J.-W. Jeong, Optimum supply air temperature ranges of various air-side economizers in a modular data center, Appl. Therm. Eng. 77 (Feb. 2015) 163e179. [17] R. Udakeri, V. Mulay, D. Agonafer, “Comparison of overhead supply and underfloor supply with rear heat exchanger in high density data center clusters, in: Twenty-fourth Annual IEEE Semiconductor Thermal Measurement and Management Symposium, 2008. Semi-Therm 2008, 2008, pp. 165e172. [18] R. Schmidt, R.C. Chu, M. Ellsworth, M. Iyengar, D. Porter, V. Kamath, B. Lehman, Maintaining Datacom Rack Inlet Air Temperatures with Water Cooled Heat Exchanger, Jan. 2005, pp. 663e673. [19] A. Almoli, A. Thompson, N. Kapur, J. Summers, H. Thompson, G. Hannah, Computational fluid dynamic investigation of liquid rack cooling in data centres, Appl. Energy 89 (1) (Jan. 2012) 150e155. [20] K. Karki, S. Novotny, A. Radmehr, S. Patankar, Use of passive, rear-door heat exchangers to cool low to moderate heat loads, ASHRAE Trans. 117 (2) (2011). [21] M.A. Abd El-Baky, M.M. Mohamed, “Heat pipe heat exchanger for heat recovery in air conditioning, Appl. Therm. Eng. 27 (4) (Mar. 2007) 795e801. [22] V. Pandiyarajan, M. Chinna Pandian, E. Malan, R. Velraj, R.V. Seeniraj, Experimental investigation on heat recovery from diesel engine exhaust using finned shell and tube heat exchanger and thermal storage system, Appl. Energy 88 (1) (Jan. 2011) 77e87. [23] K. Nemati, B.T. Murray, B. Sammakia, Experimental characterization and modeling of a water-cooled server cabinet, in: 2014 IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2014, pp. 723e728. [24] J. Zachary Woodruff, P. Brenner, A.P.C. Buccellato, D.B. Go, Environmentally opportunistic computing: a distributed waste heat reutilization approach to energy-efficient buildings and data centers, Energy Build. 69 (Feb. 2014) 41e50. [25] T. Brunschwiler, B. Smith, E. Ruetsche, B. Michel, Toward zero-emission data centers through direct reuse of thermal energy, IBM J. Res. Dev. 53 (3) (May 2009) 11, 1e11:13. [26] K. Ebrahimi, G.F. Jones, A.S. Fleischer, A review of data center cooling technology, operating conditions and the corresponding low-grade waste heat recovery opportunities, Renew. Sustain. Energy Rev. 31 (Mar. 2014) 622e638. [27] S. Zimmermann, I. Meijer, M.K. Tiwari, S. Paredes, B. Michel, D. Poulikakos, Aquasar: a hot water cooled data center with direct energy reuse, Energy 43 (1) (Jul. 2012) 237e245. [28] S. Zimmermann, M.K. Tiwari, I. Meijer, S. Paredes, B. Michel, D. Poulikakos, Hot water cooled electronics: exergy analysis and waste heat reuse feasibility, Int. J. Heat Mass Transf. 55 (23e24) (Nov. 2012) 6391e6399. [29] Z. Song, B.T. Murray, B. Sammakia, A dynamic compact thermal model for data center analysis and control using the zonal method and artificial neural networks, Appl. Therm. Eng. 62 (1) (Jan. 2014) 48e57. [30] W.H. Tang, Q.H. Wu, Z.J. Richardson, Equivalent heat circuit based power transformer thermal model, Electr. Power Appl. IEE Proc. 149 (2) (Mar. 2002) 87e92. [31] J.V. Beck, K.J. Arnold, Parameter Estimation in Engineering and Science, Wiley, , New York, NY, 1977. [32] R.H. Byrd, J.C. Gilbert, J. Nocedal, A trust region method based on interior point techniques for nonlinear programming, Math. Program 89 (1) (Nov. 2000) 149e185. [33] The Green Grid, PUETM: a Comprehensive Examination of the Metric, White Paper #49, 2012.
Contents lists available at ScienceDirect
Applied Thermal Engineering journal homepage: www.elsevier.com/locate/apthermeng
Research paper
Experimental and numerical study of water-cooled datacom equipment pahne Le Masson a, Patrick Glouannec b Fabien Douchet a, b, *, David Nortershauser a, Ste a b
Orange Labs, 2 Avenue Pierre Marzin, 22300 Lannion, France LIMATB, Rue de Saint Maud e, Universit e de Bretagne Sud, 56321 Lorient Cedex, France
h i g h l i g h t s A study of an electronic rack cooled by air to water heat exchanger is carried out. Experimental study with a substantial instrumentation at different scales is performed. The energy efficiency of the cooling system is highlighted. Numerical model of the system by using nodal approach is defined and validated.
a r t i c l e i n f o
a b s t r a c t
Article history: Received 20 January 2015 Accepted 14 March 2015 Available online 1 April 2015
This paper presents an experimental and numerical study of an electronic rack cooled by a finned tube heat exchanger. The objectives are to evaluate cooling and energy performance of this system and formulate a numerical model based on measurements. Experimentation has been carried out using commercial servers. A substantial instrumentation has been conducted at different scales (servers, rack and exchanger). Several tests have been made with different sets of parameters like water inlet temperature or power dissipated by servers. In each case, the heat exchanger has removed more than 90% of electrical power consumed by the rack (with no chiller use). Furthermore, promising results are obtained with this system, especially the reduction of energy consumption of cooling part compared to traditional air cooling. Finally, the rack and heat exchanger are simplified into two numerical models which can predict temperature outputs as a function of defined inputs (water and air flow rate and temperature, power dissipation). Validation tests have been carried out with different sequences of measure and numerical models have given satisfactory results. They will be duplicated to compute the impact of this cooling system at a data center room scale. © 2015 Elsevier Ltd. All rights reserved.
Keywords: Data center Cooling Instrumentation Heat exchanger Energy efficiency Numerical model
1. Introduction With the expansion of Internet services (cloud, social networking, etc …), thermal management in telecommunication systems and data centers has become a major issue for operators. In 2005, data centers represented 1% of the worldwide electricity consumption [1]. According to Van Heddeghem [2], worldwide data centers electricity consumption has grown from 2007 to 2012 by 4% per year. Typically, the power load per rack starts from 5 kW for low density equipment and can reach 40 kW for high density system like supercomputers [3,4]. * Corresponding author. Orange Labs, 2 Avenue Pierre Marzin, 22300 Lannion, France. Tel.: þ33 296072334. E-mail address: [email protected] (F. Douchet). http://dx.doi.org/10.1016/j.applthermaleng.2015.03.030 1359-4311/© 2015 Elsevier Ltd. All rights reserved.
The Power Usage Effectiveness (PUE) is energy efficiency metric for data centers defined as the ratio of total energy consumption of the data center to energy consumption used by the IT equipment [5]. Typical values of PUE are around 1.6, and can decrease to 1.2 in case of best cooling practices application [6]. These values confirm the fact that cooling architecture represents 25% or more of the total power consumption of data centers [7]. To maintain high reliability in datacenters, it is recommended to maintain datacom equipment into specific climatic range according to European Telecommunications Standards Institute (ETSI) [8]. As power consumption has increased, the power dissipation of telecommunication equipment has dramatically raised too. With high density servers and power supplies, electronic equipment is fitted with high power fans to remove heat. It will become more and more difficult to cool IT equipment by air cooling with Computer
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Nomenclature C Cp h
thermal capacity, J K1 specific heat capacity, J kg1 K1 thermal conductance, W K1
m P Q t T
mass flow rate, kg s1 power, W volumetric flow, m3 s1 time, s temperature, K
·
Greek symbols ε effectiveness F thermal heat flux, W
Room Air Conditioning (CRAC) systems usually used in data centers. In the most of them, racks are arranged in rows with cold and hot aisles configurations [9]. Row containments can be added to eliminate hot spots and air recirculation [10]. Fulpagare et al. [11] performed numerical studies to quantify the effect of plenum obstructions on total CRAC flow rates to increase data center performances. The authors indicated that obstructions in plenum chambers lead to decreased airflow rates by as much as 80%, with an increase in the occurrence of hot spots. Nevertheless, the use of chillers is very expensive in terms of energy consumption. Free cooling [12,13], and container data centers [14] are innovations that have reduced the part of cooling consumption by using direct freshair from outside. Zhang et al. [15] made an experimental study for free cooling of data centers by combining refrigeration and thermosyphon loops independently of each other. They also show that annual energy-saving rates are between 5.4 and 47.3% with an indoor temperature is set on 27 C. Moreover, Ham et al. [16] analyzed the effect of supply air temperature ranges for different configurations of modular data center. Energy consumption reductions appear when air inlet temperature is set on 18e23 C. Higher supply air temperatures are not recommended due to a rise of CRAC energy fans which increase cooling energy consumption. However, the efficiency depends on the location of the data centers because of annual humidity and temperature conditions. Switching to liquid cooling seems to be an alternative option to make energy efficiency cooling systems. Several studies deal with Computational Fluid Dynamics (CFD) simulations of water-cooled rear-door heat exchanger. Udakeri et al. [17] have investigated hybrid cooling solutions which consist in associated air and liquid cooling for 32 kW racks. They concluded that underfloor supply configuration has better performance than overhead supply configuration. In this case, heat exchangers have removed 55% of the total 32 kW rack with a temperature of air chiller entering the room fixed at 13 C. Some additional studies have shown that the use of liquid cooling with rear heat exchanger have drastically reduced the power consumption of CRAC systems [18,19]. Furthermore, Karki et al. [20] studied the impact of using rear-door heat exchangers as the sole source of cooling. Simulations have been made with moderate rack heat loads (3e6 kW) and the water inlet temperature and flow rate have been fixed at 12.8 C and 0.44 L/s. For these loads and water temperature and flow rate conditions, it has not been necessary to install heat exchanger on all of racks, 52%e70% of racks were enough. In addition, this technology could be transferred to other technological areas as well, as for waste heat recovery in air conditioning (in general) [21] or from diesel engine exhaust [22].
Indices a c HE R w
351
air component heat exchanger rack water
Exponents amb ambient exp experimental in inlet out outlet PS power supplies S servers sim simulated
The most of these research publications have used commercial CFD tools without any experimental studies. However, a recent study has characterized heat exchanger effectiveness by monitoring air and water temperatures [23]. The feature of this experiment is the horizontal position of exchanger which is installed at the bottom of the rack, below servers. An air duct with additional fans allows air flow circulation through heat exchanger. Possibility of heat reuse is one of the advantages of liquid cooling with hot water data center cooling. Several studies have shown the benefits of waste heat recovery from data centers [24,25]. Ebrahimi et al. [26] have listed some waste energy recovery techniques like district heating/hot water production, power plant co-location, absorption cooling or biomass colocation. Zimmermann et al. [27,28], have also reported exergy analysis of electronics cooling and have introduced a new metric, the economic value of heat. They have concluded that waste heat from data centers (with water temperatures above 60 C) could be used directly for space heating. According to these studies, it seems to be interesting to carry out an experimental study in order to use measurement data to model cooling system with non-commercial software. For instance, Song et al. [29] have used zonal method to describe the airflow and temperature patterns in a data center. This paper focuses on the cooling of a rear-door heat exchanger mounted on an electronic rack. No chiller has been used during the tests so inlet water temperature has been dependent on outside conditions. The objective is to estimate the energy efficiency of such system and to predict its functioning for any sets of input parameters. The paper presents an experimental and numerical study of the use of air to water finned tube heat exchanger to cool a rack. The first part is dedicated to the experimental setup to measure transient thermal response and energy efficiency of the system. In the second part numerical models of the rack and the exchanger are implemented and described. Results of measurements and simulations are presented in the third part. 2. Experimental set-up Fourteen IBM BladeCenter HS21 servers and eight power supplies were placed in a 1900 rack (Fig. 1). In total, the rack was constituted with four blowers to remove heat from servers and power supplies. Power supplies had additional smaller fans. The airflow rate was taken at the front side of the rack and vented warmer at the rear side due to the heat dissipation of electronic components. The airflow rate provided by the fans was a function of
352
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 1. Picture of half of the power supplies and servers.
the rack air inlet temperature. A cross-flow heat exchanger is mounted at the rear side of the rack to remove heat dissipated by the servers. An air duct was added between the rack and the heat exchanger to prevent air recirculation so all of the airflow from rack went through the heat exchanger. The rack and the heat exchanger were placed in a climatic chamber with a volume of 27 m3. The heat exchanger was connected to a water cooling loop which was completed with a pump and an outside heat exchanger to remove heat from the water loop. To evaluate the temperature evolution of the different equipment, a full array of instruments was installed. Several K type thermocouples were located at different positions (Fig. 2). One thermocouple was placed on front side of each of fourteen servers to measure the air inlet temperature (Tin R;a ). Furthermore, sixteen calibrated thermocouples were used to measure the air temperature of the two sides of the heat exchanger out (Tout R;a and THE;a ). Thermocouples were inserted to measure inlet (Tin ) and outlet water temperature (Tout w w ) of the heat exchanger. Moreover, eight thermocouples measured the temperature of the climatic chamber (Tamb ). a In addition, several thermocouples were placed to measure air temperature through the servers and some surface temperature components (Fig. 3). For these measurements, homemade thermocouples with wire diameter of 0.1 mm were used. Thus they
were placed between the surface component and heat sink. For the HDD component, ambient temperature was measured. Four servers on the fourteen were totally instrumented to evaluate the reproducibility of temperatures. In order to measure the airflow rate through the rack, four hotwire anemometer (Kimo CTV 210) were placed on the rear side of
Fig. 3. Picture of server and components.
Fig. 2. Schematic overview of experimental bench.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
the heat exchanger. A convergent duct was used to reduce the outlet surface from 750 mm 400 mm to 220 mm 220 mm in order to measure air velocity through a smaller surface. So each anemometer measured air velocity through a surface of 110 mm 110 mm. Further air velocity measurements on the front side of servers were carried out with the same sensors. Moreover, an electromagnetic flow meter (IFM Electronic SM8050) was installed in the water cooling loop to measure the water flow rate through the heat exchanger. Finally, electrical power consumption measurements of servers, power supplies, blowers, pump and external heat exchanger were taken with an energy analyzer (Fluke 434). All the sensors were connected to an Agilent 34972A or Fluke Hydra Series II data logger. Thermocouples were calibrated with a thermostat (Lauda RE104) and PT-100 sensor (Pico SE012þPT-104 data logger) for more accuracy on temperature measurements. The evaluation of the uncertainties on the sensors and the measurement chain is summarized in Table 1. During these experiments, several operating parameters were modified independently. The first parameter was the air inlet temperature of the rack. However, this temperature was directly related to the heat exchanger water inlet temperature. For these tests, the water target temperature was between 17 and 27 C, which correspond to an air temperature between 20 and 30 C. The water flow rate provided by the pump was the second parameter. Due to the pressure drop in the water cooling loop, the flow rate covered by the pump with variable speed ranged from 10 to 16 L/min. Finally, the power dissipated by the servers was the third parameter. Linux distribution was installed on each server and stress software was used to increase electrical power consumption of components (especially CPUs). Two modes of power dissipation were defined. The first was Pmin, whose only operating system was running, and the second mode Pmax whose components are stressed. 3. Numerical model In this part, we will present in a first time experimental measurements in transient state to see the thermal response of rack and heat exchanger in relation to operating parameter variations. Based on these results, the second part will focus on numerical model which were taken on. 3.1. Measurement series In order to predict temperature evolutions of rack and heat exchanger, tests were carried out with operating parameter variations (Fig. 4). During this measurement series, the inlet air temperatures varied in increments of 5 C. The water flow rate remained constant during 42 h at 14.6 L/min whereas the rack power consumption was set at 2,750 W and then at 4,150 W for the last hours. Fig. 5 shows the experimental results for the rack and heat exchanger temperatures. We notice that surface component temperatures are not homogeneous. Indeed, lowest temperatures are
processor temperatures (TCPU) whereas highest temperatures are RAM temperatures (TRAM). The power consumption variation is clearly seen at the 42nd hour with a quickly increase of the most of temperatures. We observe that the rack outlet temperature (Tout R;a ) is lower than the server outlet temperature (T42cm). This is due to the air mixing between hot air from the servers and colder air from the power supplies. Moreover, air temperature at x ¼ 14 cm (T14cm) is close to processor temperatures at Pmin and lower than it at Pmax with a difference of 7 C. It can be explained by the fact that thermocouple which measures air temperature at x ¼ 14 cm is not at the good place to measure hot air after processors. Temperatures at x ¼ 29 and x ¼ 42 cm are more revealing to evaluate air temperature evolution through servers. The temperatures of the heat exchanger were also measured during the same time. We note that the air outlet temperature out (Tout HE;a ) and water outlet temperature (Tw ) were very close all the time. The maximal difference between these two temperatures was about 0.5 C. 3.2. Nodal models Fig. 6 shows the nodal representation of the rack by electrical analogy. Nodes and conductances correspond to temperatures and heat transfers by conduction and convection, respectively. Moreover, heat capacitances are represented with capacitances [30]. The airflow is modeled by directional fluid conductances. The value of airflow rate for a given inlet temperature is gotten thanks to the experimental results described in the next part. Input variables are the rack inlet temperature (TiR;a ), the climatic chamber temperature (Tamb ), and the power consumption of the a rack (Pmin or Pmax) as function of time. The total airflow through the rack is divided into two non-equal parts in order to model the airflow through servers (Q Sa ) and power supplies (Q PS a ). According to experimental results shown in paragraph 3.1, some physical points are selected to be the nodes of numerical models. Node 1 (TSa;1 ) and node 2 (TSa;2 ) correspond to the air temperature at x ¼ 29 cm and x ¼ 42 cm, respectively (Fig. 3). They represent the equivalent temperatures in the first part (from 0 to 29 cm) and the second part (from 29 to 42 cm) of servers. In addition, node 3 (TSc;1 ) and node 4 (TSc;2 ) represent the equivalent component temperatures in each part. The total power consumption of servers is divided into two fractions P and P’ and associated to node 3 and node 4. The convection/conduction heat transfers between nodes 1, 3 and 2, 4 are represented by heat conductances h13 and h24. Heat transfer between ambient air (Tamb ) and servers are also taken into a account.
Table 1 Sensors and evaluation of their measure uncertainties. Sensor
Accuracy (MV ¼ Measured Value)
Thermocouple type K Anemometer Water flow meter Electrical power
±0.1 C ±3% MV ±0.1 m/s ±2% MV ±0.5 L/min ±1.5% MV ±10 W
353
Fig. 4. Operating parameter variations.
354
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 5. Experimental temperature profiles as function of time a) inside servers, b) for heat exchanger.
Fig. 6. Nodal representation of rack model.
The heat transfer by convection through the power supplies is also modeled. The node TPS a corresponds to the outlet air temperature of the power supplies. The power dissipation P00 of power supplies represents the thermal power loses during AC/DC conversion. Power supply efficiency is defined thanks to power measurements and is evaluated around 90%. Node 6 corresponds to the air outlet temperature of the rack (Tout R;a ) where airflow from servers and power supplies join. In order to solve the model, the heat equation was solved for each node. Then this system is discretized using an implicit order scheme and the derivative was approximate by finite differences. Major equations of the rack model are given below:
. · S S a S in S dt ¼ h00 1 Tamb þ ma Cap TR;a Ca;1 $dTa;1 Ta;1 Ta;1 S S Ta;1 þ h31 Tc;1
consumption P of the first part of servers is also injected into equation (2).
. · S · PS out S out out dt ¼ ma Cap Ta;2 þ ma Cap TaPS TR;a Ca;4 $dTR;a TR;a
A nodal representation of the cross flow heat exchanger was also made (Fig. 7). Input parameters are the water inlet temperature · · (Tiw ), the water mass flow rate (m w ), the airflow rate (m a ) and the out air inlet temperature (TR;a ), which is calculated from the rack model (node 6). Two intermediary nodes were introduced, corresponding to the arithmetic mean of the inlet and the outlet measured temm perature of air (Tm HE;a ) and water (Tw ). Finally, node 10 and node 11 represent the air outlet temperature (ToHE;a ) and the water outlet temperature (Tow ).
(1)
with Ca,1 the heat capacitance of the air, ṁSa the airflow rate through servers. h00 1 and h31 are heat conductances to model heat transfers with ambient air and surface components.
. S S S dt¼ h13 Ta;1 þP C1 $dTc;1 Tc;1
(2)
with C1 the heat capacitance of components in the first part of servers. h13 corresponds to heat conductance h31 described previously and represents heat transfers with air. The equivalent power
(3)
Fig. 7. Nodal representation of heat exchanger.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
355
Heat balance on each allows defining a system of differential equations. The equation for the node 9 is given below:
· m m m in m þ mw C w Cw;1 $dTw Tw dt¼ h89 THE;a p Tw Tw
(4)
3.3. Identification procedure The objective was to estimate the value of the different heat conductances and heat capacity thanks to experimental measurements. The method of estimation known as the least squares estimation method was used [31]. The normalized cost function introduced is given by the following equation:
f ðC; hÞ ¼
1 Nmes
m N i2 mes h X X exp Tj ðpÞ Tjsim ðpÞ
(5)
j¼1 p¼1
where Nmes is the total number of time steps and m is the number of nodes. C ¼ [C1, C2, …] and h ¼ [h100 , h13, …] are the values of heat capacitance and heat conductances and are the unknown parameters to identify. The algorithm used consists in minimizing, for several sets of values x¼(C, h), the normalized cost function. This is a gradientbased method to find minimum of constrained nonlinear multivariable function [32]. 4. Results and discussion In the first part, cooling efficiency of the solution is discussed. Steady state measurements are described and studied. According to measurements, reproducibility was observed in spite of little temperature differences. To compare the results with traditional air cooling, additional measurements were also made with the rack without the heat exchanger by using the air conditioning system of the climatic chamber. In the second part, transient state measurements are presented. The results of the parameter estimation are given and the numerical models are validated using several sets of measurements. 4.1. Steady state experimental results 4.1.1. Airflow rate measurement As airflow rate depends on rack air inlet temperature, measurements have been done for air inlet temperature varying from 15 C to 45 C (Fig. 8). The uncertainty was evaluated at 5%. From 15 C to 25 C, the airflow rate is constant at 1,066 m3/h and increases from 25 C to 33 C to reach a value of 1,894 m3/h. The airflow rate is also constant from 33 C to 39 C, and from 39 C the airflow rate is maximal with a value near 2,020 m3/h. Airflow rate measurements have also shown that only 32% of the total airflow passes through the servers. We assume that the remaining airflow rate is equal to the airflow rate through the power supplies. Nevertheless, heat exchanger added a pressure drop resulting in 20% airflow rate decrease. At the end, these measurements allow to calculate the airflow rate through the rack and the heat exchanger whatever the rack inlet temperature. 4.1.2. Temperature measurements and effectiveness Fig. 9 shows the temperatures of components at Pmin and Pmax for different values of rack inlet temperature. The measurements were taken in steady state for 1 h. In each case, temperatures do not reach the maximal recommended operating temperatures.
Fig. 8. Evolution of airflow rate with air inlet temperature.
An increase of the temperatures between the two power modes was observed. On average, the surface temperatures of components rose 13.8 C for CPUs, 8.7 C for the chipset and the SAS, 12.6 C for the RAM and finally 7.7 C for the HDD. Compared with air cooling results, temperatures of components were 2.5 C higher on average with a maximum of 8.1 C for the SAS. This disparity can be explained by the airflow rate reduction described previously. In few cases, the temperatures of components when air inlet temperature was near 30 C, were lower than cases that had air inlet temperature near 25 C (for example, temperatures of SAS at Pmin). This phenomenon was due to the raise of the airflow rate. Furthermore, temperature variations through the servers show the power distribution along them (Fig. 10). Logically, we observe that power dissipation is more important at the beginning of the servers because of CPUs’ localization. Between x ¼ 0 and x ¼ 29 cm, the temperatures increased by 10 and 16.4 C on average at Pmin and Pmax, respectively. Then, the temperature evolution is lower between x ¼ 29 cm and x ¼ 42 cm with a raise of 3.1 and 4.3 C for each power dissipation. At x ¼ 42 cm, the temperatures values at Pmax are closest than at Pmin with a maximal difference of 2.5 C against 8.1 C at Pmin. Finally, fluid temperatures and effectiveness of the heat exchanger are summarized in Table 2. The inlet water temperatures are close to the water temperature set points fixed at 17, 22 and 27 C. The inlet air temperature of the heat exchanger is the same as the outlet temperature of the rack. These values increased normally at the higher power dissipation. Moreover, the outlet water temperatures and outlet air temperatures of the heat exchanger were very close. The average difference between these temperatures was 0.2 C and 0.4 C at Pmin and Pmax, respectively. The effectiveness of the heat exchanger is given by the following relationship:
. i o i i THE;a ε ¼ THE;a THE;a Tw
(6)
The average effectiveness decreased at the higher airflow rate and water inlet temperature. Note that for all experiments the water flow rate is unchanged and equal to 15.7 L/min. The variation in the effectiveness for close airflow rate is due to the measurement accuracy.
4.1.3. Power measurements and energy efficiency In order to evaluate the ratio of heat removed by the heat exchanger, heat balances were calculated at the rack and heat exchanger levels (Fig. 11). In addition, electrical power measurements of the blowers and the servers (with power supplies) were measured to compare electrical and heat balances. In steady state the heat gain in air (Fa) through the rack and the heat gain in water
356
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
Fig. 9. Component temperatures a) at Pmin, b) at Pmax.
Fig. 10. Evolution of T(x) for different values of Tin R;a and power dissipation.
Fig. 11. Electrical (dashed lines) and thermal (dotted lines) balance.
(Fw) through the heat exchanger are given by the following equations:
· out in Fa ¼ ma Cap TR;a TR;a
(7)
· out in Fw ¼ mw Cw p Tw Tw
(8)
Uncertainties were evaluated at 6% and 12% for the air-energy balance and the water-energy balance, respectively. Logically, the power consumption of the blowers rose with the rack inlet temperature like the airflow rate. At 20 C, the blowers used roughly 280 W against 880 W at 32.2 C. This represented an increase of 214% from the minimal value of power consumption. The power consumption of the servers and the power supplies was around 2,755 W at Pmin and reached to 4,150 W at Pmax. For each power dissipation, these values were stable whatever the rack inlet temperature. For rack inlet temperatures between 20.1 and 26.6 C, the air thermal balance was about 2771 and 4,103 W for Pmin and Pmax, Table 2 Fluid temperatures and effectiveness of heat exchanger. Pmin Tin ( C) ·R;a ma (kg/s) Tin w ( C) Tout w ( C) Tin HE;a ( C) Tout HE;a ( C) ε
20.1 0.356 17.6 20.0 27.8 20.0 0.76
Pmax 25.1 0.351 22.8 25.1 32.9 25.0 0.78
30.9 0.522 28.2 30.8 37.3 30.8 0.71
21.7 0.355 17.7 21.2 33.0 21.6 0.75
26.6 0.392 22.9 26.4 37.1 26.6 0.74
32.2 0.571 27.9 31.8 40.7 32.2 0.66
respectively. This represents roughly 100% of the electrical power consumption of the servers and the power supplies. The water thermal balance was around 2561 W at Pmin and 3809 W at Pmax for the same temperature range. These thermal balances corresponded to an average ratio of 92% of heat removed from the rack. However, at temperatures close to 30 C or more, this ratio was higher (more than 115% for air thermal balance and 105% for water thermal balance) but still within uncertainty ranges. The energy efficiency of the cooling system is also presented. We recall that the power consumption of the rack and the cooling system represents the total power of the system (PIT þ PCooling) and the power consumption of the cooling system is only given by those of pump and outdoor water to air heat exchanger. All of these values for the different cases presented in this part are summarized in Table 3. As the power consumption of the pump was always constant at 21 W, the power variations of the cooling system were due to the outdoor heat exchanger. Outside temperatures and the amount of heat removed explained these power variations. Finally, to evaluate the energy efficiency of the cooling system, we used a variant of the PUE, named pPUE (partial Power Usage Effectiveness) [33]. The pPUE has a close definition as PUE and in our case it focuses on the cooling energy consumption:
Table 3 Power consumption and energy efficiency of experimental bench. Pmin Tin R;a ( C) PIT (W) PCooling (W) pPUE
20.1 3,001 73 1.02
Pmax 25.1 3,047 65 1.02
30.9 3,510 56 1.02
21.7 4,365 101 1.02
26.6 4,497 84 1.02
32.2 5,094 69 1.01
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
pPUE ¼ ðrack energy þ cooling energyÞ=rack energy . PIT ¼ PIT þ Pcooling
(9)
The energy efficiency indicator of the cooling system is roughly 1.02. Compared with literature, this value is very low and satisfying despite low power load used. In addition, this study has given great results and a substantial database of measurements.
357
The values of the other conductances are assumed constant; one observes a short variation for high airflow rate. Fig. 12 gives the calculated heat conductances for the heat exchanger. These conductances were function of the total volumetric airflow rate and the water flow rate through the heat exchanger. A first order regression was made to express heat conductance linear dependency in term of these two operating parameters. For example, the relationship for the heat conductance h6,11 is given by the equation:
4.2. Parameter identification and validation of numerical model
h6;11 ðQa ; Qw Þz 4:27$105 Qa Qw þ 435Qa 86:7$103 Qw þ 81 4.2.1. Parameter identification In a first time, the parameter identification was done in steady state to find the heat conductances. Indeed, in steady state, the left part of equations (1)e(4) is equal to zero. The measurement data presented previously is used (Fig. 4). Fig. 12 shows the results for the server model. We observe that the heat conductance h13 depends on airflow rate through the rack and power dissipation. At Pmin, the value is near 150 W/K whereas at Pmax the value is roughly 250 W/K. Then, to model these dependencies, the following relations are retained:
h13 QaS z
(
185 QaS þ 134 ðPmin modeÞ 287 QaS þ 211 ðPmax modeÞ
(10)
(11) Once the heat conductances identified, the entire measurement series are exploited to estimate thermal capacities. After optimization, the calculated temperatures are close to the experimental data (Fig. 13). The corresponding heat capacitance values of each node for the rack and heat exchanger models are summarized in Table 4. For the rack, we observe that heat capacitances of nodes 1 and 3 are higher than those of other nodes (Table 5). These nodes corresponded to the physical points which were at the beginning of the servers. It can be explained by the fact that most of power dissipation occurs here. The rest of the heat capacitances were close.
Fig. 12. Calculation of heat conductances as function of airflow rate, water flow and power dissipation a) of the rack, b) of the heat exchanger.
Fig. 13. Model simulation on identification measurement series a) of rack model, b) of heat exchanger model.
358
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
capacitances from air side (Ca,5, Ca,6) were lower than those from water side (Cw,1, Cw,2).
Table 4 Heat capacitances of the rack model.
Thermal capacity (kJ K1)
Ca,1
Ca,2
C1
C2
Ca,3
Ca,4
Total
16.8
3.5
14.6
4.1
5.9
5.6
50.5
Table 5 Heat capacitances of the heat exchanger model.
Thermal capacity (kJ K1)
Ca,5
Cw,1
Ca,6
Cw,2
Total
43.5
47.1
56.4
61.9
208.9
Considering the total mass of the rack, the sum of all capacitances gave a value which was constant. Indeed, taking the heat capacity of steel (465 J/kg/K) and the supposed mass of the IT equipment (130 kg, from the equipment datasheet), we obtain a value near 60 kJ/K. All of the heat capacitances of the heat exchanger model had the same order of magnitude. Indeed, the values of the capacitances Ca,5 and Cw,1 were around 43.5 and 47.1 kJ/K whereas for the outlet temperature nodes, the values of Ca,6 and Cw,2 were 56.4 and 61.9 kJ/K. This disparity can be explained by the choice of the intermediate nodes defined as the arithmetic average of inlet and outlet temperatures which not correspond to the temperatures in the middle of the heat exchanger. We noticed that the heat
Fig. 14. Operating parameter variations for model validation.
4.2.2. Model validation The numerical models of rack and heat exchanger defined previously were tested on additional measurement data with variable variations of the operating input (Fig. 14). This step is important to valid numerical models of each system with other values of operating parameters. Air inlet temperature went from 24.7 to 27.8 C, the inlet water temperature was near 22 C whereas water flow rate was ranged from 10 to 14.7 L/min. Server power consumption varied during the tests. Fig. 15 shows the results for these operation parameters. We notice that the average deviation of temperatures between experiment and simulation is less than 7% for the rack model and around 4% for the heat exchanger model. The simulated temperatures are close to the experimental data but the rack outlet temperature (ToR,a) is overestimated (þ0.7 C) at Pmax and underestimated (0.6 C) at Pmin. Despite these disparities, the temperature predictions were acceptable throughout the tests. 5. Conclusion This paper presented an experimental and numerical study of air to water heat exchanger mounted on an electronic rack. The experimental part of the study was carried out with a substantial instrumentation at different scales of the system. In order to characterize thermal response of the installation, a monitoring of temperatures, electrical power, air and water flow were realized. The temperature of electronic component is maintained under their maximum operating temperatures. Air and water thermal balances showed that more 90% of the heat dissipated by the rack was removed by the heat exchanger. Promising results were also obtained with the estimation of the energy efficiency metric of the cooling system which had very low values. This was mainly due to the low power consumption of the pump and the outside heat exchanger. The partial energy efficiency indicator pPUE had very promising values close to 1.02, which is less than traditional air cooling. In parallel, thermal numerical models were defined at the scale of servers/rack and heat exchanger by using nodal approach. System of linear differential equations was written by heat balance equation for each node of the two proposed models. Thus, by using least squares estimation method and experimental data, the heat capacitances and heat transfer coefficients were evaluated for rack
Fig. 15. Model validations on additional measurement series a) of rack model, b) of heat exchanger model.
F. Douchet et al. / Applied Thermal Engineering 84 (2015) 350e359
and heat exchanger. Furthermore, complementary experimental measurements were used to validate the numerical models. The models were tested on two measurements series. The results were very satisfactory, with an error on temperature predictions less than 10%. These numerical models will be able to predict the temperatures of each system as function of fluid inlet temperatures, flow rates and power consumption of the electronic equipment. To estimate the impact of such cooling solution for a data center, they will be duplicated at a room scale. References [1] J.G. Koomey, Worldwide electricity used in data centers, Environ. Res. Lett. 3 (3) (Jul. 2008) 034008. [2] W. Van Heddeghem, S. Lambert, B. Lannoo, D. Colle, M. Pickavet, P. Demeester, Trends in worldwide ICT electricity consumption from 2007 to 2012, Comput. Commun. 50 (Sep. 2014) 64e76. [3] J.F. Karlsson, B. Moshfegh, Investigation of indoor climate and power usage in a data center, Energy Build. 37 (10) (Oct. 2005) 1075e1083. [4] S.V. Garimella, L.-T. Yeh, T. Persoons, Thermal management challenges in telecommunication systems and data centers, IEEE Trans. Compon. Packag. Manuf. Technol. 2 (8) (Aug. 2012) 1307e1316. [5] G.A. Brady, N. Kapur, J.L. Summers, H.M. Thompson, A case study and critical assessment in calculating power usage effectiveness for a data centre, Energy Convers. Manag. 76 (Dec. 2013) 155e161. [6] J. Cho, T. Lim, B.S. Kim, Viability of datacenter cooling systems for energy efficiency in temperate or subtropical regions: case study, Energy Build. 55 (Dec. 2012) 189e197. [7] K. Kant, Data center evolution: a tutorial on state of the art, issues, and challenges, Comput. Netw. 53 (17) (Dec. 2009) 2939e2965. [8] ASHRAE TC 9.9, 2011 Thermal Guidelines for Data Processing Environments e Expanded Data Center Classes and Usage Guidance, ASHRAE, 2011. [9] S.-Y. Jing, S. Ali, K. She, Y. Zhong, State-of-the-art research study for green cloud computing, J. Supercomput 65 (1) (Jul. 2013) 445e468. [10] B. Fakhim, M. Behnia, S.W. Armfield, N. Srinarayana, Cooling solutions in an operational data centre: a case study, Appl. Therm. Eng. 31 (14e15) (Oct. 2011) 2279e2291. [11] Y. Fulpagare, G. Mahamuni, A. Bhargav, Effect of plenum chamber obstructions on data center performance, Appl. Therm. Eng. 80 (Apr. 2015) 187e195. [12] K.-P. Lee, H.-L. Chen, Analysis of energy saving potential of air-side free cooling for data centers in worldwide climate zones, Energy Build. 64 (Sep. 2013) 103e112. [13] T. Lu, X. Lü, M. Remes, M. Viljanen, Investigation of air management and energy performance in a data center in Finland: case study, Energy Build. 43 (12) (Dec. 2011) 3360e3372. [14] H. Endo, H. Kodama, H. Fukuda, T. Sugimoto, T. Horie, M. Kondo, Effect of climatic conditions on energy consumption in direct fresh-air container data centers, in: Green Computing Conference (IGCC), 2013 International, 2013, pp. 1e10. [15] H. Zhang, S. Shao, H. Xu, H. Zou, C. Tian, Integrated system of mechanical refrigeration and thermosyphon for free cooling of data centers, Appl. Therm. Eng. 75 (Jan. 2015) 185e192.
359
[16] S.-W. Ham, J.-S. Park, J.-W. Jeong, Optimum supply air temperature ranges of various air-side economizers in a modular data center, Appl. Therm. Eng. 77 (Feb. 2015) 163e179. [17] R. Udakeri, V. Mulay, D. Agonafer, “Comparison of overhead supply and underfloor supply with rear heat exchanger in high density data center clusters, in: Twenty-fourth Annual IEEE Semiconductor Thermal Measurement and Management Symposium, 2008. Semi-Therm 2008, 2008, pp. 165e172. [18] R. Schmidt, R.C. Chu, M. Ellsworth, M. Iyengar, D. Porter, V. Kamath, B. Lehman, Maintaining Datacom Rack Inlet Air Temperatures with Water Cooled Heat Exchanger, Jan. 2005, pp. 663e673. [19] A. Almoli, A. Thompson, N. Kapur, J. Summers, H. Thompson, G. Hannah, Computational fluid dynamic investigation of liquid rack cooling in data centres, Appl. Energy 89 (1) (Jan. 2012) 150e155. [20] K. Karki, S. Novotny, A. Radmehr, S. Patankar, Use of passive, rear-door heat exchangers to cool low to moderate heat loads, ASHRAE Trans. 117 (2) (2011). [21] M.A. Abd El-Baky, M.M. Mohamed, “Heat pipe heat exchanger for heat recovery in air conditioning, Appl. Therm. Eng. 27 (4) (Mar. 2007) 795e801. [22] V. Pandiyarajan, M. Chinna Pandian, E. Malan, R. Velraj, R.V. Seeniraj, Experimental investigation on heat recovery from diesel engine exhaust using finned shell and tube heat exchanger and thermal storage system, Appl. Energy 88 (1) (Jan. 2011) 77e87. [23] K. Nemati, B.T. Murray, B. Sammakia, Experimental characterization and modeling of a water-cooled server cabinet, in: 2014 IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2014, pp. 723e728. [24] J. Zachary Woodruff, P. Brenner, A.P.C. Buccellato, D.B. Go, Environmentally opportunistic computing: a distributed waste heat reutilization approach to energy-efficient buildings and data centers, Energy Build. 69 (Feb. 2014) 41e50. [25] T. Brunschwiler, B. Smith, E. Ruetsche, B. Michel, Toward zero-emission data centers through direct reuse of thermal energy, IBM J. Res. Dev. 53 (3) (May 2009) 11, 1e11:13. [26] K. Ebrahimi, G.F. Jones, A.S. Fleischer, A review of data center cooling technology, operating conditions and the corresponding low-grade waste heat recovery opportunities, Renew. Sustain. Energy Rev. 31 (Mar. 2014) 622e638. [27] S. Zimmermann, I. Meijer, M.K. Tiwari, S. Paredes, B. Michel, D. Poulikakos, Aquasar: a hot water cooled data center with direct energy reuse, Energy 43 (1) (Jul. 2012) 237e245. [28] S. Zimmermann, M.K. Tiwari, I. Meijer, S. Paredes, B. Michel, D. Poulikakos, Hot water cooled electronics: exergy analysis and waste heat reuse feasibility, Int. J. Heat Mass Transf. 55 (23e24) (Nov. 2012) 6391e6399. [29] Z. Song, B.T. Murray, B. Sammakia, A dynamic compact thermal model for data center analysis and control using the zonal method and artificial neural networks, Appl. Therm. Eng. 62 (1) (Jan. 2014) 48e57. [30] W.H. Tang, Q.H. Wu, Z.J. Richardson, Equivalent heat circuit based power transformer thermal model, Electr. Power Appl. IEE Proc. 149 (2) (Mar. 2002) 87e92. [31] J.V. Beck, K.J. Arnold, Parameter Estimation in Engineering and Science, Wiley, , New York, NY, 1977. [32] R.H. Byrd, J.C. Gilbert, J. Nocedal, A trust region method based on interior point techniques for nonlinear programming, Math. Program 89 (1) (Nov. 2000) 149e185. [33] The Green Grid, PUETM: a Comprehensive Examination of the Metric, White Paper #49, 2012.