Energy saving evaluation of a novel energy system based on spray cooling for supercomputer center

Accepted Manuscript Energy saving evaluation of a novel energy system based on spray cooling for supercomputer center

Hua Chen, Wen-long Cheng, Wei-wei Zhang, Yu-hang Peng, Li-jia Jiang PII:

S0360-5442(17)31608-0

DOI:

10.1016/j.energy.2017.09.089

Reference:

EGY 11583

To appear in:

Energy

Received Date:

16 December 2016

Revised Date:

30 August 2017

Accepted Date:

19 September 2017

Please cite this article as: Hua Chen, Wen-long Cheng, Wei-wei Zhang, Yu-hang Peng, Li-jia Jiang, Energy saving evaluation of a novel energy system based on spray cooling for supercomputer center, Energy (2017), doi: 10.1016/j.energy.2017.09.089

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT Highlights: 

A novel energy system based on spray cooling for supercomputer center is proposed.



The system integrates spray cooling with waste heat driven absorption chiller.



Modeling of integrated energy system under different working conditions.



Inlet temperature strongly affects cooling capacity and optimum temperature exists.



The system can achieve energy saving of 49% and PUE within best practice scenario.

ACCEPTED MANUSCRIPT

1

Energy saving evaluation of a novel energy system based on

2

spray cooling for supercomputer center

3

Hua Chen, Wen-long Cheng, Wei-wei Zhang, Yu-hang Peng, Li-jia Jiang

4

Department of Thermal Science and Energy Engineering, University of Science and

5

Technology of China, Hefei, Anhui 230027, PR China

6 7

Abstract: To improve the energy efficiency of supercomputer center, a novel energy

8

system aimed at enhancing cooling efficiency while reusing waste heat is proposed.

9

The energy system integrates a plug-type spray cooling system with a two-stage

10

absorption chiller driven by spray cooling waste heat. Overall modeling of integrated

11

energy system is analyzed based on spray cooling model and absorption chiller

12

model. Energy saving evaluation is conducted based on Dawning 5000A

13

supercomputer in China. It is found that the novel energy system is much efficient

14

than the original energy system in all seasons. The energy saving effect is highly

15

affected by inlet temperature of spray cooling. With the increase of inlet temperature,

16

the spray cooling capacity decreases while the absorption cooling capacity increases.

17

Thus, an optimal inlet temperature of 55℃ is obtained at which the lowest cooling

18

power consumption, lowest power utilization effectiveness (PUE) and highest energy

19

saving efficiency (ESE) can be achieved. Taken Dawning 5000A supercomputer for

20

example, the system can achieve ESE as high as 49% and PUE within best practice

21

scenario of 1.44. At the optimal design, cooling power consumption only accounts for

22

16%. Power consumption devoted to running the IT equipment is improved from 60%

23

to 67%.

24

Keywords: Energy saving; Supercomputer; Spray cooling; Multi-nozzle array;

25

Absorption chiller; Energy efficiency. 

Corresponding author. Tel./Fax: +86 551 63600305. E-mail address: [email protected] 1

ACCEPTED MANUSCRIPT 26 27

Nomenclature

28

C

specific heat, kJ/(kg K)

29

COP

coefficient of performance

30

CLF

cooling load factor

31

CPP

cost payback period, year

32

G

mass flux of spray droplets, kg/(m2s)

33

h

enthalpies, kJ/kg

34

HEG

heat exchanger in generator

35

HEE

heat exchanger in evaporator

36

HP

high pressure

37

HPA

high pressure absorber

38

HPG

high pressure generator

39

HPHE

high pressure solution heat exchanger

40

LP

low pressure

41

LPA

low pressure absorber

42

LPG

low pressure generator

43

LPHE

low pressure solution heat exchanger

44

m

mass flow rate, kg/s

45

Nu

Nusselt number

46

Pr

Prantle number

47

PUE

power usage effectiveness

48

Q

heat flow, kW

49

Re

Reynolds number

50

T

temperature, oC

51

We

Webber number

52

W

53

x

power consumption, kW mass concentration of LiBr solution 2

ACCEPTED MANUSCRIPT 54 55

Greek letters

56

λ

57

ρ

density, kg/m3

58



dimensionless temperature

heat conductivity, W/(m K)

59 60

Subscrips

61

A

62

ac

air conditioner

63

chw

chilled water

64

cool

cooling system

65

cw

cooling water

66

C

condenser

67

drop

droplet

68

E

evaporator

69

film

water film on heating surface

70

IT

IT equipment

71

in

inlet

72

out

outlet

73

S

strong solution

74

sc

spray cooling

75

sat

saturation

76

tower cooling tower

77

W

weak solution

78

wc

water-cooled cabinet

absorber

79 80 81

1. Introduction The increasing demand for high-performance computing has resulted in a 3

ACCEPTED MANUSCRIPT 82

dramatic increase of energy consumption as well as high heat flux dissipation density

83

in supercomputer center. It was estimated that data centers are responsible for nearly

84

2% of the worldwide total electricity usage [1]. In 2009, China's data center consumed

85

around 36 billion kWh, which accounts for 1% of China’s total electricity

86

consumption. In 2011, electricity consumption of China's data center has increased to

87

50 billion kWh [2]. Almost 40% of this energy consumption is spent to operate the

88

cooling systems [3], resulting in a significant energy waste and making it very

89

important to minimize the energy consumption of the cooling system. Consequently, a

90

more rational energy management and a more efficient cooling method are in urgent

91

requirement.

92

Generally, liquid cooling can provide higher cooling performance than air

93

cooling [4,5]. Single phase liquid cooling such as forced convection [6], microchannel

94

[7], jet impingement [8] and spray cooling [9-11] has become an unavoidable and

95

effective alternative to dissipate high heat flux density of electronic components.

96

Compared with other liquid cooling methods, spray cooling has numerous advantages

97

such as high heat flux removal capacity, low coolant mass flux, low temperature

98

difference, and so on [9-11]. Electronic components showed higher reliability and

99

lower temperature after using spray cooling technology. At present, spray cooling has

100

been applied in the Cray X1 supercomputer successfully [12]. Shedd et al. [13,14]

101

conducted experiments to cool a large area of 70mm×70mm and studied the cooling

102

characteristics of various nozzle array arrangements. Cheng et al. [15] established

103

integrated spray cooling system in a compact spray chamber to study the cooling

104

performance of larger area surface. Zhang et al. [16,17] designed a novel plug-chip

105

spray cooling enclosure to cool the integrated electronic system and studied the

106

effects of inclination angle and flow rate on spray cooling performance.

107

Usually, spray cooling system operates at two modes: open-loop and closed-

108

loop. In open-loop operation, cooling water is supplied continually and the waste

109

water is discharged directly, which not only leads to a huge requirement of water 4

ACCEPTED MANUSCRIPT 110

consumption but also results in a significant amount of energy waste. In closed-loop

111

operation, cooling water is recirculated in the system, thus the above problems can be

112

avoided. However, in order to cool the recycle water, additional cooling devices such

113

as cooling towers and condensers are needed, which will lead to unnecessary energy

114

consumption and result in low energy efficiency.

115

To solve above problems, absorption chiller is adopted in this paper to reuse and

116

cool the recycle water of spray cooling system. Many researchers have proved the

117

economic potential and energy efficiency of waste heat driven absorption chiller [18-

118

20]. Regarding to the maximum allowable junction temperature of electronic

119

components, which is no more than 85℃ in most cases [21], the highest spray

120

cooling outlet water temperature is no more than 85℃. Since NH3/H2O absorption

121

system requires generator temperature in the range of 95 to 170℃ [22], NH3/H2O

122

working pair is not applicable here. However, this low temperature waste heat is

123

suitable for LiBr/H2O system, which operates at generator temperature in the range of

124

70 to 95℃ [22]. Thus, to better reuse this low temperature waste heat, two-stage

125

LiBr/H2O absorption chiller was proposed [23-25]. Sumathy et al. [23] found that

126

two-stage chiller could achieve slightly higher COP than single-stage system with a

127

cost reduction of about 50% at low heat source temperature of 80℃. What’s more,

128

usable temperature drop is much larger than the single-stage system. Many studies

129

have focused on two-stage absorption chiller [23-27], and most of them were applied

130

in solar collector. However, the application in data center is rare to be seen.

131

In summary, spray cooling is an efficient and powerful cooling method among

132

the existing technologies. However, the additional chillers for treating the recycle

133

water will lead to unnecessary energy consumption. Hence, in order to further

134

improve the energy efficiency of supercomputer center, a novel energy system based

135

on spray cooling system integrated with waste heat reusing absorption chiller is

136

proposed in this paper. To evaluate the energy saving effect of novel energy system,

137

power consumption and cooling arrangement of Dawning 5000A supercomputer is 5

ACCEPTED MANUSCRIPT 138

used to offer data reference. Overall modeling of novel energy system under different

139

working conditions is conducted. The effect of inlet temperature on energy saving

140

efficiency, power usage effectiveness and cooling load factor is analyzed.

141 142

2.

Integrated energy system design

143

The primary objective of this study is to improve the energy efficiency of

144

supercomputer by reducing the power consumption of cooling facilities. To achieve

145

this objective, a novel energy system based on plug-type spray cooling system

146

integrated with two-stage absorption chiller which is driven by the waste heat

147

discharged from spray cooling is proposed. For comparison, the actual operational

148

results of the original energy system in Dawning 5000A supercomputer are collected

149

from reference [28-31]. Dawning 5000A supercomputer in China is one of the top

150

supercomputer in Asia [28,29]. The details of rack numbers and cooling methods in

151

Dawning 5000A supercomputer are summarized in Table 1. There are 65 racks in the

152

Dawning 5000A system, including 42 compute racks, 10 I/O and communication

153

racks and 13 storage racks. The maximum heat load of one compute rack is 23 kW.

154

The maximum heat load of I/O and communication racks and storage racks are 45 kW

155

and 70 kW, receptivity. In total, the maximum power load of Dawning 5000A is 1081

156

kW, and the rated power load is about 800 kW. The power consumption of different

157

subsystem, including IT equipment, power supply loss, cooling system and other

158

equipment are summarized in Table 2.

159

The cooling method of original energy system in Dawning 5000A supercomputer

160

is using water-cooled cabinets supplement with air conditioners [30,31]. The water-

161

cooled cabinets are used to cool compute racks and the air conditioners are used to

162

cool the remaining racks. The water-cooled cabinet is an enclosed cabinet with a

163

closed air circulation system and a heat exchanger in it. Each compute rack is placed

164

in one water-cooled cabinet. The cooling mechanism of water-cooled cabinet is using

165

chilled water to reduce the air temperature in the enclosed cabinet and then the 6

ACCEPTED MANUSCRIPT 166

electronic components in the compute rack is cooled by air convection heat transfer.

167

Similarly, the remaining racks are cooled by air conditioners through air convection

168

heat transfer. However, the poor thermal properties of air would result in a low heat

169

transfer of air convection, a large temperature rise of air flow and a large temperature

170

gradient within the rack [3]. What’s more, the temperature of some locations can

171

reach quite a high value due to localized hot spots. Thus, the supply temperature of

172

the original system should be unnecessarily low to maintain the maximum local

173

temperature of the electronic components within the temperature limit. This will lead

174

to unnecessary energy consumption and result in low energy efficiency.

175

To improve the energy efficiency of supercomputer, the advantages of novel

176

energy system against the original energy system are summarized as follows:

177

(1) The application of plug-type spray cooling system in replacement of water-cooled

178

cabinet enables an appreciable reduction of water consumption and cooling power

179

consumption. Firstly, spray cooling could achieve high heat flux removal capacity

180

with low coolant mass flux and low temperature differences. The cold plates of

181

spray cooling system are directly mounted on the heating surface of electronic

182

components. They are in direct contact with the heating devices which could offer

183

superior performance compared to air convection. Secondly, spray cooling is able

184

to provide high cooling capacity at relatively high inlet water temperature, which

185

eliminates the use of chilled water, thus reducing the power consumption. Finally,

186

cold pates are mounted separately on different heating devices, which could

187

achieve a thermal control at individual cold plate levels. This will eliminate the

188

unnecessary overcooling in some locations and reduce the power consumption.

189

(2) The reuse of waste heat produced by spray cooling system not only improves the

190

energy utilization efficiency, but also eliminates the need of additional cooling

191

devices to cool the recycle coolant in spray cooling cycle, which in turn reduces

192

the system energy requirement.

193

(3) The integration of absorption chiller driven by spray cooling waste heat could 7

ACCEPTED MANUSCRIPT 194

produce extra cooling capacity. Thus it could replace or supplement the air

195

conditioning system and further reduce the heat load of air conditioner.

196 197

2.1 Integrated energy system description

198

As shown in Fig. 1, the novel energy system is composed of a plug-type spray

199

cooling system and a waste heat reusing absorption chiller. The former spray cooling

200

system is utilized to replace the water-cooled cabinet and cool computer racks. The

201

latter absorption chiller is used to not only reuse and cool the recycle water in spray

202

cooling system but also partially eliminate the heat load of the air conditioner. The

203

whole cooling process contains four main parts: spray cooling cycle to cool computer

204

rack, heat exchanger in generator (HEG) to reuse waste heat, absorption cooling cycle

205

driven by waste heat, heat exchanger in evaporator (HEE) to supplement air

206

conditioner.

207

Spray cooling system is applied in cooling computer racks by installing many

208

cold plates at the heating surface of each electronic component. The cold plates are

209

cooled by cold water sprayed out by nozzles. After spray cooling process, the cold

210

water becomes hot waste water. Then, the hot water is circulated to HEG to drive the

211

absorption chiller. After heat exchanging process, the cooled water return to spray

212

cooling system and continue to cool computer racks. In absorption cooling cycle,

213

water is used as refrigerant and lithium bromide is used as absorbent. Water vapor

214

generated in generator flows to the condenser and provides cooling effect in the

215

evaporator. The cooling capacity is taken away by chilled water through HEE.

216

Finally, the chilled water is circulated to the air conditioning facilities to supplement

217

air conditioner. Through using spray cooling in computer racks along with using

218

chilled water produced by absorption chiller, the heat load of air conditioner could be

219

reduced a lot or even be totally eliminated, resulting in considerable energy savings.

220 221

2.2 Plug-type spray cooling system 8

ACCEPTED MANUSCRIPT 222

As shown in Fig. 2, the plug-type spray cooling system is composed of many

223

plug-type spray chambers (only three are illustrated in Fig. 2 to simplify the cooling

224

system), a cooling water tank, a liquid pump, liquid delivery and return pipes, valves

225

and other accessories. The plug-type spray chamber is designed to install on different

226

electronic components easily and quickly. It is comprised of cold plate, plug pins,

227

multi-nozzle array and liquid distribution chamber. Cold plates of spray chambers

228

could be mounted easily and quickly on the heating surface of electronic components

229

through plug pins. Heat flux generated in electronic components are removed via cold

230

plates. To achieve this purpose, the cold plate is mounted as close as possible on the

231

heating surface and is made by high thermal conductivity copper. Each spray chamber

232

is connected to the cooling water tank through liquid delivery pipe. The spray flow

233

rate could be adjusted by the flow control value according to different cooling

234

requirements. The multi-nozzle array is composed of numerous small pressure-swirl

235

nozzles assembled uniformly in a thin plate. The spray chamber is mainly designed to

236

distribute water evenly on to the cold plate. In order to make the flow of each nozzle

237

even and to discharge waste water conveniently, the water inlet is arranged at the top

238

of spray chamber and the water outlet is arranged at the bottom of spray chamber

239

respectively.

240

During cooling process, working fluid stored in cooling water tank is pumped

241

into the liquid delivery pipe and evenly distributed to the inlet of each spray chamber.

242

In each spray chamber, water is sprayed out by multi-nozzle array and atomized into

243

tiny droplets with high velocity. Then, the high velocity droplets impinge on the cold

244

plate, remove large amount of heat and become hot waste water. Finally, the hot

245

waste water accumulates at the bottom of spray chamber and is discharged into liquid

246

return pipes through the water outlet. In order to reuse this low-grade heat of waste

247

water, the main liquid return pipe is connected to the generator of absorption chiller.

248

The hot waste water is reused as the heat source of the generator to drive the

249

absorption chiller. After heat exchange process in generator, the cooled water is 9

ACCEPTED MANUSCRIPT 250

recirculated to the cooling water tank and completes the full cooling cycle.

251 252

2.3 Two-stage absorption chiller

253

Two-stage LiBr/H2O absorption chiller is very efficient to reuse and cool the

254

recycle water of spray cooling system. As shown in Fig. 3, it is comprised of a

255

refrigerant cycle and two stages of solution cycles, namely, a high pressure (HP) stage

256

and a low pressure (LP) stage. The HP stage consists of a HP generator, a HP solution

257

heat exchanger (HPHE) and a HP absorber. Similarity, the LP stage consists of a LP

258

generator, a LP solution heat exchanger (LPHE) and a LP absorber.

259

In solution cycles, water is used as refrigerant and lithium bromide is used as the

260

absorbent. Waste water from the spray cooling system is supplied as the heat source

261

for HP and LP generators to generate water vapor from the weak solution. After the

262

evaporation of water, the remaining LiBr/H2O strong solution is precooled in the

263

solution heat exchanger before flowing back to the absorber. The water vapor

264

generated in HP generator flows to the condenser and provides cooling effect in the

265

evaporator. However, the water vapor generated in LP generator is circulated to the

266

HP absorber, absorbed by the strong solution coming from HP generator and does not

267

provide cooling effect. Then the weak solution in HP absorber is reformed and

268

pumped back to HP generator.

269

In refrigerant cycle, water vapor generated in HP generator is condensed in the

270

condenser and the condensation latent heat is removed by cooling tower. Then, the

271

condensate water coming from condenser is circulated to the evaporator through an

272

expansion valve. In the evaporator, the water evaporates and provides cooling effect

273

which is taken away by chilled water. Finally, the water vapor is recycled to the LP

274

absorber and is absorbed by the strong solution coming from the LP generator. Thus,

275

the entire closed cooling cycle is completed.

276

10

ACCEPTED MANUSCRIPT 277

3. Thermodynamic analysis of integrated energy system

278

3.1 Performance analysis of spray cooling

279

Restricted by temperature limits of electronic components, water spray cooling is

280

dominated by single-phase heat transfer. In this non-boiling regime, spray cooling

281

system performs with high operation stability and uniform heat flux distribution

282

although with limited heat dissipation capability. This heat transfer mode is suitable

283

for the thermal control of electronic components in supercomputer center which is

284

very sensitive to temperature. In order to apply this effective spray cooling system in

285

supercomputer center, experimental results and mathematical model of our previous

286

works [15-17,33-35] are adopted and briefly summarized in Appendix A and B

287

respectively. Based on these researches, spray cooling performance is analyzed.

288

To evaluate the cooling performance of spray cooling system, equivalent COP

289

(Coefficient of performance) of spray cooling (COPsc) is defined as the ratio of spray

290

cooling heat removal capacity (Qspray) to total electrical power of spray cooling system

291

(Wspray) as follows:

292

COPsc  Qspray Wspray

293

A higher COPsc means higher cooling capacities with lower operating costs. Qspray can

294

be calculated based on our previous model presented in reference [33-35]. It is highly

295

dependent on working conditions, especially required thermal control temperature,

296

inlet water temperature and spray flow rate. Wspray is mainly composed of power to

297

run the water pump and power to cool recycle coolant in spray cooling cycle. In novel

298

integrated energy system, the power to cool recycle coolant is eliminated by reusing

299

the spray cooling waste heat to drive an absorption chiller. So the Wspray is equal to the

300

pump power and can be estimated based on our previous experimental results

301

presented in Appendix A.

(1)

302

For comparison, the equivalent COP of water-cooled cabinet in original

303

Dawning 5000A supercomputer (COPwc) is calculated as the ratio of water-cooled

304

cabinet cooling capacity to total power consumption of water-cooled cabinet as 11

ACCEPTED MANUSCRIPT 305

follows:

306

COPwc  Qwc Wwc

(2)

307 308

3.2 Mathematical model of two-stage absorption chiller

309

To analyze the cooling performance of absorption chiller, a mathematical model

310

is established based on conservation and balance equations. The basic cycle process

311

and P-T-X diagram are shown in Fig. 4. The two-stage absorption chiller is mainly

312

composed of two generators, two absorbers, a condenser and an evaporator. To

313

simplify the model, two solution heat exchangers and pumps between generator and

314

absorber are not considered. The following assumptions are made to establish the

315

mathematical model:

316

(1) The system is in steady state. Refrigerant in generator and absorber is in

317 318 319

equilibrium state. (2) The outlet conditions of LiBr aqueous solution and refrigerant at each component are in saturation states.

320

(3) The outlet temperatures of HPG and LPG are 5℃ lower than the heat source inlet

321

temperatures. The outlet temperatures of HPA and LABS are 4℃ higher than the

322

cooling water temperature.

323 324

(4) The heat loss and pressure drop in all components are negligible. Based on above assumptions, the mass and energy equations of each component

325

are obtained as follows:

326

m

327

x

328

Q   mout hout   min hin

329

The energy equations of heat exchangers in each component are obtained as follows:

330

Q  mC (Tin  Tout )

331

The state equations of LiBr aqueous solution and saturation water are obtained as

in

in

  mout

(3)

min   xout mout

(4) (5)

(6)

12

ACCEPTED MANUSCRIPT 332

follows:

333

f LiBr  P, T , x   0

(7)

334

f LiBr  h, T , x   0

(8)

335

f water  P, T   0

(9)

336

f water  h, T   0

(10)

337

where, m, x, p, T, h are the mass flow rates, mass concentrations, pressures,

338

temperatures and enthalpies of working fluids, respectively. The enthalpies can be

339

calculated by the corresponding temperature, concentration, and pressure according to

340

correlations given in literatures [36,37].

341

Thermal COP (COPthernal) is defined as the ratio between heat flow in evaporator

342

and heat flow in generator. It is used to evaluate the utilization efficiency of heat

343

source as follows:

344

COPthermal  QE （） QLPG  QHPG

345

Electrical COP (COPelectric) is defined as the ratio between the cooling capacity and

346

the power consumption of absorption chiller. It is used to evaluate the energy

347

efficiency of absorption cooling system. The power consumptions of absorption

348

chiller is composed of solution pump power consumption and cooling tower power

349

consumption. The power consumption of solution pump is negligible compares to

350

other components[38]. However, the electric fan and water pump equipped in cooling

351

tower cannot be neglected. So the COPelectric can be calculated as follows:

352

COPelectric  QE Wtower

353

where, Wtower can be calculated according to the COP of cooling tower [39]. The

354

cooling demand of cooling tower (Qtower) is calculated by the sum of heat flow in

355

absorber and condenser.

356

Qtower  QC  QA

(11)

(12)

(13)

357 13

ACCEPTED MANUSCRIPT 358

3.3 Comprehensive performance analysis

359

Overall simulation of novel integrated energy system is conducted based on

360

spray cooling model and absorption chiller model. The spray cooling inlet

361

temperature has a great effect on the cooling capacity and outlet temperature of spray

362

cooling system. The outlet water of spray cooling system is reused to drive the

363

absorption chiller, so the spray cooling characteristics will influence the absorption

364

cooling performance. As the water cooled by absorption chiller is recirculated to

365

spray cooling system, the absorption chiller will in turn influence the spray cooling

366

performance.

367

In this study, the focus is to minimize the power consumption of cooling system

368

(Wcool) and increase the energy efficiency of Dawning 5000A supercomputer. To

369

better present the energy efficiency improvement of the novel system, the evaluation

370

indicators such as energy saving efficiency (ESE), power usage effectiveness (PUE)

371

and cooling load factor (CLF) are introduced.

372

In order to understand the energy saving effect of novel energy system, energy

373

saving efficiency (ESE) is defined as the ratio between power consumption of novel

374

energy system and original energy system as follows:

375

ESE  1 

Wcool of new cooling system Wcool of original cooling system

(14)

376

The most widely accepted evaluation standard is PUE. It is defined as the ratio

377

between total power consumption of supercomputer and the power required to run IT

378

equipment. PUE is used to evaluate the energy efficiency of a supercomputer and can

379

be calculated as follows:

380

PUE 

381

where, Wtotal is composed of IT equipment power consumption (WIT), power supply

382

loss of power distribution unit (Wloss), power used to operate the cooling system

383

(Wcool) and power consumption of other assistant equipments (Welse).

Total electrical power consumption Wtotal  IT equipment power consumption WIT

14

(15)

ACCEPTED MANUSCRIPT 384

Wtotal  WIT  Wloss  Wcool  Welse

385

PUE represents the fraction of total electrical power consumption that devoted to

386

running IT equipment. Lower PUE value means higher energy efficiency because it

387

means more energy consumed in supercomputer is actually devoted to computing IT

388

equipment. According to the reports of Environmental Protection Agency [40],

389

current data centers have an average PUE of 2.0. By using alternative efficiency

390

scenario assumptions, lower PUE value could be achieved bellow 1.7 in the Improved

391

Operation Scenario and bellow 1.5 in the Best Practice Scenario. To achieve lower

392

PUE, efforts should be made to maximize the power devoted to IT equipment such as

393

using more efficient microprocessors, servers and storage devices, and to minimize

394

the power consumption of ancillary infrastructures such as using more efficient power

395

distribution systems and cooling systems.

396

(16)

Cooling Load Factor (CLF) is defined as the ratio between power used to operate

397

the cooling system and power required to run IT equipment as follows:

398

CLF 

399

CLF represents the power consumption devoted to cool one unit IT load. Thus, the

400

lower the CLF is, the more efficient is the cooling system.

Cooling system power consumption Wcool  IT equipment power consumption WIT

(17)

401

In the novel integrated energy system, Wcool is mainly composed of power

402

consumption caused by spray cooling system (Wsc), air conditioner (Wac), and cooling

403

tower in absorption chiller (Wtower).

404

Wcool  Wsc  Wac  Wtower

405

where, Wsc can be calculated based on the heat removal requirement of spray cooling

406

system in supercomputer (Qsc) and the equivalent COP of spray cooling system

407

(COPsc) as follows:

408

Wsc  Qsc COPsc

409

where, Qsc is equal to the heat load of computer racks because the spray cooling

(18)

(19)

15

ACCEPTED MANUSCRIPT 410

system is used to replace the water-cooled cabinet and cool computer racks. The heat

411

loads of Dawning 5000A supercomputer are summarized in Table 1. COPsc is

412

calculated based on our previous experimental and numerical results as explained in

413

section 3.1.

414

In novel integrated cooling system, Wac is much lower than that in original

415

energy system, since the total heat load of air conditioner is reduced by the

416

supplemental absorption chiller. So Wac can be calculated as follows:

417

Wac  (Qac  Qev ) COPac

418

(20)

Wtower is the power consumption of cooling tower in absorption chiller. It can be

419

calculated according to the COP of cooling tower [38] as follows:

420

Wtower  Qtower COPtower

421

where, the cooling demand of cooling tower (Qtower) is calculated by Eq.13. It is the

422

sum of heat flow in absorber and condenser as analyzed in section 3.2.

(21)

423 424

4. Results and discussion

425

4.1 Spray cooling performance

426

As inlet water temperature is the key parameter which affects spray cooling

427

performance, the effect of inlet water temperature on spray cooling outlet water

428

temperature, heat removal capacity and COPsc is studied. Regarding to the

429

temperature limits (85℃ in most cases) of electronic components, the thermal control

430

temperature (maximum temperature on the heating surface) of spray cooling system is

431

set at 85℃ and a lower thermal control temperature of 80℃ is also simulated for

432

comparison. The inlet water temperature is in the range of 40 to 80℃. Through

433

simulation, the outlet water temperature range of 75-83.5℃ is obtained at thermal

434

control temperature of 85℃ as shown in Fig. 5. This temperature range is suitable for

435

LiBr/H2O absorption chiller, which operates at generator temperature in the range of

436

70 to 95℃[22]. The heat removal capacity results at different thermal control

437

temperatures are also shown in Fig. 5. It is found that the inlet water temperature has a 16

ACCEPTED MANUSCRIPT 438

great influence on spray cooling heat transfer performance. With the increase of inlet

439

water temperature, the outlet water temperature increases almost proportionally and

440

finally approaches close to the thermal control temperature. However, the heat

441

removal capacity decreases proportionally with the increase of inlet water

442

temperature. Since the inlet water temperature is equivalent to the outlet temperature

443

of HEG in absorption chiller, the cooling performance of spray cooling system is

444

highly dependant on the performance of absorption chiller. Meanwhile, the outlet

445

water of spray cooling system is the heat source of absorption chiller. Thus, the

446

performance of spray cooling system will influence the cooling performance of

447

absorption chiller as well. Moreover, higher thermal control temperature results in

448

higher heat removal capacity and higher outlet water temperature, which is beneficial

449

to not only the spray cooling system but also the absorption chiller.

450

Fig. 6 shows the effect of inlet water temperature on COPsc results under

451

different thermal control temperatures. It is found that the COPsc decreases sharply

452

with the increase of inlet water temperature. This is because the increase of inlet water

453

temperature will greatly decrease the cooling capacity of spray cooling, however the

454

power consumption is almost independent of inlet water temperature. For comparison,

455

the COPwc of original cooling system is calculated by Eq.2 and the COPwc value of

456

4.8 is marked in Fig.6. At thermal control temperature of 85℃, all the COPsc results

457

are larger than COPwc. At thermal control temperature of 80℃, the COPsc are larger

458

than COPwc when the inlet temperature is lower than 75℃.This indicates that the

459

spray cooling system is always superior to the original water-cooled cabinets when

460

thermal control temperature is larger than 85℃. The comparison of COP results

461

shows that the plug-type spray cooling system is much more efficient than the water-

462

cooled cabinet system at appropriate conditions.

463 464 465

4.2 Absorption chiller performance The waste heat temperature is a key parameter affecting the absorption chiller 17

ACCEPTED MANUSCRIPT 466

performance since it is the heat source temperature of generator. It should be noted

467

that the waste heat temperature is equivalent to the outlet water temperature of spray

468

cooling. According to the spray cooling analysis in section 4.1, the temperature range

469

of waste heat is 75-83.5℃. To study the effect of waste heat temperature on

470

absorption chiller performance, the simulation parameters are set according to

471

reference [41,42] as listed in Table 3. The cooling capacity, power consumption and

472

COP results are shown in Fig. 7 and 8.

473

Fig. 7 shows the effect of waste heat temperature on cooling capacity and power

474

consumption. It is found that the two-stage LiBr absorption chiller cannot be operated

475

when the waste heat temperature is lower than 63℃. Thus, to ensure the effective

476

operation of absorption chiller, the outlet water temperature of spray cooling system

477

should be higher than 63℃. With the increase of waste heat temperature, the cooling

478

capacity increases rapidly at first, and then increase slowly. But when the waste heat

479

temperature is larger than 80℃, the increase of cooling capacity becomes very small.

480

Since the higher waste heat temperature (corresponds to a higher outlet temperature of

481

spray cooling system) results in lower spray cooling capacity, waste heat with

482

temperature larger than 80℃ is not beneficial to the overall cooling system. The

483

cooling capacity is as high as 80kW when the waste heat temperature reaches 80℃

484

and the corresponding power consumption is about 38kW. This amount of cooling

485

capacity is able to eliminate the heat load of air conditioners in storage racks.

486

Fig.8 shows the effect of waste heat temperature on thermal COP and electrical

487

COP of absorption chiller. The same variation trend of COP results with cooling

488

capacity is also shown in the figure. With the increase of waste heat temperature, both

489

the thermal COP and electrical COP increase rapidly at first. But when the waste heat

490

temperature exceeds 80℃, the thermal COP and electrical COP reach 0.33 and 2.1

491

respectively and almost no longer increase. This indicates that the absorption chiller

492

experiences better cooling performance with a higher waste heat temperature, since

493

more cooling capacity can be achieved under this condition. In conclusion, to obtain 18

ACCEPTED MANUSCRIPT 494

high COP values, it is necessary to operate the generator at higher temperature.

495

However, the further increase of waste heat temperature will decrease the cooling

496

performance of spray cooling system. Thus, the best temperature range of waste heat

497

is between 75-80℃ taking into consideration the optimization of overall integrated

498

cooling system. Regarding to the waste heat temperature range of 75-83.5℃, the

499

cooling capacity, thermal COP and electrical COP of absorption chiller is about

500

80kW, 0.33 and 2.1 respectively. This proves that the performance of absorption

501

chiller is excellent in our novel energy system.

502 503

4.3 Comprehensive performance of integrated energy system

504

In order to better present the energy efficiency improvement of our novel

505

integrated energy system, a comparison study between the original energy system and

506

novel energy system is conducted. In the simulation, the configurations of spray

507

cooling system such as water temperature, flow rate and cold plate number are listed

508

in Table 4. As the spray cooling inlet water temperature has a great effect on the novel

509

integrated energy system, the comprehensive performance of the novel system is

510

evaluated as a function of the spray cooling inlet water temperature under different

511

working conditions.

512

Fig. 9 shows the effect of spray cooling inlet water temperature on cooling

513

system power consumption (Wcool) and energy saving efficiency (ESE) in winter.

514

With the increase of spray cooling inlet temperature, Wcool decreases a lot at first,

515

reaches the lowest value of 155kW at inlet temperature of 55℃, and then increases

516

rapidly. This indicates that an optimal inlet temperature exists at which the lowest

517

Wcool is obtained. Fig. 9 also shows that the ESE increases with the increase of inlet

518

temperature at first, and then decrease when the inlet temperature exceeds 55℃. The

519

maximum ESE of 49% is obtained at inlet temperature of 55℃. This non-

520

monotone effect is caused by the interaction between spray cooling system and

521

absorption chiller. As explained in section 4.1 and 4.2, with the increase of inlet water 19

ACCEPTED MANUSCRIPT 522

temperature, the outlet water temperature of spray cooling system increases but

523

cooling capacity decreases. Since the outlet water of spray cooling system is the heat

524

source of absorption chiller, the cooling capacity of absorption chiller increases

525

sharply with the increase of inlet water temperature at first. As the increase effect of

526

absorption chiller is stronger than the decrease effect of spray cooling system at lower

527

temperatures, the ESE increases at first. With the further increase of inlet water

528

temperature, the spray cooling capacity continues increasing but the cooling capacity

529

of absorption chiller almost no longer increase. So the ESE begins to decrease.

530

Fig. 10 shows the effect of spray cooling inlet water temperature on power usage

531

effectiveness (PUE) and cooling load factor (CLF) in winter. Similar to Fig. 9, the

532

PUE and CLF values of novel system decrease at first, and then increase with the

533

increase of inlet temperature. The lowest PUE and CLF values of 1.44 and 0.22 are

534

obtained at inlet temperature of 55℃. For comparison, PUE and CLF values of the

535

original supercomputer system (PUEoriginal, CLForiginal) in winter are also marked in the

536

figure. It is found that no matter how the inlet temperature increase, the PUE and CLF

537

values of novel system are always much lower than the original system. All the PUE

538

and CLF results of novel system are better than the original system with inlet

539

temperature range from 40 to 70℃. According to reference [40], PUEoriginal value of

540

1.66 is in the degree of Improved Operation Scenario. However, the novel system

541

could achieve PUE of 1.44 within best practice scenario under appropriate conditions.

542

According to above analysis, the comparison results between original energy

543

system and novel energy system at optimal design are listed in Table 5. It is surprising

544

that although the PUE of original system has been low enough, the novel system

545

could further reduce the PUE to a level of Best Practice Scenario [40]. What’s more,

546

by using the novel system, total power consumption devoted to operate IT equipment

547

is improved from 60% to 69%, which means a high level of energy efficiency. Power

548

consumption of cooling facilities is reduced from 27% to 16%, which means a high

549

level of energy saving. To further prove the reliability and energy efficiency of the 20

ACCEPTED MANUSCRIPT 550

novel energy system, we have conducted another case study at thermal control

551

temperature of 80℃. The power consumption and energy efficiency results at this

552

case are also listed in Table 5. Although a lower and stricter thermal control

553

temperature will increase the power consumption of cooling system, the ESE is still as

554

high as 40%. This proves the energy saving effect of the novel energy system.

555

In order to evaluate the energy efficiency of novel system at different working

556

conditions, the performance comparison of different seasons during one year is

557

conducted between novel system and original system. The performance variation of

558

original system is listed in Table 2. According to above analysis, the performance of

559

novel system at the optimal inlet temperature over four seasons is shown in Fig. 11-

560

12. Fig. 11 illustrates the comparison of Wcool between novel and original systems in

561

different seasons. It is found that Wcool is lowest in winter and highest in summer. For

562

both systems, the power consumption in summer is about 30% higher than that in

563

winter. In summer, due to high ambient temperature, the power consumption of

564

cooling system is much higher than that in other seasons. Compared with original

565

system, novel system could reduce power consumption of about 49% in winter, 49%

566

in spring and autumn, and 48% in summer. This indicates that the energy saving

567

effect of novel system is remarkable in all seasons. Fig. 12 shows the comparison of

568

PUE between novel and original systems in different seasons. For both systems, PUE

569

value in summer is very close to that in other seasons. In addition, compared with

570

original system, novel system could reduce PUE below 1.5 to best practice scenario in

571

all seasons. This proves that the novel cooling system is superior to original system all

572

the year round.

573 574

4.4 Simple economic analysis of integrated energy system

575

A simple economic analysis is performed on the novel energy system to provide

576

more insights into the application potential of the novel system. The initial investment

577

cost and operation cost of novel energy system and original energy system are listed 21

ACCEPTED MANUSCRIPT 578

in Table 6. In economic analysis, the total cooling capacity of both system is 1081kW.

579

For novel system, the cooling capacity of spray cooling system, absorption chiller and

580

air conditioners are 966 kW, 80kW and 35kW, respectively. For original system, the

581

cooling capacity of water-cooled cabinets and air conditioners are 966 kW and

582

115kW respectively. For novel energy system, the largest cost is the multi-nozzle

583

array in spray cooling system. The multi-nozzle arrays are still specialty product at

584

present. They are customized and fabricated commercially, so the price is quite high.

585

As the maximum cooling load of spray cooling system is as high as 966kW, the

586

demand of multi-nozzle arrays, cold plates, water tanks, pumps, pipes and other

587

additional components is increased dramatically. Thus, the cost is increased

588

substantially. The absorption chiller used in this paper is more complex than

589

commercial air conditioner, so its price is consequently much higher. Various

590

additional costs are also listed in Table 6. It is found that the total cost of the novel

591

system is 325,690 dollars, while the total cost of original system is 101,200 dollars.

592

Although the novel system costs 3 times of the original system, a large amount of

593

power consumption savings caused by novel system could lower the operation cost

594

greatly. The cost payback period (CPP) can be calculated as follows:

595

CPP 

596

Through analysis, a payback period of 1.5 years can be achieved, which proves the

597

application potential of the novel energy system.

Total cost of novel system-Total cost of original system Operation cost of original system- Operation cost of novel system

(22)

598 599

5. Conclusions

600

In order to improve the energy efficiency of supercomputer center, a novel

601

energy system based on spray cooling was proposed in this paper. The novel system

602

was composed of a plug-type spray cooling system and a waste heat reusing

603

absorption chiller. Overall simulation of the novel energy system was conducted

604

based on spray cooling model and absorption chiller model. Energy saving evaluation 22

ACCEPTED MANUSCRIPT 605

was compared with the original energy system of Dawning 5000A supercomputer

606

system. The energy saving efficiency (ESE), power usage effectiveness (PUE) and

607

cooling load factor (CLF) under different working conditions were analyzed. The key

608

findings can be summarized as follows:

609

(1) Overall simulation results showed that the novel energy system is superior to

610

original energy system of dawning 5000A supercomputer in all seasons. The

611

novel system could achieve ESE as high as 49% and PUE within best practice

612

scenario of 1.44.

613

(2) The energy saving effect is strongly affected by inlet water temperature of spray

614

cooling. With the increase of inlet water temperature, the spray cooling capacity

615

decreases while the absorption cooling capacity increases.

616

(3) There exists an optimal inlet temperature (e.g. 55℃ at thermal control

617

temperature of 85℃), at which the lowest cooling power consumption, PUE, CLF

618

and highest ESE can be achieved. At the optimal design, cooling power

619

consumption of novel energy system only accounts for 16% in the total power

620

consumption. Power consumption that devoted to running the IT equipment is

621

improved from 60% to 69%.

622

(4) Plug-type spray cooling system is much more efficient than water-cooled cabinet

623

adopted in Dawning 5000A supercomputer. The spray cooling capacity increases

624

with the increase of inlet water temperature and thermal control temperature.

625

Maximum COP value of 18 is achieved at inlet temperature of 40℃ and thermal

626

control temperature of 85℃.

627

(5) Two-stage LiBr/H2O absorption chiller could make good use of low temperature

628

waste heat discharged by spray cooling system. The cooling capacity, thermal

629

COP and electrical COP increase with the increase of waste heat temperature,

630

reach maximum value of 80kW, 0.33 and 2.1 respectively, and no longer increase

631

when waste heat temperature is higher than 75℃.

632 23

ACCEPTED MANUSCRIPT 633 634

Conflicts of interest There are no conflicts of interest to declare.

635 636 637 638

Acknowledgement The authors would like to thank the National Science Foundation of China (Grant No. 51376169) for the financial support.

639 640

References

641

[1] Demetriou D W, Kamath V, Mahaney H. A Holistic Evaluation of Data Center

642

Water Cooling Total Cost of Ownership. J Electron Packaging 2016, 138(1): 010912.

643

[2] Sun W C, Huang Y. Data center energy consumption model analysis and research.

644

Shanghai Energy Conserv 2014 (1): 29-32.Chinese

645

[3] Li Z, Kandlikar S G. Current status and future trends in data-center cooling

646

technologies. Heat Transfer Eng 2015, 36(6): 523-538.

647

[4] Zimmermann S, Meijer I, Tiwari M K, et al. Aquasar: A hot water cooled data

648

center with direct energy reuse. Energy 2012, 43(1): 237-245.

649

[5] Ellsworth M J, Iyengar M K. Energy efficiency analyses and comparison of air

650

and water cooled high performance servers. Proceedings of IPACK, SHTC and ICES,

651

San Francisco, 2009: 907-914.

652

[6] Daungthongsuk W, Wongwises S. A critical review of convective heat transfer of

653

nanofluids. Renew Sust Energy Rev 2007, 11(5): 797-817.

654

[7] Rezania A, Rosendahl L A. Thermal effect of a thermoelectric generator on

655

parallel microchannel heat sink. Energy 2012, 37(1): 220-227.

656

[8] Hosain M L, Fdhila R B, Daneryd A. Heat transfer by liquid jets impinging on a

657

hot flat surface. Appl Energy 2016, 164: 934-943.

658

[9] Kandlikar S G, Bapat A V. Evaluation of jet impingement, spray and

659

microchannel chip cooling options for high heat flux removal. Heat Transfer Eng

660

2007, 28(11): 911-923.

661

[10] Kim J. Spray cooling heat transfer: the state of the art. Int J Heat Fluid Fl 2007, 24

ACCEPTED MANUSCRIPT 662

28(4): 753-767.

663

[11] Silk E A, Golliher E L, Selvam R P. Spray cooling heat transfer: technology

664

overview and assessment of future challenges for micro-gravity application. Energy

665

Convers Manage 2008, 49(3): 453-468.

666

[12] Shedd T A. Next generation spray cooling: high heat flux management in

667

compact spaces. Heat Transfer Eng 2007, 28(2): 87-92.

668

[13] Pautsch A G, Shedd T A. Spray impingement cooling with single-and multiple-

669

nozzle arrays. Part I: Heat transfer data using FC-72. Int J Heat Mass Transfer 2005,

670

48(15): 3167-3175.

671

[14] Shedd T A, Pautsch A G. Spray impingement cooling with single-and multiple-

672

nozzle arrays. Part II: Visualization and empirical models. Int J Heat Mass Transfer

673

2005, 48(15): 3176-3184.

674

[15] Cheng W L, Zhang W W, Jiang L J, et al. Experimental investigation of large

675

area spray cooling with compact chamber in the non-boiling regime. Appl Therm Eng

676

2015, 80: 160-167.

677

[16] Zhang W W, Cheng W L, Shao S D, et al. Integrated thermal control and system

678

assessment in plug-chip spray cooling enclosure. Appl Therm Eng 2016, 108: 104-

679

114.

680

[17] Cheng W L, Zhang W W, Shao S D, et al. Effects of inclination angle on plug-

681

chip spray cooling in integrated enclosure. Appl Therm Eng 2015, 91: 202-209.

682

[18] Wang F J, Chiou J S, Wu P C. Economic feasibility of waste heat to power

683

conversion. Appl Energy 2007, 84(4): 442-454.

684

[19] Haywood A, Sherbeck J, Phelan P, et al. Thermodynamic feasibility of

685

harvesting data center waste heat to drive an absorption chiller. Energy Convers

686

Manage 2012, 58: 26-34.

687

[20] Javani N, Dincer I, Naterer G F. Thermodynamic analysis of waste heat recovery

688

for cooling systems in hybrid and electric vehicles. Energy 2012, 46(1): 109-116.

689

[21] Capozzoli A, Primiceri G. Cooling Systems in Data Centers: State of Art and 25

ACCEPTED MANUSCRIPT 690

Emerging Technologies. Energy Procedia 2015, 83: 484-493.

691

[22] Florides G A, Kalogirou S A, Tassou S A, et al. Modelling and simulation of an

692

absorption solar cooling system for Cyprus. Sol energy 2002, 72(1): 43-51.

693

[23] Sumathy K, Huang Z C, Li Z F. Solar absorption cooling with low grade heat

694

source-a strategy of development in South China. Sol energy 2002, 72(2): 155-165.

695

[24] Du S, Wang R Z, Lin P, et al. Experimental studies on an air-cooled two-stage

696

NH3-H2O solar absorption air-conditioning prototype. Energy 2012, 45(1): 581-587.

697

[25] Lin P, Wang R Z, Xia Z Z. Numerical investigation of a two-stage air-cooled

698

absorption refrigeration system for solar cooling: cycle analysis and absorption

699

cooling performances. Renew Energy 2011, 36(5): 1401-1412.

700

[26] Siddiqui M A. Economic analysis of two stage dual fluid absorption cycle for

701

optimizing generator temperatures. Energy Convers Manage 2001, 42(4): 407-437.

702

[27] Kim D S, Ferreira C A I. Air-cooled LiBr-water absorption chillers for solar air

703

conditioning in extremely hot weathers. Energy Convers Manage 2009, 50(4): 1018-

704

1025.

705

[28] Sun N, Kahaner D, Chen D. High-performance computing in China: research and

706

applications. Int J High Perform C 2010, 24(4): 363-409.

707

[29] Zhang Y, Sun J, Yuan G, et al. Perspectives of China’s HPC system

708

development: a view from the 2009 China HPC TOP100 list. Front Comput Sci China

709

2010, 4(4): 437-444.

710

[30] Chen Y J, Zhang Y Q, Huang K Z, et al. Energy Efficiency Analysis of Shanghai

711

Supercomputing Center. Elec Appl 2010, 29(18):18-21. Chinese

712

[31] Zhang Y Q, Chen Y Q, Huang K Z, et al. Analysis of energy efficiency

713

improvement of supercomputing center in Shanghai. Sci times 2010 (12): 299-303.

714

Chinese

715

[32] Figueredo G R, Bourouis M, Coronas A. Thermodynamic modelling of a two-

716

stage absorption chiller driven at two-temperature levels. Appl Therm Eng 2008,

717

28(2): 211-217. 26

ACCEPTED MANUSCRIPT 718

[33] Cheng W L, Han F Y, Liu Q N, et al. Experimental and theoretical investigation

719

of surface temperature non-uniformity of spray cooling. Energy 2011, 36(1): 249-257.

720

[34] Cheng W L, Han F Y, Liu Q N, et al. Spray characteristics and spray cooling

721

heat transfer in the non-boiling regime. Energy 2011, 36(5): 3399-3405.

722

[35] Zhao R, Cheng W, Liu Q, et al. Study on heat transfer performance of spray

723

cooling: model and analysis. Heat mass transfer 2010, 46(8-9): 821-829.

724

[36] Patek J, Klomfar J. A computationally effective formulation of the

725

thermodynamic properties of LiBr-H2O solutions from 273 to 500K over full

726

composition range. Int J Refrig 2006, 29(4): 566-578.

727

[37] Chua H T, Toh H K, Malek A, et al. Improved thermodynamic property fields of

728

LiBr-H2O solution. Int J Refrig 2000, 23(6): 412-429.

729

[38] Kilic M, Kaynakli O. Second law-based thermodynamic analysis of water-

730

lithium bromide absorption refrigeration system. Energy 2007, 32(8): 1505-1512.

731

[39] Hasan A, Siren K. Theoretical and computational analysis of closed wet cooling

732

towers and its applications in cooling of buildings. Energy Build 2002, 34(5): 477-

733

486.

734

[40] Brown R. Report to congress on server and data center energy efficiency: Public

735

law 109-431. Berkeley Lab 2008.

736

[41] Ma W B, Deng S M. Theoretical analysis of low-temperature hot source driven

737

two-stage LiBr/H2O absorption refrigeration system. Int J refrig 1996, 19(2): 141-146.

738

[42] Ma W B, Xia W H, Yu C B. Industrial application of the two-stage LiBr/H2O

739

absorption chiller. Refrigeration 1998, 4: 40-43. Chinese

740 741

Appendix A. Experimental results spray cooling

742

As schematized in Fig. A1, spray cooling experimental studies were conducted

743

by our previous studies [15,34]. The single-nozzle experiment and multi-nozzles

744

experiment were conducted by the works in Ref [34] and Ref [15] respectively. For

745

single nozzle spray cooling, the heat transfer correlation in non-boiling regime is as 27

ACCEPTED MANUSCRIPT 746

follows [34]:

747

Nu  0.036 Re1.04We0.28 Pr 0.51  3.02   1.53 

748

Tsurf  Tinlet Tsat is the dimensionless temperature. Further details on where,  =（）

749

dimensional analysis and correlation derivation were reported previously [34]. In this

750

correlation, Nusselt number is range from 200 to 600, Prandtl number is from 2.1 to

751

6.8 and dimensionless temperature is from 0.08 to 0.7. The correlation agrees well

752

with experimental result and the deviation is within 7%.

(A.1)

753

The heat removal capacity and heat transfer coefficient results of non-boiling

754

spray cooling were also obtained by our previous work [15,34]. Fig. A2 (a) shows the

755

single-nozzle results of Ref [34] and Fig. A2(b) shows the multi-nozzle results of Ref

756

[15]. It was found that the heat removal capacity is as high as 250W/cm2 for single

757

nozzle and as high as 80W/cm2 for multi-nozzles. This proved the high heat removal

758

potential of spray cooling. The heat removal capacity increases proportionally with

759

the increase of temperature difference (difference of surface temperature and water

760

inlet temperature) for both single and multi-nozzles. What’s more, the heat removal

761

capacity increases with the increase of spray flow rate. Compared with single-nozzle

762

spray cooling, the multi-nozzle spray cooling shows a smaller heat removal capacity

763

due to its larger surface area.

764 765

Appendix B. Mathematical model of spray cooling

766

The mathematical model of spray cooling heat transfer was fully described in our

767

previous work [33,35]. In single-phase regime, spray cooling heat transfer is

768

dominated by liquid film convection and droplets impingement. Spray cooling

769

process can be simplified as below: high-pressure working fluid is atomized through

770

nozzle and becomes tiny droplets with high velocity. The spray droplets impact on

771

heating surface continuously and form a liquid film on the heating surface. The liquid

772

film washes the surface and cools the surface by convection. The droplet impaction

773

heat transfer (Qdrop) consists two parts: heat transfer induced by droplets crossing the 28

ACCEPTED MANUSCRIPT 774

liquid film (Qdrop,1) and heat transfer induced by the droplets hitting on the heating

775

surface (Qdrop,2). In this model, the droplets are assumed to be spherical particles and

776

the film is assumed to be viscous liquid. Then heat transfer between droplets and film

777

is as follows:

778

Q drop,1  film lfilm Nudrop  Adrop (Tdrop  Tfilm )

779

The heat transfer of droplet hitting the surface can be empirically evaluated using an

780

effectiveness parameter (), which is defined as the ratio of the actual heat transfer to

781

the maximum possible heat transfer. The relationship between  and droplet Weber

782

number was concluded in Ref [33,35]. Then heat transfer of droplet hitting the surface

783

is as follows:

784

Q drop,2   G[C p ,l (Tw  Tdrop )]

785

In order to obtain film-surface convective heat transfer, the film formation and film

786

motion are modeled based on dynamics fundamentals. The mass conservation and

787

momentum conservation equations of the film are as follows:

788

(B.1)

(B.2)

du d(lfilm )  lfilm film,i  m in /  Afilm d dxi d(ufilm,i lfilm ) d

 ufilm,i lfilm

dufilm,i dxi

(B.3)

 m in uin,i /  Afilm

789

The film thickness and film velocity can be determined by these equations. Then the

790

film-surface heat transfer (Qfilm) can be expressed by an empirical correlation as

791

follows:

792

Q film  liq lfilm Nufilm  Afilm (Tw  Tfilm )

793

Thus, the equation governing energy conservation in non-boiling spray cooling is

794

given as follows:

795

Q heating =Q drop +Q film

(B.4)

(B.5)

796

The initial droplet parameters such as mean diameter, velocity and mass flux

797

were measured by the Phase-Doppler Anemometer (PDA), and analyzed in our

798

previous work [34]. Based on Eq. (B1-B5), the spray cooling performance in non29

ACCEPTED MANUSCRIPT 799

boiling regime can be calculated through simulation procedure. The model is

800

validated by the experimental results provided in Ref [35], and a favorable

801

comparison is demonstrated with a deviation below 10%.

30

ACCEPTED MANUSCRIPT Figure captions Fig.1. Schematic diagram of integrated energy system in supercomputer Fig.2. Schematic diagram of plug-type spray cooling system Fig.3. Schematic diagram of two-stage absorption chiller Fig.4. Basic cycle process and P-T-X diagram of two-stage absorption chiller Fig.5. Effect of inlet water temperature on outlet water temperature and spray cooling heat removal capacity. Fig.6. Effect of spray cooling inlet water temperature on COPsc Fig.7. Effect of waste heat temperature on cooling capacity and power consumption of absorption chiller Fig.8. Effect of waste heat temperature on thermal COP and electrical COP of absorption chiller Fig.9. Effect of spray cooling inlet water temperature on Wcool and ESE Fig.10. Effect of spray cooling inlet water temperature on PUE and CLF Fig.11. Total power consumption of cooling system, Wcool, during one year Fig.12. Power usage effectiveness, PUE, during one year Fig.A4. Experimental setup: (a) Schematic diagram of single and multi-nozzle spray cooling. (b) Compact spray chamber and Multi-nozzle array. Fig.A5. Experimental results of multi-nozzle spray cooling capacity at different water flow rate in non-boiling regime (Results reported in previous work[15,34]).

ACCEPTED MANUSCRIPT

Fig. 1. Schematic diagram of integrated energy system in supercomputer.

ACCEPTED MANUSCRIPT

Fig. 2. Schematic diagram of plug-type spray cooling system.

ACCEPTED MANUSCRIPT

Fig. 3. Schematic diagram of two-stage absorption chiller.

ACCEPTED MANUSCRIPT

Fig.4. Basic cycle process and P-T-X diagram of two-stage absorption chiller

ACCEPTED MANUSCRIPT

90

200

85

o

85 C 2

Heat removal capacity (W/cm )

o

Outlet water temperature ( C)

o

Control temperature: 80 C Outlet temperature: Heat removal capacity:

150

80

100

75

50

70

0 40

50

60

70

80

o

Inlet water temperature ( C)

Fig.5. Effect of inlet water temperature on outlet water temperature and spray cooling heat removal capacity.

ACCEPTED MANUSCRIPT

30 o

25

Thermal control temperature: 80 C o 85 C

COPsc

20 15 10 5

COPwc=4.8

0 40

50

60

70

80

o

Inlet water temperature ( C)

Fig.6. Effect of spray cooling inlet water temperature on COPsc.

ACCEPTED MANUSCRIPT

100

50

Cooling capacity Power consumption

Cooling capacity (kW)

40

60 30 o

Chiller water outlet temperature at 7 C o Cooling water inlet temperature at 32 C

40

20

Power consumption (kW)

80

20 10 60

65

70

75

80

85

o

Waste heat temperature ( C)

Fig.7. Effect of waste heat temperature on cooling capacity and power consumption of absorption chiller.

ACCEPTED MANUSCRIPT

0.5

3

Thermal COP Electrical COP

0.4

Thermal COP

o

Chiller water outlet temperature at 7 C o Cooling water inlet temperature at 32 C

0.2

1

Electrical COP

2 0.3

0.1

0.0

0 60

65

70

75

80

85

o

Waste heat temperature ( C)

Fig.8. Effect of waste heat temperature on thermal COP and electrical COP of absorption chiller.

220

200

70

In winter： Wcool ESE

60

180 50 160

140

40

Tspray,in=Tg,out Tspray,out=Tg,in

30

Energy saving efficiency (%)

Wcool of integrated cooling system (kW)

ACCEPTED MANUSCRIPT

120 50

55

60

65

70 o

Spray cooling inlet water temperature ( C)

Fig.9. Effect of spray cooling inlet water temperature on Wcool and ESE.

ACCEPTED MANUSCRIPT

0.5

PUEoriginal=1.66

CLForiginal=0.44

1.6

0.4

Tspray,in=Tg,out Tspray,out=Tg,in

1.5

0.3

In winter： PUE CLF

1.4

0.2

1.3

Cooling Load Factor (CLF)

Power usage effectiveness (PUE)

1.7

0.1 50

55

60

65

70 o

Spray cooling inlet water temperature ( C)

Fig.10. Effect of spray cooling inlet water temperature on PUE and CLF.

ACCEPTED MANUSCRIPT

500 o

Wcool (kW)

400

Spray cooling inlet water temperature: 55 C Wcool of integrated cooling system Wcool of original cooling system

300

200

100

0

Winter

Spring/autum

Summer

Four seasons in one year

Fig.11. Total power consumption of cooling system, Wcool, during one year

ACCEPTED MANUSCRIPT

2.5

Power usage effectiveness (PUE)

o

2.0

Spray cooling inlet water temperature: 55 C PUE of integrated cooling system PUE of original cooling system

1.5

1.0

0.5

0.0

Winter

Spring/autum

Summer

Four seasons in one year

Fig.12. Power usage effectiveness, PUE, during one year

ACCEPTED MANUSCRIPT

Fig.A4. Experimental setup: (a) Schematic diagram of single and multi-nozzle spray cooling. (b) Compact spray chamber and Multi-nozzle array.

200

7 6 5

100

4 0

0

10

20

30

40

50

o

60

Temperature difference, Tsurf-Tinlet, ( C)

(a) Single-nozzle results [34]

3 70

Multi-nozzle spray cooling results[15]： 2 Heating surface area of 9 cm Multi-nozzle plate composed of 8 nozzles

6

Flow rate

5

Q

h

100 15.4 L/m2s 2

12.3 L/m s 2 9.3 L/m s

4

50

3 2

0

10

20

30

40

o

Temperature difference, Tsurf-Tinlet, ( C)

50

2

2

2

8

150

Heat transfer coefficient, h, (W/cm K)

300

9

Heat removal capacity,Q, (W/cm )

10

Single-nozzle spray cooling results[34]： 2 Heating surface area of 1 cm Spray height: 2.3mm Flow rate Q h 3.6 L/h 4.2 L/h 5.20L/h

2

Heat removal capacity,Q, (W/cm )

400

Heat transfer coefficient, h, (W/cm K)

ACCEPTED MANUSCRIPT

(b) Multi-nozzle results [15]

Fig. A5. Experimental results of multi-nozzle spray cooling capacity at different water flow rate in non-boiling regime (Results reported in previous work[15,34]).

ACCEPTED MANUSCRIPT Figure captions Fig.A1. Experimental setup: (a) Schematic diagram of single and multi-nozzle spray cooling. (b) Compact spray chamber and Multi-nozzle array. Fig.A2. Experimental results of multi-nozzle spray cooling capacity at different water flow rate in non-boiling regime (Results reported in previous work[15,34]).

Fig.A1. Experimental setup: (a) Schematic diagram of single and multi-nozzle spray cooling. (b) Compact spray chamber and Multi-nozzle array.

200

7 6 5

100

4 0

0

10

20

30

40

50

o

60

Temperature difference, Tsurf-Tinlet, ( C)

(a) Single-nozzle results [34]

3 70

Multi-nozzle spray cooling results[15]： 2 Heating surface area of 9 cm Multi-nozzle plate composed of 8 nozzles

6

Flow rate

5

Q

h

100 15.4 L/m2s 2

12.3 L/m s 2 9.3 L/m s

4

50

3 2

0

10

20

30

40

o

Temperature difference, Tsurf-Tinlet, ( C)

50

2

2

2

8

150

Heat transfer coefficient, h, (W/cm K)

300

9

Heat removal capacity,Q, (W/cm )

10

Single-nozzle spray cooling results[34]： 2 Heating surface area of 1 cm Spray height: 2.3mm Flow rate Q h 3.6 L/h 4.2 L/h 5.20L/h

2

Heat removal capacity,Q, (W/cm )

400

Heat transfer coefficient, h, (W/cm K)

ACCEPTED MANUSCRIPT

(b) Multi-nozzle results [15]

Fig. A2. Experimental results of multi-nozzle spray cooling capacity at different water flow rate in non-boiling regime (Results reported in previous work[15,34]).

ACCEPTED MANUSCRIPT Table captions Table.1. Cooling capacity and heat load in Dawning 5000A supercomputer Table.2. Power consumption of each subsystem in Dawning 5000A supercomputer Table 3. Simulation conditions of two-stage absorption chiller Table.4. Configuration of spray cooling system in Dawning 5000A supercomputer Table.5. Comparison of original and novel system at optimal design Table 6. Cost of novel energy system and original energy system

ACCEPTED MANUSCRIPT

Table 1. Cooling capacity and heat load in Dawning 5000A supercomputer Component

Rack number

Maximum heat load

Cooling method

Compute racks

42

966 kW

Water-cooled cabinets

I/O and communication racks

10

45 kW

Air conditioners

Storage racks

13

70 kW

Air conditioners

ACCEPTED MANUSCRIPT

Table 2. Power consumption of each subsystem in Dawning 5000A supercomputer Subsystem

Description

IT equipment

Power consumption (kW) Winter

Spring/autumn

Summer

Computing system

692.2

714.4

789.2

Cooling

Water-cooled cabinet

207.3

232.6

301.1

system

Air conditioner

97.6

102.2

118.3

Power supply

Power supply loss of

loss

power distribution unit

145.7

89.2

88.6

6.6

4.1

4.1

1149.4

1142.5

1301.3

3.6℃

9.8℃

27.8℃

Else

Lightening and other equipments

Total

\

Average ambient temperature in Shanghai

ACCEPTED MANUSCRIPT

Table 3. Simulation conditions of two-stage absorption chiller Design parameters Waste heat temperature Tin

Values/Ranges 65~85℃

Chilled

Tchw,in

14℃

water

Tchw,out

7℃

Cooling

Tcw,in

32℃

water

Tcw,iout

27℃

PC

7.37 kPa

PE

0.93 kPa

ACCEPTED MANUSCRIPT

Table 4. Configuration of spray cooling system in Dawning 5000A supercomputer Parameters

Values/Ranges

Inlet cooling water temperature, Tin (℃)

40~80

Waste heat temperature, Tout (℃)

65-85

Total volume flow rate of water, Vin ( L/s)

3.85

Cold plate number per rack

4

Nozzle number per cold plate

100

Total cold plate number

168

ACCEPTED MANUSCRIPT

Table 5. Comparison of original and novel energy system at optimal design in winter Original energy system

Novel energy system Thermal control temperature 85℃

80℃

207.3 kW Spray cooling system

99.1 kW

124.5 kW

97.6 kW

Air conditioner

20.5 kW

23.2 kW

Absorption chiller

35.4 kW

34.1 kW

Power consumption

Power consumption

Water-cooled cabinet Air conditioner

Parameter

Value

Parameter

Value

Wcool/Wtotal

27%

Wcool/Wtotal

16%

18%

WIT/Wtotal

60%

WIT/Wtotal

69%

67%

PUE

1.66

PUE

1.44

1.48

CLF

0.44

CLF

0.22

0.26

ESE

0.49

0.40

ACCEPTED MANUSCRIPT

Table 6. Cost of novel energy system and original energy system Items

Components

Value

Novel energy system

Investment cost (cost per cooling capacity)

Operation cost

Cooling water tank

25$/kW

Liquid pump

40$/kW

Multi-nozzle array

200$/kW

Else

50$/kW

Cooling tower

150$/kW

Else

100$/kW

Air conditioner (35kW)

40$/kW

Total cost

325,690$

Electricity and maintenance

3500$/day

Spray cooling system (966kW)

Absorption chiller (80kW)

Original energy system

Investment cost

Operation cost Payback period

Water-cooled cabinet (966kW)

100$/kW

Air conditioner (115kW)

40$/kW

Total cost

101,200$

Electricity and maintenance

3900$/day 1.5years