Internet data centers participating in demand response: A comprehensive review

Renewable and Sustainable Energy Reviews 117 (2020) 109466

Contents lists available at ScienceDirect

Renewable and Sustainable Energy Reviews journal homepage: http://www.elsevier.com/locate/rser

Internet data centers participating in demand response: A comprehensive review Min Chen a, Ciwei Gao a, *, Meng Song a, Songsong Chen b, Dezhi Li b, Qiang Liu c a

School of Electrical Engineering, Southeast University, Jiangsu, 210096, China China Electric Power Research Institute, Beijing, 100192, China c Zhejiang Electric Power Corporation, Zhejiang, 310007, China b

A R T I C L E I N F O

A B S T R A C T

Keywords: Demand response Economic considerations Internet data centers Load modeling Load regulation operations

Internet data centers (IDCs), which have the potential of spatial and temporal load regulation, are excellent demand response (DR) resources. IDCs participating in DR has recently become a popular topic as it is economical and efficiently helps improve power systems. However, there remain abundant opportunities to improve this interdisciplinary domain when considering the lack of applicability of IDC load models for power systems, exploitation regarding the potential of IDC load regulation, and DR mechanisms for spatial-coupling loads. Therefore, a review summarizing the state-of-art studies around the theme of IDCs participating in DR is warranted. A comprehensive survey covering the major parts of the DR in IDCs, along with the order load modeling, load regulation operations, economic considerations, and IDCs participating in DR programs, is pre­ sented in this paper. Furthermore, the challenges and future research issues are also discussed for further participation of DR in IDCs.

1. Introduction Internet data centers (IDCs) are increasingly being deployed across different geographical locations by Internet service companies (ISCs), such as Google, Amazon, and Microsoft, to provide Internet-scale ser­ vices with low latency and high reliability. IDCs have emerged as a major electricity consumer, significantly increasing the electricity load at locations where they are built. For example, Microsoft’s IDC in Quincy, Washington, consumed 48 MW power, which is equivalent to the power consumption of approximately 40,000 households [1]. As reported in Ref. [2], data centers consumed 1.5% of the worldwide electricity supply in 2011, and this fraction is expected to grow to 8% by 2020. The brief architecture of IDC operation, which belongs to a single ISC, is shown in the upper-left part of Fig. 1. First, IDC operation starts with the arrival of the workload. The inputs include the total arriving workload and external environmental parameters. The output is the power demand of each IDC, which can be divided into four major parts: the IT equipment power consumption, the cooling system power con­ sumption, the power consumption of other equipment, and the power supply of the auxiliary energy systems. Specifically, IT equipment pro­ cesses the workload, ensuring the quality of service (QOS) of the tenants.

In addition, the cooling system provides cooling power to eliminate the heat generated by IT and other equipment, as well as the heat trans­ ferred from the outdoors, maintaining the internal ambient temperature of the IDCs. Other equipment includes lighting systems and power de­ livery systems, as well as auxiliary energy systems supplying power to the IDCs, ensuring a stable power source, reducing energy costs, and allowing the system to be more sustainable. Note that the power consumed by IT equipment and the cooling system accounts for 80%– 90% of the dynamic power consumption, excluding auxiliary energy systems [3], which can be dynamically controlled by controlling the amount of workload to be processed and the heat to be extracted, respectively. The power supply of auxiliary energy systems, except for a few intermittent renewable generators (RGs), e.g., photovoltaic and wind power, can also be dynamically controlled. However, the power consumption of other equipment is generally assumed to be constant. Practically speaking, Internet-scale services, which have gained tremendous popularity and widespread adoption owing to their flexible and on-demand nature [4], provide Internet-accessible computing re­ sources. Interestingly, these computing resources are not internally restricted to any geographic location, and can be hosted on an unknown server located far from the tenants through the links between front-end servers and IDCs. That is, the incoming workload can be distributed to various IDCs using front-end servers. Various products and services are

* Corresponding author. School of Electrical Engineering, Southeast University, No.2 Sipailou, Nanjing, PR China. E-mail addresses: [email protected] (M. Chen), [email protected] (C. Gao). https://doi.org/10.1016/j.rser.2019.109466 Received 2 May 2019; Received in revised form 28 September 2019; Accepted 3 October 2019 Available online 23 October 2019 1364-0321/© 2019 Elsevier Ltd. All rights reserved.

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

Nomenclature

Dq PUEi

Abbreviations BGs Back-up generators CCHP Combined cooling, heating, and power COP Coefficient of performance CRAC Computer room air conditioning DCSC Dynamic cluster server configuration DR Demand response DVFS Dynamic voltage and frequency scaling DW Dropping workload EDR Emergency demand response HVAC Heating, ventilation, and air conditioning IDCs Internet data centers ISCs Internet-service companies LMPs Locational marginal prices LP Linear programming MILP Mixed integer linear programming MIP Mixed integer programming MIW Migrating workload PEVs Plug-in electric vehicles PUE Power usage effectiveness QOS Quality of service RGs Renewable generators RTP Real-time pricing SOs System operators SW Shifting workload TSTI Thermal storage operation utilizing thermal inertia of buildings TSSs Thermal storage systems UPS Uninterrupted power supply VM Virtual machine

PLI;J max

Tolerant service delay of workload q [s] Ratio of total power consumption of IDC i to power consumption of IT equipment Active power flow limit from node I to node J [kW]

Variables about power consumption of IDCs pi;t Power consumption of IDC i in tth time slot [kW] Variables about workload Lt;δ;q Amount of workload q arriving at the front-end server δ in tth time slot [request/s] λi;t;δ;q Amount of workload q allocated from front-end server δ to IDC i in tth time slot [request/s] λdropped i;t;q

Amount of dropped workload q in IDC i in tth time slot [request/s]

Variables of IT equipment λi;t;δ;q;v Amount of workload q processed on server v allocated from front-end server δ to IDC i in the tth time slot [request/s] λdropped i;t;q;v pIT i;t CRi;t CRi;t;q Ii;t;v Fi;t;v mi;t

Amount of dropped workload q on server v in IDC i in the tth time slot [request/s] IT equipment power consumption in IDC i in tth time slot [kW] Total computing resources provided for processing workload in IDC i in tth time slot [request/s] Computing resources provided for processing workload q in IDC i in tth time slot [request/s] Operation status of server v in IDC i in tth time slot Frequency of server v in IDC i in tth time slot Number of active servers in IDC i in tth time slot

Variables of cooling system pcool Power consumption of cooling system in IDC i in tth time i;t slot [kW] LFi;t Ratio of IT equipment power consumption to cooling system power consumption in IDC i

Indices and sets i; j Index of IDCs N Set of geographically dispersed IDCs belonging to a single ISC t Index of time slots Τ Set of time slots I; J Index of power system nodes Π Set of power system nodes δ Index of front-end servers Φ Set of front-end servers q Index of workload types Ω Set of workload types v Index of servers Θ Set of servers r Index of PEVs Ψ Set of PEVs

Variables of auxiliary energy systems Equivalent discharging power of auxiliary energy systems in IDC i in tth time slot [kW] sci;t ; sdi;t Stored and released energy of TSSs in IDC i in tth time slot, respectively [kW] uci;t ; udi;t Charged and discharged power of UPS in IDC i in tth time slot, respectively [kW] gi;t Output of BGs in IDC i in tth time slot [kW] ei;t;r Charged power of PEV r in IDC i in tth time slot [kW]

εi;t

Variables of power systems PLI;J;t Active power flow from node I to node J in tth time slot [kW] PGI;t Active power generated at node I in tth time slot [kW]

Parameters pothers Other equipment (e.g., lighting and power facilities) power i consumption in IDC i [kW] Mi Total number of available servers in IDC i μi Average service rate of servers in IDC i [request/s] μi;v The service rate of server v in IDC i [request/s]

PDI;t

Ploss t

implemented by ISCs, such as Amazon’s EC2 [5]. In addition, links among geo-IDCs also exist. The workload, including the backup service of an IDC, which comes from a certain IDC and needs to be allocated to the target IDCs, is transferred through these links [6]. That is, the workload can be transferred among IDCs. Occasionally, the workload, such as the batch workload, has a longer tolerant service delay. That is, it

Active power demand at node I in tth time slot [kW] Total loss of active power in tth time slot [kW]

can be flexibly scheduled to any time slot before its deadline. Therefore, the amount of workload to be processed in a few IDCs in certain slots can be changed and controlled. In terms of IT equipment, there are three main types of electronic equipment hosted in a typical IDC: servers for data processing, switches for data communication, and storage equip­ ment for data storage. With dynamic voltage and frequency scaling 2

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

(DVFS) [7], dynamic cluster server configuration (DCSC) [8], virtual machine (VM) technique [9], and so on, the number and frequency of active servers can be dynamically controlled according to the demand of the computing resources, rather than keeping all servers running at their peak power. The number of active switches can be a positive to the number of active servers [10,11], and the power consumption of the storage equipment is generally assumed to be a constant. For more de­ tails regarding a network architecture and simulation environment of IT equipment, see Refs. [10,12]. IT equipment power consumption can be dynamically controlled by controlling the amount of workload to be processed. For a cooling system, three different cooling infrastructures can be used in an IDC, i.e., air cooling, liquid cooling, and immersion cooling [13]. With many technologies on cooling management, the power consumption of a cooling system can be positive to the heat to be extracted, which equals the heat generated by the IT and other equip­ ment, plus the heat transferred from outdoors, minus the difference between the total generated and transferred heat, and the extracted heat. For example, technologies on the layout planning of active servers [14]; driving fan technologies with a variable frequency in computer room air conditioning (CRAC) units, the key cooling component used in air cooling systems [15]; and technologies regarding the thermal-aware workload distribution in each IDC [16,17], can achieve such a goal. Therefore, cooling system power consumption can be dynamically controlled by controlling the heat to be extracted. Accordingly, different

from other flexible loads, and depending on the data network, IDCs have a potential temporal and spatial load regulation, which can act as extremely important demand response (DR) resources. Currently, the power grid has become a critical infrastructure that modern society cannot do without. It is a continuous challenge to keep the power supply and demand in balance, all the more so with the rise of intermittent RGs [18]. The DR, a change in the power consumption of electricity customers to better match the power supply [19], has been widely accepted as one of the key technologies of a smart grid. It is able to relieve the tension of a supply–demand imbalance, strengthen the grid’s ability to deal with power fluctuations, and improve the energy efficiency [20]. Owing to the enormous electricity consumption and spatio-temporal load regulation potential, IDCs are expected to have a major positive impact on power systems through DR programs. For example, geo-distributed IDCs can help power systems with distributed RGs to improve their robustness and load balancing. Meanwhile, DR programs provide ISCs the opportunity to lighten their huge financial burden regarding energy costs. Therefore, IDCs participating in DR has attracted increasing attention in the domain of IDCs and power systems in recent years. With the advanced communication capabilities of the smart grid, ISCs can accordingly receive effective tools to receive DR signals and participate in DR programs [13]. IDCs participating in DR can be further divided into two parts: the formulation of the IDC load characteristics,

Fig. 1. Structure of the paper. 3

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

which is illustrated in the upper-right part of Fig. 1, and IDCs partici­ pating in DR programs, an illustration of which is shown in the lower part of Fig. 1. As shown in the upper-right part of Fig. 1, the power demand constraints of IDCs can be obtained by considering the IDC load modeling, load regulation operations, and regulation considerations. IDCs can then participate in DR programs through two ways: as large users in wholesale electricity markets, and as general users in retail markets. Specifically, as described in the lower part of Fig. 1, DR pro­ grams can be classified into price-based DR programs and incentive-based DR programs. Such DR programs are designed based on the perspectives of the power systems including balancing electricity load. In price-based DR programs, such as real-time pricing (RTP) pro­ gram, ISCs provide their energy bids and then receive the clearing price when they are participating in wholesale electricity markets as large users. However, the price is determined by the contracts between the ISCs and the retailers/aggregators when they are general users in retail markets. This is similar in incentive-based DR programs, such as emer­ gency demand response (EDR) program. Note that most existing DR mechanisms are designed for flexible loads without the potential of spatial load regulation, which implies that spatial coupling is not considered in the bids of IDCs. In addition, regardless of whether large users or general users are considered, economic incentives between ISCs and their tenants regarding the participation of DR are necessary. To date, according to the degree of interaction between ISCs and the electricity markets, the DR of IDCs experiences three stages. The first stage is energy saving, where the DR of IDCs has no reference to the signals in electricity markets. After realizing an added high energy cost and the environmental impact in the form of greenhouse gas emissions [21], numerous energy-saving technologies and strategies have been carried out to find an efficient and sustainable way to operate IDCs, e.g., energy-saving technologies including DVFS [7], DCSC [8], and VM [9], and energy-saving strategies including joint control of the cooling sys­ tems and servers for holistic energy minimization [22–28]. As the sec­ ond stage, ISCs passively monitor the DR signals in electricity markets to reduce energy costs, whereas the impact of the spatial load regulation of IDCs on power systems is ignored by system operators (SOs). Since Qureshi et al. [29] first proposed that existing distributed systems of IDCs can exploit the temporal and geographic variation of electricity prices for significant economic gains in 2009, IDCs participating in DR programs, particularly price-based DR programs, has attracted signifi­ cant attention as another way to save the energy cost of ISCs. Afterward, numerous studies have expanded on Qureshi’s architecture of IDC responding to electricity prices by considering more elements, such as more diverse options for procuring electricity [30]. In the second stage, increasing attention has been paid to IDCs as special loads with the potential of both temporal and spatial load regulation. The third stage is the monitoring of DR signals by ISCs in electricity markets to reduce the energy cost; meanwhile, the impact of the spatial load regulation of IDCs on power systems is considered by the SOs. This impact includes a tur­ bulence in the electricity prices. Wang et al. [1], proposed a new pricing mechanism for IDCs to balance the electricity load without a turbulence in the electricity prices. In the third stage, the DR of IDCs improves the efficiency of power systems, while lowering the energy cost of ISCs. However, there have been fewer studies on the DR of IDCs during the third stage. Such questions as how much impact does it exactly have on a power system, and what kind of DR mechanism can properly guide the IDCs to regulate their load, remain unanswered. This means it does not make full use of the load regulation potential of IDCs to improve the efficiency of a power system. By contrast, the existing DR of IDCs may bring about a negative impact on the power system. Therefore, the issue of how SOs exploit the load regulation potential of IDCs to properly achieve the efficiency of a power system requires further attention from the research community. However, a review summarizing the state-of-art studies around the theme of IDCs participating in DR is first necessary. In addition, open research issues and challenges in the context of IDCs further participating in DR, particularly from the

perspective of power systems, also need to be investigated in detail. In previous studies, surveys on IDCs around different themes have been thoroughly considered. For instance, a survey on the technological drivers in IDCs from the perspective of multiscale thermal, electrical, and energy management is presented in Ref. [31]. Strategies on the energy efficiency and renewable energy integration in IDCs are surveyed in Ref. [32]. In addition, review on the energy management in IDCs based on geographic workload balancing is presented in Ref. [33]. The energy efficiency and DR in small and medium data centers is reviewed in Ref. [34]. However, a comprehensive survey from the perspective of the DR of IDCs used for power system operations has yet to be presented in the literature. The major contributions of this article are as follows: 1) A compre­ hensive survey focusing on the DR of IDCs for power system operations is presented, covering three parts: load modeling of IDCs, load regula­ tion operations and economic considerations in IDCs, and IDCs partici­ pating in DR programs. 2) Research issues and challenges for IDCs further participating in DR are given, which include three dimensions: the establishment of suitable IDC load models for power system opera­ tions, the expansion of the load regulation potential of IDCs, and the design of targeted DR mechanisms for spatial-coupling loads. The remainder of this paper is structured as shown in Fig. 1. The load modeling of IDCs is summarized in Section 2, followed by their load regulation operations and economic considerations in Section 3. Section 4 reviews studies on IDCs participating in DR programs. Finally, chal­ lenges and future research issues are addressed in Section 5, followed a few concluding remarks in Section 6. 2. Load modeling of IDCs Suppose an ISC having multiple geographically distributed IDCs, where each IDC is powered by a dedicated power substation in the power grid, with i 2 N denoting the set of geographically dispersed IDCs, and I 2 Π denoting the set of power system nodes, where each IDC i is located within node I. Suppose a discrete time model t 2 Τ, where the length of a time slot matches the time-scale at which the workload allocation decisions and power system dispatch decisions are updated, e. cool others g., once per hour [1]. Here, pi;t ; pIT ; εi;t denote the total power i;t ; pi;t ; pi consumption, IT equipment power consumption, cooling system power consumption, the power consumption of other equipment, and the equivalent discharge power of the auxiliary energy systems in IDC i in the tth time slot, respectively [kw], as described in Section 1: cool others pi;t ¼ pIT i;t þ pi;t þ pi

εi;t

(1)

In general, pothers is assumed as a constant, and εi;t is formulated as i their common models. In addition, pIT i;t is generally modeled as a function of the processed workload with several other controllable variables, such as the state and frequency of the servers. Considering the complexity and calculation requirements of the models, pcool i;t is generally simplified as a function of pIT i;t without other controllable variables. The

cool modeling of pIT i;t ; pi;t and the workload is summarized in Table 1, which is described in detail in the following.

2.1. Modeling workload Define δ 2 Φ as the set of front-end servers, and q 2 Ω as the set of

workload types. Three variables, i.e., Lt;δ;q ; λi;t;δ;q and λdropped , which i;t;q

denote the amount of workload q arriving at the front-end server δ, the amount of workload q allocated from the front-end server δ to IDC i, and the amount of dropped workload q in IDC i in the tth time slot, respec­ tively, are generally used to describe the workload. Two constraints, i.e., the workload balance constraint and the delay constraint, are typically used to describe the relationship between the workload scheduling and computing resources provided.

4

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

Table 1 Typical IDC load modeling in DR programs. Components

Notations

Modeling

Workload

Lt;δ;q , λi;t;δ;q ,

Workload balance constraint

P

Delay constraint

For delay-tolerant workload in off-line strategies: 8 XXtþDq =Δt λi;t;δ;q ¼ Lt;δ;q > t < i2N . X > : λi;t;δ;q λdropped � CRi;t;q i;t;q

[36]

For delay-tolerant workload in on-line strategies: Utilizing Lyapunov optimization technique. For delay-sensitive workload: 1 � Dq . P μi ð δ2Φ λi;t;δ;q λdropped Þ=mi;t i;t;q

[37]

λdropped i;t;q

Ref. i2N

PtþDq =Δt t

[6,35]

λi;t;δ;q ¼ Lt;δ;q .

δ2Φ

IT equipment

λi;t;δ;q;v ,

λdropped i;t;q;v Ii;t;v ,

Fi;t;v , mi;t , CRi;t , pIT i;t

Continuous execution constraint Bandwidth capability constraint Power consumption model

Ii;t;v ðmi;t Þ as controllable variable: PMi P pIT i;t ¼ α3;i v¼1 fIi;t;v ½α2;i ð δ2Φ λi;t;δ;1;v

Computing capability model

Cooling system

pcool i;t

Minimum ON/OFF time constraint Error-tolerant rate model Power consumption model

Other equipment

pothers i

Modeled as a constant.

Auxiliary energy systems

εi;t

Conversion

NA

No controllable variable: pIT i;t ¼ α1;i

Fi;t;v as controllable variable: PMi P pIT i;t ¼ α7;i v¼1 ½α6;i Fi;t;v ð δ2Φ λi;t;δ;1;v PMi CRi;t ¼ v¼1 ðα8;i;v Ii;t;v Fi;t;v Þ

i2N

t

λi;t;δ;q ¼ Lt;δ;q

λi;t;j;q ¼ Lt;j;q

λdropped i;t;1;v Þ þ β4;i �.

[36] [6] [37] [11,38] [39] [38,39] [36] [40] [41] [45] [35] [49] [38]

delay, a network delay is generally ignored in most related studies. Considering different deadlines, a workload is generally divided into two categories: 1) a delay-tolerant workload, the deadline of which is assumed to be more than one time slot, and 2) a delay-sensitive workload, the deadline of which is assumed to be less than one time slot. The queuing delay constraints of the two categories of workload are respectively modeled as follows. 1) Delay-tolerant workload: This can be flexibly scheduled to any time slot before its deadline. In each time slot, sufficient computing re­ sources should be provided to process the workload. As in Ref. [36], the queuing delay constraints are modeled as (2) and (5), respectively. X λi;t;δ;q λdropped � CRi;t;q (5) i;t;q

(2)

δ2Φ

where CRi;t;q denotes the provided computing resources for pro­ cessing workload q in IDC i in the tth time slot, which is described in detail in sub-section 2.2.

In addition, as described in Section 1, when the workload is from one IDC that must be allocated to other certain IDCs, the front-end server is replaced by the IDC. As in Ref. [6], (2) can be rewritten as follows: t

λdropped i;t;1;v Þ þ β2;i �g.

Modeled as their common models. X � ½ðpi;t β7;i Þ=α11;i � � β8;i;t . i2N β7;i � pi;t � β9;i

i2N

i2N

λdropped Þ þ β1;i . i;t;1

Models accurate to specific refrigeration equipment.

For a delay-sensitive workload, the tolerant service delay of which is generally in [ms] or [s], and the integral with respect to time is generally ignored. As in Ref. [35], (2) can be rewritten as follows: X λi;t;δ;q ¼ Lt;δ;q (3)

XXtþDq =Δt

δ2Φ ðλi;t;δ;1

cool Models accurate to pIT ¼ α10;i;t pIT i;t : pi;t i;t þ β5;i;t

(1) Workload balance constraint: This is the most important constraint when the workload is migrated among different IDCs, which means the total amount of workload allocated from each front-end server to the IDCs should equal the amount of workload arriving at the front-end server. Defining Dq as the tolerant ser­ vice delay of workload q [s], the workload balance constraint can be described as (2). XXtþDq =Δt

P

[25,38]

In addition [37], utilizes a Lyapunov optimization technique rather than (2) or (5) to ensure that a delay-tolerant workload can be processed before its deadline, which is suitable for on-line strategies.

(4)

2) Delay-sensitive workload: Queueing models are generally adopted to model the queueing delay within one time slot. For example, a M/M/ 1 queueing model is adopted in Refs. [25,38], which is described as (6), where it is assumed that the servers in each IDC are homogeneous.

(2) Delay constraint: To satisfy the QOS, sufficient computing re­ sources should be provided to process all types of workloads within their deadlines. The delay generally comprises two com­ ponents: a network delay, which is experienced while the work­ load is outside of the IDCs, and a queueing delay, which is experienced while the workload is at the IDCs. In Ref. [35], the network delay is defined as a constant proportional to the dis­ tance. Given that it is relatively small compared with a queueing

μi

1 λ i;t;δ;q δ2Φ

P

λdropped i;t;q

�� � Dq mi;t

(6)

where μi denotes the average service rate of servers in IDC i, and mi;t denotes the number of active servers in IDC i in the tth time slot.

5

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

In addition to the above, a continuous execution constraint, i.e., a batch workload may require a continuous execution time block, as considered and modeled in Ref. [36]. A bandwidth capability constraint, i.e., a situation in which the total workload passed in each time slot cannot exceed the maximum bandwidth capability, is considered and modeled in Ref. [6]. For more details, please [6] and the references therein.

a linear function of the workload processed on it and its fre­ quency. This is modeled as (10), where Fi;t;v is a controllable variable, whereas the operation status of the servers is a known quantity rather than a controllable variable therein. (2) Computing capability model: The computing resources in each IDC are limited, which is generally modeled as the total computing capability of the servers. The computing capability of server v is generally represented by μi;v , which denotes the service rate [request/s]. In addition, μi;v is related to Fi;t;v . Define CRi;t as the available computing capability in IDC i in the tth time slot. Then, CRi;t can be modeled as a function of Ii;t;v ðmi;t Þ and Fi;t;v , which is described as (11). It is degenerated into a function of Ii;t;v ðmi;t Þ in Ref. [38], which is shown in (12), and as a function of Fi;t;v in Ref. [39], which is shown in (13). XMi CRi;t ¼ ðα8;i;v Ii;t;v Fi;t;v Þ (11) v¼1

Note that λi;t;δ;q ; λdropped are controllable variables in these models, i;t;q P where λdropped can be no more than δ2Φ λi;t;δ;q . i;t;q 2.2. Modeling IT equipment

Define v 2 Θ as the set of servers. Five variables, i.e., Ii;t;v ; Fi;t;v ; mi;t ; CRi;t , and pIT i;t , are generally used to describe IT equipment, where Ii;t;v denotes the operation status of server v, Fi;t;v denotes the frequency of server v, CRi;t denotes the total computing resources provided for pro­ cessing the workload in IDC i in the tth time slot [request/s]. Two models, namely, a power consumption model and a computing capa­ bility model, are generally used to describe the relationship between the workload scheduling and pIT i;t .

(12)

CRi;t ¼ μi mi;t CRi;t ¼

XMi v¼1

ðα9;i;v Fi;t;v Þ

(13)

where α8;i ; α9;i , which are empirical constants, have positive values.

(1) Power consumption model: In general, there are three types of power consumption models of IT equipment according to the controllable variables, which are modeled as follows. X � λi;t;δ;1 λdropped þ β1;i pIT (7) i;t;1 i;t ¼ α1;i

In addition to the above, the minimum ON/OFF time constraint, i.e., in which it is desirable to keep a server ON or OFF for a minimum period of time after it is woken up or shut down because the wakeup and shutdown of a server cannot be applied too frequently for the reliability of the server, is considered and modeled in Ref. [36]. The error-tolerant rate model is considered and modeled in Ref. [40], where μi;v is multi­ plied with the fraction of the server failure.

δ2Φ

where α1;i ; β1;i , which are empirical constants, have positive values. ( " ! #) X XMi dropped I þ β (8) ¼ α α λ λ pIT 3;i i;t;v 2;i i;t;δ;1;v 2;i i;t;1;v i;t v¼1

2.3. Modeling cooling system

δ2Φ

Owing to the law of energy conservation, pcool is a function of the i;t power consumption of the IT equipment and the external heat. Considering the complexity and calculation requirements of the models, pcool i;t is generally simplified as a function of the power consumption of the IT equipment without other controllable variables. In general, there are two types of power consumption models of a cooling system according to the assumed simplification, which are respectively modeled as follows.

where Mi is the total number of available servers in IDC i, and α2;i ; α3;i ; β2;i , which are empirical constants, have positive values. 1) No controllable variable: This approach is generally used in related studies and is generally modeled as the linear function of the processed workload. As in Ref. [37], it is modeled as (7), where it is assumed that one kind of workload is applied. 2) Using Ii;t;v ðmi;t Þ as a controllable variable: Zhang et al. [11], modeled pIT i;t as a linear function of the total power consump­ tion of the servers, and modeled the power consumption of each server as the linear function of the workload processed on it, where the workload processed on server v can be P described as ð δ2Φ λi;t;δ;q;v λdropped i;t;q;v Þ. It is modeled as (8),

(1) Models accurate to pIT i;t : Such models have typically been used in related studies. Assume there is a local optimizer that can dynamically allocate active servers and cooling infrastructures to even out the indoor temperature in each IDC. As in Ref. [41], this is directly modeled as a function of pIT i;t , which is described as (14).

where Ii;t;v is the controllable variable. In addition, under the assumption that the servers in IDC i are homogeneous, Chen et al. [38] simplified controllable variable Ii;t;v to mi;t , which is modeled as (9). " # X � dropped λ þ β pIT (9) ¼ α m α λ 5;i i;t 4;i i;t;δ;1 3;i i;t;1 i;t

IT pcool i;t ¼ α10;i;t pi;t þ β5;i;t

(14)

where α10;i;t ; β5;i;t , which are empirical constants, have positive values, and are related to the energy efficiency of the IDCs, the initial temperature set-point, and the outdoor climate.

δ2Φ

where α4;i ; α5;i ; β3;i , which are empirical constants, have positive values. " ! # XMi X dropped IT þ β4;i (10) pi;t ¼ α7;i v¼1 α6;i Fi;t;v λi;t;δ;1;v λi;t;1;v

In particular, α10;i;t is represented by ðPUEi 1Þ, and β5;i;t equals zero in Ref. [41]. The power usage effectiveness (PUE) is the most commonly used descriptor of the energy efficiency of the IDCs initiated by a Green Grid [42], representing the ratio of the total power consumption of the IDCs to the power consumption of the IT equipment. PUE values may be quite different in various IDCs, which is partly accounted for by different cooling system designs [43] as well as the local climate represented by the ambient temperature, humidity, and wind speed [8,36,44,45]. In addition, α10;i;t ; β5;i;t vary with the change in the outdoor climate. For example, α10;i;t is defined as 1=LFi;t in Ref. [36], where LFi;t denotes the ratio of the power consumption of the IT equipment to the cooling

δ2Φ

where α6;i ; α7;i ; β4;i , which are empirical constants, have positive values. 3) Using Fi;t;v as a controllable variable: Li et al. [39], modeled pIT i;t as the linear function of the total power consumption of the servers, and modeled the power consumption of each server as 6

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

system power consumption in IDC i, which is changed with the time and place.

Table 2 Typical load regulation operations and economic considerations in IDCs participating in DR programs.

(2) Accurate models for the specific refrigeration equipment: In a few studies, particularly in the heating, ventilation, and air condi­ tioning (HVAC) domains, power consumption models of the cooling system are generally accurately modeled to specific refrigeration equipment. In these models, pcool is generally i;t modeled as the sum of the total power consumption of the refrigeration equipment, and the power consumption of each refrigeration equipment is modeled as a function of the power consumption of the IT equipment in each rack, the temperature of the cooling air supplied, and so on. Theories of thermodynamics, and the coefficient of performance (COP), which is an efficiency metric for CRAC, are generally adopted in the modeling of the power consumption of the refrigeration equipment. For more detailed and accurate modeling of the specific refrigeration equipment, please see Refs. [45–47] and [48], and the references therein.

Load regulation operations Associated with scheduling workload

MIW

DW MIW/ SW/DW

Associated with changing indoor temperature set-point of cooling system Associated with utilizing auxiliary energy systems

2.4. Conversion of IDC load model for power system operations Considering that general IDC load models are represented by work­ load and servers as described above, which may puzzle SOs when they apply them to a specific power system operation, a few researchers have converted the general load model into a clearer and simpler version. For example, Chen et al. [38] converted the load model represented by the workload and servers into (15), where the first part represents the spatial coupling among the geo-IDCs, and the second part represents the range power consumption of each IDC. Compared with the general case, (15) has the following advantages: 1) It is made up of linear formulas, which can be better embedded in a specific power system operation and reduce the computing pressure. 2) The controllable variable, i.e., pi;t , is electrical, and the load characteristics of the IDCs are demonstrated explicitly, which makes the IDC load model easier to be understood by the SOs. 3) It is an encapsulated model, which helps protect the privacy of the ISCs. X� �� � � pi;t β7;i α11;i � β8;i;t (15) i2N β7;i � pi;t � β9;i

TSTI

Economic considerations

Ref.

Bandwidth cost

[49, 67] [50] [35] [37]

Extra energy cost Extra delay cost Cost of dropping workload Queueing delay cost Startup cost of servers Cost of environmental impact NA

TSSs UPS PEVs

Depreciation cost

BGs RGs CCHP systems

Generation cost

[35] [50] [50] [53] [49] [37] [41, 68] [37] [58] [64]

the tenants can obtain the best services, with 4), IT equipment can operate under the best ambient temperature, and with 5), ISCs require the minimum investment cost. The ideas regulating the power demand of IDCs can be grouped into three categories when compared with the base state. Specifically, 1) scheduling workload, regulating the power consumption of the IT equipment, and the power consumption of the cooling system, 2) changing the indoor temperature set-point, regulating the power con­ sumption of the cooling system, and 3) utilizing auxiliary energy sys­ tems, regulating the power demand of IDCs from the power systems. These are summarized in detail as follows. 3.1.1. Operations associated with scheduling workload (1) Migrating workload (MIW) [35–41]: This is a technology by which the workload can be allocated to various IDCs by front-end servers. It is generally conducted using geographical workload balancing technology and is particularly suitable for a delay-sensitive workload, which has higher QOS and lower computing requirements. It enables IDCs to achieve a spatial load regulation potential and has been used in most strategies of IDCs participating in DR programs. (2) Shifting workload (SW) [44,50–52]: This is a technology by which the workload can be flexibly scheduled to any time slot before the deadlines, and is suitable for a delay-tolerant work­ load, with a lower QOS requirement. It enables IDCs to achieve a temporal load regulation potential. (3) Dropping workload (DW) [37,41,44]: This is a technology by which the computing requirement can be reduced by dropping some of the workload. It is often used in search engines, financial calculations, and simulations, where approximate results can be accepted.

where α11;i represents the dynamic power consumption processing unit workload in IDC i; β7;i ; β9;i represent the power consumption limits of IDC i; and β8;i;t represents the amount of total workload arriving in IDC i in the tth time slot. 3. Load regulation operations and economic considerations in IDCs The load regulation operations and economic considerations in IDCs are summarized in Table 2 and are described in detail in the following. 3.1. Load regulation operations The power demand without considering the DR signals is defined as the baseline of the power demand of IDCs. In general, the base state can be summarized as follows: 1) The workload is processed as closely as possible. That is, the workload will be allocated to the nearest IDC if it can be processed in any IDC. 2) The workload is processed as quickly as possible. That is, the workload will be processed at once if sufficient computing resources are available. 3) The workload is processed as much as possible. That is, none of the workload is dropped if the computing resources are sufficient. 4) The indoor temperature in each IDC remains at the initial temperature set-point. 5) Power is supplied only by power systems. It is a responsible approach because with 1–3),

3.1.2. Operation associated with changing indoor temperature set-point of cooling system Thermal storage operation utilizing thermal inertia of buildings (TSTI) [53]: This is a technology by which the power consumption of the cooling system can be changed, and the ambient temperature can maintain the requirement of the IT equipment. In Ref. [54], it is sug­ gested that the number of HVAC units needed varies significantly with 7

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

the baseline, high and low indoor temperature settings, and the outdoor temperatures. That is, the power consumption of the cooling system can be changed by controlling its indoor temperature set-point. Further­ more, considering that the cooling power can be stored in an inherent thermal mass, such as over-cooled air and a raised metal floor, they can be over-cooled to a lower temperature by the cooling system first, where the indoor temperature set-point is lower and the power consumption of the cooling system increases. They then absorb heat as a cooling unit, which changes the amount of heat to be extracted by the cooling system, and decreases the power consumption of the cooling system, that is, the process of TSTI changing the power consumption of the cooling system. Considering that IDCs are equipped with a significant cooling system, the TSTI can significantly improve the temporal load regulation poten­ tial of each IDC.

cost of utilizing auxiliary energy systems, which are detailed in the following. 3.2.1. Cost of scheduling workload A dynamic control of the server status when regulating the power demand of IDCs through scheduling workload is unavoidable. Therefore, in addition to the costs of migrating or dropping the workload, a queueing delay cost and startup cost of servers resulting from a dynamical control of the servers are also considered. In addition, considering that the electricity used by geo-distributed IDCs may be under distinct carbon emission levels, such as the electricity generated by traditional generators and RGs, the cost of the environmental impact can also be attributed to the scheduling workload. (1) Cost of migrating workload: 1) Bandwidth cost: Owing to the spatial diversity, traffic between different pairs of front-end servers and IDCs goes through different links, which incurs different bandwidth costs. This is generally modeled as the product of the bandwidth cost co­ efficient and the amount of migrated workload [49]. In another study, Tran et al. [67] externalized the bandwidth cost coefficient as the transmission delay from the front-end servers to the IDCs, which is proportional to the distance and is assumed to be a constant. The bandwidth cost is finally formulated as a quadratic function of the amount of migrated workload. 2) Extra energy cost: The workload can be migrated among geodistributed IDCs using optical fiber technology. Compared with the energy consumed by the optical fiber interface, the energy consumed by the optical transceivers and optical ca­ bles is relatively small and generally ignored. Chen et al. [50], modeled the extra energy cost as the product of electricity prices and the electricity consumption of an optical fiber interface, i.e., wavelength division multiplexing. 3) Extra delay cost: The delay cost is incurred by the delay experienced by the workload and generally consists of two parts: a) the network delay when the workload is outside the IDCs, i.e., an extra delay cost, and b) the queueing delay when the workload is at the IDCs. Wierman et al. [35] modeled the extra delay cost as a product of the network delay and the amount of processed workload, where the network delay is assumed to be a constant, which is proportional to the distance between the front-end servers and IDCs. (2) Cost of dropping the workload: Yu et al. [37] modeled the cost of dropping the workload as a product of the penalty factor and the amount of workload dropped. (3) Queueing delay cost: The queueing delay cost is generally modeled as the product of the queueing delay and amount of processed workload. Wierman et al. [35] modeled the queueing delay strictly decreasing with the active servers, strictly increasing with the processed workload, and strictly convex with the active servers and the processed workload. (4) Startup cost of servers: Considering that the startup/shutdown of the servers impacts their lifetime and reliability, Chen et al. [50] considered the startup cost of the servers and modeled it as a constant. (5) Cost of environmental impact: If a carbon tax is implemented on generators, the cost of the environmental impact could be passed on to electricity consumers through locational marginal prices (LMPs). However, the general policy is still under development and is not yet popular [50]. Therefore, to evaluate the environ­ mental impact of geo-distributed IDCs, Chen et al. [50] modeled the cost of the environmental impact as a product of the carbon emission rate and the electricity consumed by each IDC.

3.1.3. Operations associated with utilizing auxiliary energy systems (1) Thermal storage systems (TSSs) [49,55]: TSSs commonly use chilled liquid or ice to act as a thermal battery. IDCs can chill liquid at night (when the electricity prices are lower), and during the day, pump the chilled liquid around the facility for cooling. The operation of a TSS can reduce the power consumption of IDCs during periods of peak prices. (2) Power storage systems [30,44,56]: Power storage systems commonly use an uninterrupted power supply (UPS). The UPS used in IDCs ensures the stability of the power supply, with the capacity maintaining the operation of an IDC for approximately 5–10 min, which is generally used after the main power supply is interrupted and before backup generators (BGs) are started. Under the premise of ensuring the safety of the power supply, the charging and discharging operation of a UPS can reduce the power consumption of the IDCs during the period of peak prices and peak workload, although frequent charge and discharge op­ erations will reduce its lifetime. (3) BGs [37,51,57]: BGs commonly use diesel generators, ensuring the stability of power supply, and are generally used after the main power supply is interrupted. The operation of BGs can reduce the power consumption of IDCs during periods of peak prices. However, these generators typically have higher genera­ tion costs and are not clean. (4) RGs [58–63]: To address the challenges posed by the energy consumption on the development of IDCs, ISCs are starting to build their own local green power plants near IDCs, such as wind power plants and photovoltaic power plants. However, the elec­ tricity generated by such devices is generally intermittent and uncertain. (5) Combined cooling, heating, and power (CCHP) systems [64]: A CCHP system, which realizes an efficient energy utilization, can provide a region with cooling, heating, and power through various primary energy conversion technologies. A CCHP system can also power IDCs where the local power supply is tight. (6) Plug-in electric vehicles (PEVs) [39,41,65]: Through the joint management of IDCs and PEVs, the load volatility and power consumption during the period of peak price can be reduced. Note that a profit distribution (or cost distribution) may arise during a joint management. 3.2. Economic considerations The revenue of ISCs is from processing the workload [66]. ISCs can also obtain revenue in electricity markets when they regulate their power demand regarding price or regulation signals [39,50]. However, the economic costs of regulating the power demand should also be considered. Considering that the TSTI is rarely used in multiple IDCs, the economic considerations, which are generally considered, generally have two categories: 1) the cost of the scheduling workload and 2) the 8

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

3.2.2. Cost of utilizing auxiliary energy systems It is known that the charging and discharging of energy storage systems affect their lifetime and reliability, such as TSSs [49], UPS [37], and PEVs [41,68]. In general, their depreciation cost can be modeled proportionally to the charged energy with a depreciation cost coefficient. In terms of utilizing their self-contained generators, such as BGs [37], RGs [58], and CCHP systems [64], investment and generation costs are incurred. In general, their generation cost can be modeled as a function of the energy produced.

does not mean that regulating the power demand incurs losses of the ISCs, which will be described in detail in section 4.2. This is formulated as a MIP problem, which is directly solved using business optimization tools. The proposed strategy can reduce the total operation cost of a power system. XXh� � � �� i (18) F G PGI;t þ F IDC pi;t Δt min G PI;t ;λi;t;δ;1 t2Τ I2Π

where FG ð ⋅Þ; FIDC ð ⋅Þ are the cost function of the generation and IDC load control, respectively. Unlike the above studies, Chen et al. [38], before modeling the day-ahead economic dispatch, conducted a conservation on the IDC load model through (15) to make it more suitable for a power system oper­ ation. The day-ahead economic dispatch is finally formulated as an MIP program, with the aim of minimizing the generation cost of the power systems, the objective function of which is shown in (19). The control­ lable variables are PGI;t and pi;t . In addition, the sensitivity of the regu­ lation potential of the IDCs to Dq was also exactly derived. The proposed strategy has a positive significance for the power grid companies owing to a reduced operation cost, which also verifies that the regulation po­ tential of the IDCs increases with an increase in Dq , and the sensitivity of the regulation potential to Dq decreases with an increase in Dq . XXh � � i min FG PGI;t Δt (19) G

4. IDCs participating in DR programs Existing studies on IDCs participating in DR programs are presented in three parts in this section: 1) power system perspectives, which refer to economy dispatch on IDCs, 2) interactions between SOs and ISCs, which include IDCs participating in incentive-based and price-based DR programs, and 3) incentive mechanisms between ISCs and their tenants. Table 3 shows several typical studies. 4.1. Power system perspectives This refers to economic dispatch studies considering the load regu­ lation potential of IDCs, which can help evaluate the load regulation potential of IDCs and their economic benefits to power systems. In general, the decision variables are the output and the power demand of the IDCs. The constraints generally comprise power demand constraints of the IDCs and conventional power system operation constraints, such as the power flow constraints, voltage constraints, ramping constraints, and so on. A few studies with different objectives and controllable var­ iables are reviewed in this section, where MIW is the only load regula­ tion operation adopted. Mohsenian-Rad et al. [69], proposed a day-ahead economic dispatch model with the aim of balancing the electrical load, the objective function of which is shown in (16). The controllable variables are PGI;t

PI;t ;pi;t

4.2. Interactions between SOs and ISCs Most studies on IDCs participating in DR programs have focused on the interactions between SOs and ISCs, particularly load regulation strategies for IDCs in price-based DR programs. Related studies with different DR scenarios and load regulation strategies are reviewed in this section. 4.2.1. IDCs participating in incentive-based DR programs Most existing incentive-based DR programs in electricity markets, such as an EDR, have not considered loads with the potential of spatial load regulation. Therefore, DR bids of IDCs lack spatial coupling load characteristics. Related studies have focused on how IDCs bid their regulation amount in incentive-based DR programs. In terms of real-time load control, they generally aim to minimize the difference between the regulation signals and the real power consumption, as well as minimize the regulation costs. Several studies with different goals are reviewed in this section. Li et al. [39], proposed a framework of joint energy management of a single data center and PEVs for frequency regulation. A strategy, determining the best baseload and capacity (regulation up/down) values over a multi-hour operating period to minimize electricity fees and maximize regulation service revenues, was developed. The controllable variables are Fi;t;v ; uci;t ; udi;t and ei;t;r , where uci;t ; udi;t denote the charged and discharged power of UPS in IDC i in the tth time slot, respectively [kW], and ei;t;r denotes the charged power of PEV r 2 Ψ in IDC i in the tth time slot [kW]. The proposed strategy has brought eco­ nomic benefits to both the data center and PEV owners. Note that the research object is a single data center, where a spatial load regulation is not considered. In another study conducted by Chen et al. [50], the research object is a set of geo-distributed IDCs, where the spatial load regulation of the IDCs is considered. The DR capability of the IDCs is rigorously formu­ lated as changes in the electricity consumption when migrating the workload among the distributed IDCs in different time zones. The pro­ posed strategy determines the optimal hourly DR capabilities of indi­ vidual IDCs when considering an uncertain arrival workload, where MIW and SW are adopted, and the controllable variables are λi;t;δ;q and Ii;t;v . This is formulated as a stochastic optimization problem for

and λi;t;δ;1 , where PGI;t denotes the active power generated at node I in the tth slot. This is formulated as a min-max problem, which is solved by transferring into an equivalent linear programming (LP). The proposed strategy significantly helps in load balancing the power grid and makes the grid more reliable and robust with respect to a link breakage and variations in the load demand. . � � min : max PLI;J;t PLI;J max (16) G PI;t ;λi;t;δ;1

where PLI;J;t is the active power flow from node I to node J in the tth slot,

and PLI;J max is the active power flow limit from node I to node J. Wu et al. [70], proposed a day-ahead economic dispatch model with the aim of reducing the power loss, the objective function of which is shown in (17). The controllable variables are PGI;t ; Fi;t;v and λi;t;δ;1 . This is formulated as a mixed integer programming (MIP), which is solved using a particle swarm algorithm. The proposed strategy has a positive significance for power grid companies owing to a reduced power loss. � i X � XXh� G min Ploss PI;t PDI;t Δt (17) t Δt ¼

PG ;F ;λ I;t i;t;v i;t;δ;1 t2Τ

t2Τ I2Π

t2Τ I2Π

where Ploss is the active power loss in the tth slot, and PDI;t is the active t power demand at node I in the tth slot, which includes the power de­ mand of the IDC located within node I. Cao et al. [71], with an aim to minimize the total cost of generation and load control, established the objective function of a day-ahead economic dispatch as (18). The controllable variables are PGI;t and λi;t;δ;1 . The cost of controlling the IDC load therein is the reward given by the SOs to the ISCs owing to the power demand regulation. Note that this 9

10

RTP, which is affected by IDC power demand

RTP, which is affected by IDC power demand

Regulation markets; RTP, which is affected by IDC power demand

Designing pricebased DR programs & Participating in

Designing pricebased DR programs & Participating in

Participating in incentive-based DR programs & price-based DR programs

Spatial & Temporal

Spatial

Spatial

Spatial

Spatial

Spatial & Temporal

MIW; SW

MIW

MIW

MIW

MIW

MIW; UPS; RGs

MIW; TSSs

MIW; SW; DW; UPS; BGs; RGs

Spatial & Temporal

Spatial & Temporal

MIW,

Spatial

MIW; DW; PEVs

MIW; SW; DW; UPS; BGs; RGs

Spatial & Temporal

Spatial & Temporal

MIW

MIW

Load regulation operations

Spatial

Spatial

Load regulation

Extra energy cost; Server startup cost; Environmental impact cost

Bandwidth cost

NA

NA

NA

NA

Dropping workload cost; Server startup cost; Battery depreciation cost; Generation cost of BGs Bandwidth cost; TSS depreciation cost

Queueing delay cost; Dropping workload cost; PEV depreciation cost NA

Dropping workload cost; Battery depreciation cost; Generation cost of BGs

NA

Queueing delay cost

Regulation costs considered

In the above table, gi;t denotes the output of the BGs in IDC i in the tth time slot.

RTP, which is affected by IDC power demand

RTP, which is assumed as a constant

Participating in price-based DR programs

Participating in price-based DR programs

RTP, which is assumed as a constant RTP, which is assumed as a constant

Participating in price-based DR programs Participating in price-based DR programs

RTP, which is affected by IDC power demand

RTP, which is assumed as a constant

Participating in price-based DR programs

Participating in price-based DR programs

RTP, which is assumed as a constant RTP, which is assumed as a constant

Participating in price-based DR programs Participating in price-based DR programs

RTP, which is assumed as a constant

NA

Economic dispatch

Participating in price-based DR programs

DR scenarios

Topic

Table 3 Typical researches on IDCs participating in DR programs.

λi;t;δ;q ; Ii;t;v

λi;t;δ;q ; mi;t

λi;t;δ;q ; mi;t

λi;t;δ;q

λi;t;δ;q ; Ii;t;v ðmi;t Þ

uci;t ; udi;t

λi;t;δ;q ; mi;t ;

sci;t ; sdi;t

mi;t ;

λi;t;δ;q ;

gi;t

uci;t ; udi;t ;

λdropped ; i;t;q

λi;t;δ;q ;

λi;t;δ;q ; mi;t

ei;t;r

λdropped ; i;t;q

λi;t;δ;q ;

gi;t

uci;t ; udi;t ;

λdropped ; i;t;q

λi;t;δ;q ;

λi;t;δ;q ; mi;t

λi;t;δ;q ; mi;t

Controllable variables

Minimize operation cost: electricity fee þ regulation costs DR reward

Minimize operation cost: electricity fee þ regulation costs

Minimize electricity fee

Minimize electricity fee

Maximum profit: profit of processing workload electricity fee Minimize electricity fee

Minimize operation cost: electricity fee þ regulation costs

Minimize operation cost: electricity fee þ regulation costs

Minimize operation cost: electricity fee þ regulation costs Minimize electricity fee

Minimize operation cost: electricity fee þ regulation costs

Minimize operation cost: electricity fee þ regulation costs Minimize electricity fee

Objectives of ISCs

NA

Balance electricity load

Balance electricity load

NA

NA

NA

NA

NA

NA

NA

NA

NA

Minimize generation cost

Objectives of DR programs

MILP

Non-convex, which is a twostage Stackelberg game

Non- convex, which is a bilevel quadratic program

Non-convex

Non-convex

Non-convex

Non-convex

MILP

Non-convex

Non-strictly convex

Non-convex

MIP

MIP

Problems formulation

Solved by business optimization

Solved by designed distributed algorithm, where initial problem is decomposed into strictly convex sub-problems Solved by branch and bound algorithm; Solved by designed heuristic algorithm, where initial problem is decomposed into convex sub-problems Solved by designed iterative and distributed algorithm, where initial problem is decomposed into convex sub-problems

Solved by business optimization tool

Solved by designed online algorithm, where initial problem is decomposed into strictly convex subproblems Solved by transferring initial problem into MILP

Solved by business optimization tool

Solved by designed heuristic algorithm

Solved by designed distributed algorithm, where initial problem is decomposed into strictly convex sub-problems Solved by decomposing initial problem into strictly convex sub-problems

Solved by business optimization tool

Solved by business optimization tool

Algorithms

IDC power demand over time and space flattened; ISC electricity fee reduced. ISC operation cost reduced.

Power grid reliability improved; ISC electricity fee reduced.

Electricity fee reduced by 33.5%; A cost budget on electricity fee can be enforced. Electricity fee reduced.

High-risk energy choices, such as DR, can be utilized.

An explicit tradeoff between cost saving and QOS can be obtained.

ISC operation cost reduced.

ISC operation risk reduced.

ISC operation cost reduced by 5.324%.

Electricity fee reduced by more than 6%. ISC operation cost reduced.

Generation cost reduced by 1.5%.

Simulation results

[50]

[67]

[1]

[73]

[11]

[30]

[49]

[44]

[72]

[68]

[37]

[8]

[38]

Ref. M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

minimizing the total energy cost, which is finally transferred into a mixed integer linear programming (MILP) problem by adopting a scenario-based approach through a Monte Carlo simulation. The pro­ posed strategy enhances the financial situation of the ISCs and improves the environmental impact of distributed IDCs. Unlike the above studies, Basmadjian et al. [18], introduced Green­ SDA and GreenSLA contracts rather than using a modeling to make the IDCs fit a DR. It has been suggested to consider IDCs as candidates for DR because they are large electricity consumers and are able to sufficiently adapt their power profile. To unlock this potential, contracts, collabo­ ration and economic incentives between SOs and ISCs (GreenSDA), as well as ISCs and their tenants (GreenSLA), have been proposed. Note that the research object is also a single data center, where the spatial load regulation is not considered.

the IDCs is contained within a certain range and enforces the dynamic prices across all locations to have an average price ceiling. This can prevent the aggregator from charging the maximum possible price in all locations and guarantee that the dynamic price will not arbitrarily in­ crease the electricity fee of the ISCs. 2) A set of geo-IDCs powered by different power systems. Tran et al. [67] proposed a pricing scheme constructed based on a two-stage Stackelberg game. Each aggregator sets a real-time price to maximize its own profit during Stage I, and the ISC then minimizes its electricity fee through a workload allocation and server control during Stage II. In Stage I, a unique Nash equilibrium was shown to exist. In Stage II, an iterative and distributed algorithm, which can converge to equilibrium, is proposed. The proposed pricing scheme can bring about the “right prices” for the “right demands”.

4.2.2. IDCs participating in price-based DR programs Several notable studies from multiple fields (such as information engineering, computer engineering, and electrical engineering) with different price-based DR program scenarios and load regulation strate­ gies are reviewed in this section, where the spatial load regulation of IDCs are generally considered.

4.2.2.2. Load regulation strategies in price-based DR programs. In general, in deregulated markets, the electricity price varies with time, and space is one of the most important factors considered by the ISCs when they are committed to reduce their energy costs, whether or not they participate in an incentive-based DR program [50]. We review the related load regulation strategies along with three dimensions:

4.2.2.1. Price-based DR programs. (1) Price-based DR programs in the wholesale electricity market

(1) Load regulation operations adopted and their economic considerations

In general, RTP, determined by all participants, is adopted as a DR program in most studies, which can be divided into two types of scenario:

To simplify the model, MIW is generally adopted independently, such as in Refs. [1,67,73], and [74], where the queueing delay cost is considered in Ref. [73] and the bandwidth cost is considered in Ref. [67]. With further development of such research, the joint man­ agement of multiple load regulation operations are considered, along with multiple economic considerations. For example, Zhang et al. [11] adopted MIW and DW, Li et al. [36] adopted MIW and SW, and Ghamkhari et al. [30] adopted MIW, UPS, and RGs, without considering the extra regulation costs. In addition, Chen et al. [50] adopted MIW and SW, which considered the costs of extra energy, server startup, and environmental impact. Wang et al. [58] and Kiani et al. [66] adopted MIW and RGs, with both considering the generation cost of the RGs. Liu et al. [64] adopted MIW and CCHP, and considered the generation cost of CCHP systems. Guo et al. [49] adopted MIW, SW, TSSs, and RGs, which considered the bandwidth cost and depreciation cost of the TSSs. Li et al. [44] adopted MIW, SW, DW, UPS, and BGs, which considered dropping the workload cost, startup cost, battery depreciation cost, and generation cost of conventional generators. Yu et al. [68] adopted MIW, SW, DW, and PEVs, which considered the queueing delay cost, dropping workload cost, and depreciation cost of the PEVs.

1) The power demand of IDCs is relatively too small to affect the clearing price of the market. In general, the real-time price is assumed as a known constant, which can be predicted by the ISCs [30,49,68,72]. 2) The power demand of IDCs is relatively large, thereby affecting the clearing price of the market. That is, ISCs are price-makers in deregulated electricity markets. In general, the real-time price is modeled as a linear function of pi;t [11]. However, it is generally challenging to obtain price impact models when taking into account the dynamic characteristics (e.g., time-varying power demand and supplies) and actual physical constraints (e.g., active/reactive power flow balancing, transmission congestion, and power loss) of power grids. Therefore, Yu et al. [73] defined the price-sensitivity coeffi­ cient as the ratio of the percentage change in price to the percentage change in power demand of IDCs, which can be obtained by exploiting a large amount of interactive information between the ISCs and main grids. A price-sensitivity aware workload scheduling scheme was also proposed.

(2) Problem formulation

(2) Price-based DR programs from aggregators

It is generally assumed that the arriving workload and electricity prices can be forecasted accurately. This objective is generally formu­ lated to minimize the costs, which include electricity fees and regulation costs. Weight factors are generally to set for different costs when mul­ tiple costs are considered [37]. In addition, considering that the purpose of ISCs is to make a profit, a few studies have formulated their objectives to maximize such profit, where the revenue of the processing workload is generally considered [30]. The controllable variables include λi;t;δ;q ,

The price is determined by the contracts between the ISCs and aggregators, which is more flexible than the price set in the wholesale electricity market. Therefore, targeted price-based DR programs can be designed to guide the load regulation and avoid negative effects brought about by IDCs, which can be divided into two types of scenario: 1) A set of geo-IDCs powered by the same power system.

, Ii;t;v ðmi;t Þ, Fi;t;v , and auxiliary energy systems. For example, the λdropped i;t;q

Wang et al. [1], proposed a pricing mechanism, which is formulated as a bi-level quadratic program and is solved using a heuristic algorithm. The aggregator chooses the proper pricing mechanisms to balance the electricity load at the upper level, and the ISC then minimizes its elec­ tricity fee at a lower level through a workload allocation and server control. The proposed pricing mechanism ensures the price charged to

variable is λi;t;δ;q in Ref. [6]; λi;t;δ;q and λdropped in Ref. [37]; λi;t;δ;q and mi;t in i;t;q

Refs. [11,38]; λi;t;δ;q and Ii;t;v in Ref. [36]; λi;t;δ;q and Fi;t;v in Ref. [39]; λi;t;δ;q , Ii;t;v , and Fi;t;v in Ref. [8]; λi;t;δ;q , sci;t , and sdi;t in Ref. [49]; and λi;t;δ;q ,

uci;t , and udi;t in Ref. [30], where sci;t and sdi;t denote the stored and released energy of the TSSs in IDC i in the tth time slot, respectively. Furthermore, 11

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

a few studies introduced the uncertainty of real-time electricity prices, electricity generated by the RGs, and the arriving workload into their models, which are generally formulated as stochastic program problems. For example, the authors of [72,75] formulated the model as a risk-constrained expected cost minimization problem, which considers the uncertainties in the price and workload. The proposed strategies achieved an optimal tradeoff between the operation risk and expected cost of the ISCs by introducing the conditional value at risk.

5. Research issues and challenges In this section, several issues around IDCs participating in DR are proposed, requiring more attention from the research community: (1) Establishment of suitable IDC load models for power system operations Considering that general IDC load models are not necessarily known to the SOs because they are represented based on the servers and workload, the authors of [38] converted the initial IDC load model into one with electrical variables and explicit load characteristics. However, with more load regulation operations expanded to the IDCs, which is described in detail below, the initial IDC load model becomes more complex and obscured to power systems. For example, it involves thermodynamic equations when considering the TSTI. The conversion of the initial IDC load model into a more suitable form for a power system operation, which should be simpler and clearer, will be an important issue in the future.

(3) Algorithms A few studies formulated their problems in forms that can be easily solved by existing business optimization tools directly, such as smallscale MILP [40,44,50,74], and convex optimization problems [58,59, 66,68]. They can also be further simplified and decomposed to improve the calculation efficiency. For example, the authors of [74] simplified the MILP into LP, whereas the authors of [58,68] decomposed the convex optimization problems into several sub-problems. However, with multiple adopted load regulation operations and ac­ curacy requirements, the controllable variables increase, and the models become more complex. This results in a non-convexity of the formulated problems, even if the integer constraints on the servers and workload are relaxed. It is generally difficult to obtain the optimal solutions of nonconvex problems using existing business optimization tools [76]. Therefore, sub-optimal solutions are generally acceptable, and can be obtained by designing heuristic algorithms [72,77–79], or using busi­ ness optimization tools directly [8,57]. In addition, they can also be solved by being approximately transferred into convex optimization problems [30,75,80]. In general, off-line strategies generally have high requirements regarding the problem formulation and algorithm design, focusing on the global optimum. However, with on-line strategies [37,41,81], simpler and more efficient algorithms are generally required to quickly obtain a sub-optimal solution, where distributed algorithms are gener­ ally adopted [37,41].

(2) Expansion of load regulation potential of IDCs An expanded architecture of the DR operation of the IDCs is pro­ posed, as shown in Fig. 2. Three main expansion directions are sug­ gested: 1) the cooperation of multiple load regulation operations, 2) the cooperation among multiple ISCs, and 3) the cooperation among data networks, power systems, and natural gas systems. First, considering that the TSTI can improve the temporal load regulation potential of each IDC significantly, the utilization of the TSTI in a single data center has been attempted in Ref. [26]. However, the cooperation of the TSTI and other load regulation operations in multiple IDCs has been rarely studied. Rather than a simple summation, further improvements will be achieved in which the thermal storage optimiza­ tion among multiple time slots can be further expanded to multiple IDC systems when the GLB is added to the TSTI. In addition, it was verified that the load regulation potential of the IDCs increases with an increase in the tolerant service delay, and that the sensitivity of the regulation potential to the tolerant service delay decreases with an increase in the tolerant service delay [38]. The adjustment of the tolerant service delay can be a new idea to expand load regulation potential of the IDCs. The cooperation of multiple load regulation operations and the corre­ sponding coupling models, which make full use of the DR potential and generate a significant improvement, will be an important issue in the future. Second, considering that a shared economy can increase the value of goods for businesses, individuals, the community, and society in general [82], the sharing of computing resources has achieved significant attention. However, regarding computing resources, this has a limited scope for the IDCs belonging to the same ISC. The sharing of computing resources among multiple ISCs, particularly in terms of techniques and profit allocations, will be an attractive issue, and will further upgrade the utilization of computing resources and the load regulation potential of IDCs. Third, considering that the operations of power and natural-gas systems have become increasingly interdependent [83], the adaptation of CCHP systems in IDCs brings an interdependency among data net­ works and power and natural gas systems. However, ways to operate data networks and power and natural gas systems when considering their coordinating optimization have rarely been studied and will become an attractive issue.

4.3. Incentive mechanisms between ISCs and their tenants The workload in colocation IDCs is generally managed by the tenants themselves, who are typically charged based on a usage-based or flatrate pricing. Therefore, tenants generally have no incentive to coop­ erate with the ISCs regarding a DR. To break such a “split incentive”, a few studies [51,52] have recently tried using market-based mechanisms inside the IDCs. Guo et al. [51], designed an economic incentive mechanism for an EDR, which motivates the tenants to reduce their power demand during emergency periods. The interaction among the ISC and tenants is modeled based on the Nash bargaining theory. Furthermore, based on two bargaining protocols, i.e., sequential bargaining and concurrent bargaining, the optimal solutions of the load reduction and reimburse­ ment for each tenant are also derived. The proposed bargaining approach is beneficial to both the ISC and tenants, which also reduces the carbon emissions into the environment through the participation in the EDR. By contrast with [51], Zhan et al. [52] proposed a novel incentive mechanism for price-based DR programs. The proposed pricing mech­ anism is not dynamic. That is, it keeps the pricing for computing re­ sources unchanged for a long period. The proposed mechanism consists of two parts: 1) charging tenants based on a usage-based pricing, and 2) rewarding tenants proportionally based on the length of time that the tenants set as deadlines for completing their workload. The proposed pricing mechanism effectively reduces the peak power consumption and electricity fee of the IDCs without decreasing the profit of the ISC.

(3) Design of targeted DR mechanisms for spatial-coupling loads With the high penetration of distributed generators, flexible loads, particularly loads such as IDCs, which have a potential of temporal and spatial load regulation, will take on more important roles in future 12

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

Fig. 2. Expanded architecture of the DR operation of IDCs.

energy systems. However, most existing DR programs do not consider the loads with a spatial load regulation potential. Application scenarios and targeted DR mechanisms for spatial-coupling loads regulating their power demand, as well as their expansion planning [84], need to be further researched to achieve the desired goals. In addition, the coop­ eration among multiple ISCs, the mechanisms among multiple ISCs, and the economic incentives to their tenants regarding the participation of a DR, such as privacy protection and revenue allocation, need to be further studied.

the startup cost of the servers, and the cost of the environmental impact. The regulation cost of the utilization of auxiliary energy systems generally includes the depreciation cost and generation cost. 3) The strategies for IDCs participating in DR programs can be grouped into three categories: power system perspectives, interactions between SOs and ISCs, and incentive mechanisms between ISCs and their tenants. In general, participating in DR programs can increase the profits of ISCs and improve the efficiency of power systems, where, appropriate DR programs between SOs and ISCs, and incentive mechanisms between ISCs and their tenants, are necessary. However, most existing DR mechanisms are designed for flexible loads without the potential of spatial load regulation, which means that the IDC bids may be without a spatial coupling. Furthermore, this study identified several research issues and chal­ lenges for the further DR operation of IDCs, and includes three di­ mensions: the establishment of suitable IDC load models for power system operation, the expansion of the IDC load regulation potential, and the design of targeted DR mechanisms for spatial-coupling loads. Accordingly, IDCs, which have a potential of temporal and spatial load regulation, can be extremely important DR resources. The DR of IDCs can bring about a significant effect on the cost savings of ISCs and an improved efficiency in energy systems. This is an interdisciplinary domain, where computer scientists and engineers from academia and industry still have abundant opportunities to contribute.

6. Conclusion The DR of IDCs has been an active area of research in recent years. A comprehensive survey covering the major parts of such DR was pre­ sented herein, in order of IDC load modeling, load regulation operations, economic considerations, and IDCs participating in DR programs. Spe­ cifically, 1) the load models of IDCs are generally divided into four parts: IT equipment, the cooling system, other equipment, and auxiliary en­ ergy systems. Compared with more elaborative models in the literature focusing on a single data center, simplifications have generally been adopted when focusing on multiple IDCs to improve the applicability, particularly in modeling the cooling system. In addition, IDC load models are generally represented by the workload and servers, which may puzzle SOs when apply them to a specific power system operation. 2) The ideas regulating the power demand of IDCs can be grouped into three categories: scheduling workload, TSTI, and the utilization of auxiliary energy systems. Compared with the first and third categories, the TSTI is rarely adopted in multiple IDCs. The regulation cost of scheduling workload generally includes the cost of migrating the workload, the cost of dropping the workload, the queueing delay cost,

Acknowledgements This work was supported by the National Natural Science Foundation of China (51207029), the State Grid Corporation of China project (YD71-18-002) and the Fundamental Research Funds for the Central 13

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466

Universities（2242019R20032）.

[29] Qureshi A, Weber R, Balakrishnan H, Maggs B, Guttag J. Cutting the electric bill for internet-scale systems. Comput Commun Rev 2009;39:123. https://doi.org/ 10.1145/1594977.1592584. [30] Ghamkhari M, Wierman A, Mohsenian-Rad H. Energy portfolio optimization of data centers. IEEE Trans Smart Grid 2017;8:1898–910. https://doi.org/10.1109/ TSG.2015.2510428. [31] Garimella SV, Persoons T, Weibel J, Yeh LT. Technological drivers in data centers and telecom systems: multiscale thermal, electrical, and energy management. Appl Energy 2013;107:66–80. https://doi.org/10.1016/j.apenergy.2013.02.047. [32] Or� o E, Depoorter V, Garcia A, Salom J. Energy efficiency and renewable energy integration in data centres. Strategies and modelling review. Renew Sustain Energy Rev 2015;42:429–45. https://doi.org/10.1016/j.rser.2014.10.035. [33] Rahman A, Liu X, Kong F. A survey on geographic load balancing based data center power management in the smart grid environment. IEEE Commun Surv Tutorials 2014;16:214–33. https://doi.org/10.1109/SURV.2013.070813.00183. [34] Vasques TL, Moura P, de Almeida A. A review on energy efficiency and demand response with focus on small and medium data centers. Energy Effic 2019;12: 1399–428. https://doi.org/10.1007/s12053-018-9753-2. [35] Wierman A, Liu Z, Andrew LLH, Lin M, Low S. Greening geographical load balancing. IEEE/ACM Trans Netw 2014;23:657–71. https://doi.org/10.1109/ tnet.2014.2308295. [36] Li J, Bao Z, Li Z. Modeling demand response capability by internet data centers processing batch computing jobs. IEEE Trans Smart Grid 2015;6:737–47. https:// doi.org/10.1109/TSG.2014.2363583. [37] Yu L, Jiang T, Zou Y. Distributed real-time energy management in data center microgrids. IEEE Trans Smart Grid 2018;9:3748–62. https://doi.org/10.1109/ TSG.2016.2640453. [38] Chen M, Gao C, Chen S, Li D, Liu Q. Bi-level economic dispatch modeling considering the load regulation potential of internet data centers. Proc CSEE 2019; 39:1301–13. https://doi.org/10.13334/j.0258-8013.pcsee.180456. [39] Li S, Brocanelli M, Zhang W, Wang X. Integrated power management of data centers and electric vehicles for energy and regulation market participation. IEEE Trans Smart Grid 2014;5:2283–94. https://doi.org/10.1109/TSG.2014.2321519. [40] Tripathi R, Vignesh S, Tamarapalli V. Optimizing green energy, cost, and availability in distributed data centers. IEEE Commun Lett 2017;21:500–3. https:// doi.org/10.1109/LCOMM.2016.2631466. [41] Yu L, Jiang T, Zou Y. Distributed online energy management for data centers and electric vehicles in smart grid. IEEE Internet Things J 2016;3:1373–84. https://doi. org/10.1109/IECON.2016.7793620. [42] Belady CL, Rawson A, Pfleuger AJ, Cader DT. The green grid data center power efficiency metrics. PUE and DCiE 2008. https://doi.org/10.1016/S0927-0507(07) 15006-4. [43] Garimella SV, Yeh L, Persoons T. Thermal management challenges in telecommunication systems and data centers. IEEE Trans Compon Packag Manuf Technol 2012;2:1307–16. https://doi.org/10.1109/TCPMT.2012.2185797. [44] Li J, Qi W. Toward optimal operation of internet data center microgrid. IEEE Trans Smart Grid 2018;9:971–9. https://doi.org/10.1109/TSG.2016.2572402. [45] Van Damme T, De Persis C, Tesi P. Optimized thermal-aware job scheduling and control of data centers. IEEE Trans Control Syst Technol 2019;27:760–71. https:// doi.org/10.1109/TCST.2017.2783366. [46] Fang Q, Wang J, Gong Q, Song M. Thermal-aware energy management of an HPC data center via two-time-scale control. IEEE Trans Ind Informatics 2017;13: 2260–9. https://doi.org/10.1109/TII.2017.2698603. [47] Ibrahim M, Bhopte S, Sammakia B, Murray B, Iyengar M, Schmidt R. Effect of transient boundary conditions and detailed thermal modeling of data center rooms. IEEE Trans Compon Packag Manuf Technol 2012;2:300–10. https://doi.org/ 10.1109/TCPMT.2011.2175926. [48] Vaghefi SA, Jafari MA, Zhu J, Brouwer J, Lu Y. A hybrid physics-based and data driven approach to optimal control of building cooling/heating systems. IEEE Trans Autom Sci Eng 2016;13:600–10. https://doi.org/10.1109/ TASE.2014.2356337. [49] Guo Y, Gong Y, Fang Y, Khargonekar PP, Geng X. Energy and network aware workload management for sustainable data centers with thermal storage. IEEE Trans Parallel Distrib Syst 2014;25:2030–42. https://doi.org/10.1109/ TPDS.2013.278. [50] Chen Z, Wu L, Li Z. Electric demand response management for distributed largescale internet data centers. IEEE Trans Smart Grid 2014;5:651–61. https://doi.org/ 10.1109/TSG.2013.2267397. [51] Guo Y, Li H, Pan M. Colocation data center demand response using nash bargaining theory. IEEE Trans Smart Grid 2018;9:4017–26. https://doi.org/10.1109/ TSG.2016.2647246. [52] Zhan Y, Ghamkhari M, Xu D, Ren S, Mohsenian-Rad H. Extending demand response to tenants in cloud data centers via non-intrusive workload flexibility pricing. IEEE Trans Smart Grid 2018;9:3235–46. https://doi.org/10.1109/TSG.2016.2628886. [53] Wang Y, Wang X, Zhang Y. Leveraging thermal storage to cut the electricity bill for datacenter cooling. Proc 4th Work. Power-Aware Comput. Syst. 2011:1–5. https:// doi.org/10.1145/2039252.2039260. [54] Lu N. An evaluation of the HVAC load potential for providing load balancing service. IEEE Trans Smart Grid 2012;3:1263–70. https://doi.org/10.1109/ TSG.2012.2183649. [55] Zheng W, Ma K, Wang X. TE-Shave: Reducing data center capital and operating expenses with thermal energy storage. IEEE Trans Comput 2015;64:3278–92. https://doi.org/10.1109/TC.2015.2394381. [56] Yao J, Liu X, Zhang C. Predictive electricity cost minimization through energy buffering in data centers. IEEE Trans Smart Grid 2014;5:230–8. https://doi.org/ 10.1109/TSG.2013.2274525.

References [1] Wang H, Huang J, Lin X, Mohsenian-Rad H. Proactive demand response for data centers: a win-win solution. IEEE Trans Smart Grid 2016;7:1584–96. https://doi. org/10.1109/TSG.2015.2501808. [2] Jonathan G. Koomey. Growing in data center electricity use 2005 to 2010. 2011. https://doi.org/10.1088/1748-9326/3/3/034008. [3] Ahmad F, Vijaykumar TN. Joint optimization of idle and cooling power in data centers while maintaining response time. ACM SIGPLAN Not 2010;45:243. https:// doi.org/10.1145/1735971.1736048. [4] Shuja J, Gani A, Shamshirband S, Ahmad RW, Bilal K. Sustainable cloud data centers: a survey of enabling techniques and technologies. Renew Sustain Energy Rev 2016;62:195–214. https://doi.org/10.1016/j.rser.2016.04.034. [5] Amazon elastic compute cloud n.d. https://en.wikipedia.org/wiki/Amazon_Elasti c_Compute_Cloud (accessed April 25, 2019).. [6] Lu X, Kong F, Liu X, Yin J, Xiang Q, Yu H. Bulk savings for bulk transfers: minimizing the energy-cost for geo-distributed data centers. IEEE Trans Cloud Comput 2017;7161:1–14. https://doi.org/10.1109/TCC.2017.2739160. [7] Grunwald D, Morrey CB, Levis P, Neufeld M, Farkas KI, III CM, et al. Policies for dynamic clock scheduling. In: Proc. 4th Conf. Symp. Oper. Syst. Des. Implement. 4. USENIX Association; 2000. [8] Li J, Li Z, Ren K, Liu X. Towards optimal electric demand management for internet data centers. IEEE Trans Smart Grid 2012;3:183–92. https://doi.org/10.1109/ TSG.2011.2165567. [9] Nathuji R, Schwan K. VirtualPower: coordinated power management in virtualized enterprise systems. ACM SIGOPS - Oper Syst Rev 2007;41:265. https://doi.org/ 10.1145/1323293.1294287. [10] Al-Fares M, Loukissas A, Vahdat A. A scalable, commodity data center network architecture. Comput Commun Rev 2008;38:63. https://doi.org/10.1145/ 1402946.1402967. [11] Zhang Y, Wang Y, Wang X. Electricity bill capping for cloud-scale data centers that impact the power markets. Proc Int Conf Parallel Process 2012:440–9. https://doi. org/10.1109/ICPP.2012.23. [12] Kliazovich D, Bouvry P, Khan SU. GreenCloud: a packet-level simulator of energyaware cloud computing data centers. J Supercomput 2012;62:1263–83. https:// doi.org/10.1007/s11227-010-0504-1. [13] Zhang W, Wen Y, Wong YW, Toh KC, Chen CH. Towards joint optimization over ICT and cooling systems in data centre: a survey. IEEE Commun Surv Tutorials 2016;18:1596–616. https://doi.org/10.1109/COMST.2016.2545109. [14] Lee EK, Viswanathan H, Pompili D. Proactive thermal-aware resource management in virtualized HPC cloud datacenters. IEEE Trans Cloud Comput 2017;5:234–48. https://doi.org/10.1109/TCC.2015.2474368. [15] Joshi Y, Kumar P. Energy efficient thermal management of data centers. 2012. https://doi.org/10.1007/978-1-4419-7124-1. [16] Zhao X, Peng T, Qin X, Member S, Hu Q, Ding L, et al. Feedback control scheduling in energy-efficient and thermal-aware data centers. IEEE Trans Syst 2015;46:1–13. https://doi.org/10.1109/TSMC.2015.2434797. [17] Knorn F, Shorten R, O’Mahony D, Doyle J. Distributed thermal aware load balancing for cooling of modular data centres. IET Control Theory & Appl 2013;7: 612–22. https://doi.org/10.1049/iet-cta.2011.0733. [18] Basmadjian R, Botero JF, Giuliani G, Hesselbach X, Klingert S, De Meer H. Making data centers fit for demand response: introducing GreenSDA and GreenSLA contracts. IEEE Trans Smart Grid 2018;9:3453–64. https://doi.org/10.1109/ TSG.2016.2632526. [19] Demand response n.d. https://en.wikipedia.org/wiki/Demand_response (accessed April 25, 2019).. [20] Song M, Gao C, Yan H, Yang J. Thermal battery modeling of inverter air conditioning for demand response. IEEE Trans Smart Grid 2018;9:5522–34. https://doi.org/10.1109/TSG.2017.2689820. [21] Romero M, Hasselqvist H, Svensson G. Supercomputers keeping people warm in the winter. In: 2nd int. Conf. ICT sustain.; 2014. p. 324–32. https://doi.org/ 10.2991/ict4s-14.2014.40. [22] EC3 Wu J, Jin Y, Yao J. Cutting cooling energy consumption through weatheraware geo-scheduling across multiple data centers. IEEE Access 2017;6:2028–38. https://doi.org/10.1109/ACCESS.2017.2781309. [23] Yang L, Deng Y, Yang LT, Lin R. Reducing the cooling power of data centers by intelligently assigning tasks. IEEE Internet Things J 2018;5:1667–78. https://doi. org/10.1109/JIOT.2017.2783329. [24] Zheng K, Zheng W, Li L, Wang X. PowerNetS: coordinating data center network with servers and cooling for power optimization. IEEE Trans Netw Serv Manag 2017;14:661–75. https://doi.org/10.1109/TNSM.2017.2711567. [25] Wan J, Gui X, Zhang R, Fu L. Joint cooling and server control in data centers: a cross-layer framework for holistic energy minimization. IEEE Syst J 2018;12: 2461–72. https://doi.org/10.1109/JSYST.2017.2700863. [26] Al-Qawasmeh AM, Pasricha S, Maciejewski AA, Siegel HJ. Power and thermalaware workload allocation in heterogeneous data centers. IEEE Trans Comput 2015;64:477–91. https://doi.org/10.1109/TC.2013.116. [27] Bilal K, Khan SU, Liu H, Lin M, Liu B, Yang LT, et al. Thermal-aware and DVFSenabled big data task scheduling for data centers. IEEE Trans Big Data 2017;4: 177–90. https://doi.org/10.1109/tbdata.2017.2763612. [28] Shah AJ, Krishnan N. Optimization of global data center thermal management workload for minimal environmental and economic burden. IEEE Trans Compon Packag Technol 2008;31:39–45. https://doi.org/10.1109/TCAPT.2007.906721.

14

M. Chen et al.

Renewable and Sustainable Energy Reviews 117 (2020) 109466 [70] Gao C, Wu G, Chen S. A model aimed at reducing power net loss considering frequency scaling of servers in geo-distributed data centers. Proc CSEE 2019;39: 1673–1681þ1863. https://doi.org/10.13334/j.0258-8013.pcsee.180243. [71] Cao X, Gao C, Li D, Yang J. Mixed operation model of data network and power network and its participation in the economic operation of power system. Proc CSEE 2018;38:1448–56. https://doi.org/10.13334/j.0258-8013.pcsee.170611. [72] Rao L, Liu X, Xie L, Pang Z. Hedging against uncertainty: a tale of Internet data center operations under smart grid environment. IEEE Trans Smart Grid 2011;2: 555–63. https://doi.org/10.1109/TSG.2011.2159523. [73] Yu L, Jiang T, Zou Y. Price-sensitivity aware load balancing for geographically distributed internet data centers in smart grid environment. IEEE Trans Cloud Comput 2018;6:1125–35. https://doi.org/10.1109/TCC.2016.2564406. [74] Rao L, Liu X, Xie L, Liu W. Coordinated energy cost management of distributed internet data centers in smart grid. IEEE Trans Smart Grid 2012;3:50–8. https:// doi.org/10.1109/TSG.2011.2170100. [75] Yu L, Jiang T, Cao Y, Zhang Q. Risk-constrained operation for internet data centers in deregulated electricity markets. IEEE Trans Parallel Distrib Syst 2014;25: 1306–16. https://doi.org/10.1109/TPDS.2013.2297095. [76] Boyd S, Vandenberghe L. Convex optimization 2004. https://doi.org/10.1109/ TAC.2006.884922. [77] Wendell P, Jiang JW, Freedman MJ, Rexford J. DONAR: decentralized server selection for cloud services. Comput Commun Rev 2012;40:231. https://doi.org/ 10.1145/1851275.1851211. [78] Wang P, Rao L, Liu X, Qi Y. D-Pro: Dynamic data center operations with demandresponsive electricity prices in smart grid. IEEE Trans Smart Grid 2012;3:1743–54. https://doi.org/10.1109/TSG.2012.2211386. [79] Wang Y, Lin X, Pedram M. A stackelberg game-based optimization framework of the smart grid with distributed PV power generations and data centers. IEEE Trans Energy Convers 2014;29:978–87. https://doi.org/10.1109/TEC.2014.2363048. [80] Shao H, Rao L, Wang Z, Liu X, Wang Z, Ren K. Optimal load balancing and energy cost management for internet data centers in deregulated electricity markets. IEEE Trans Parallel Distrib Syst 2014;25:2659–69. [81] Chen T, Wang X, Giannakis GB. Cooling-aware energy and workload management in data centers via stochastic optimization. IEEE J Sel Top Signal Process 2016;10: 402–15. https://doi.org/10.1109/JSTSP.2015.2500189. [82] Sharing economy n.d. https://en.wikipedia.org/wiki/Sharing_economy (accessed April 25, 2019).. [83] Chen S, Wei Z, Sun G, Cheung KW, Wang D. Identifying optimal energy flow solvability in electricity-gas integrated energy systems. IEEE Trans Sustain Energy 2017;8:846–54. https://doi.org/10.1109/TSTE.2016.2623631. [84] Ahmad I, Khodayar ME, Vafamehr A, Lin J, Manshadi SD. A framework for expansion planning of data centers in electricity and data networks under uncertainty. IEEE Trans Smart Grid 2017;10:305–16. https://doi.org/10.1109/ tsg.2017.2738444.

[57] Bajracharyay L, Awasthi S, Chalise S, Hansen TM, Tonkoski R. Economic analysis of a data center virtual power plant participating in demand response. IEEE Power Energy Soc. Gen. Meet. 2016:1–5. https://doi.org/10.1109/ PESGM.2016.7741726. [58] Wang H, Ye Z. Renewable energy-aware demand response for distributed data centers in smart grid. In: 2016 IEEE green energy syst. Conf.; 2016. https://doi. org/10.1109/IGESC.2016.7790076. [59] Ghamkhari M, Mohsenian-Rad H. Energy and performance management of green data centers: a profit maximization approach. IEEE Trans Smart Grid 2013;4: 1017–25. https://doi.org/10.1109/TSG.2013.2237929. [60] Nadjaran Toosi A, Qu C, de Assunç~ ao MD, Buyya R. Renewable-aware geographical load balancing of web applications for sustainable data centers. J Netw Comput Appl 2017;83:155–68. https://doi.org/10.1016/j.jnca.2017.01.036. [61] Arnone D, Barberi A, La Cascia D, Sanseverino ER, Zizzo G. Smart grid integrated green data centres as ancillary service providers. In: 5th int. Conf. Clean electr. Power renew. Energy resour. Impact. IEEE; 2015. p. 170–7. https://doi.org/ 10.1109/ICCEP.2015.7177619. [62] Kiani A, Ansari N. A fundamental tradeoff between total and brown power consumption in geographically dispersed data centers. IEEE Commun Lett 2016;20: 1955–8. https://doi.org/10.1109/LCOMM.2016.2598535. [63] M€ asker M, Nagel L, Brinkmann A, Lotfifar F, Johnson M. Smart Grid-aware scheduling in data centres. Comput Commun 2016;96:73–85. https://doi.org/ 10.1016/j.comcom.2016.04.021. [64] Liu Q, Chen S, Chen M, Gao C. Energy management for internet data centers considering the coordinating optimization of workload and CCHP system. In: 2018 2nd IEEE conf. Energy Internet energy syst. Integr., IEEE; 2018. p. 1–5. https://doi. org/10.1109/EI2.2018.8582512. [65] Zhang S, Sun X, Liu H. Combining data centers with electric vehicle battery swapping stations for grid regulation. In: 2nd int Conf Intell Green build Smart grid, IGBSG 2016; 2016. p. 0–5. https://doi.org/10.1109/IGBSG.2016.7539453. [66] Kiani A, Ansari N. Profit maximization for geographically dispersed green data centers. IEEE Trans Smart Grid 2018;9:703–11. https://doi.org/10.1109/ TSG.2016.2562565. [67] Tran NH, Tran DH, Ren S, Han Z, Huh EN, Hong CS. How geo-distributed data centers do demand response: a game-theoretic approach. IEEE Trans Smart Grid 2016;7:937–47. https://doi.org/10.1109/TSG.2015.2421286. [68] Yu L, Jiang T, Zou Y, Sun Z. Joint energy management strategy for geo-distributed data centers and electric vehicles in smart grid environment. IEEE Trans Smart Grid 2016;7:2378–92. https://doi.org/10.1109/TSG.2016.2542261. [69] Mohsenian-Rad A-H, Leon-Garcia A. Coordination of cloud computing and smart power grids. In: First IEEE int Conf Smart grid commun., IEEE; 2010. p. 368–72. https://doi.org/10.1109/smartgrid.2010.5622069.

15