A System of Systems approach for data centers optimization and integration into smart energy grids

Accepted Manuscript A system of systems approach for data centers optimization and integration into smart energy grids Marcel Antal, Claudia Pop, Tudor Cioara, Ionut Anghel, Ioan Salomie, Florin Pop

PII: DOI: Reference:

S0167-739X(17)31012-9 http://dx.doi.org/10.1016/j.future.2017.05.021 FUTURE 3469

To appear in:

Future Generation Computer Systems

Received date : 22 November 2016 Revised date : 8 May 2017 Accepted date : 14 May 2017 Please cite this article as: M. Antal, et al., A system of systems approach for data centers optimization and integration into smart energy grids, Future Generation Computer Systems (2017), http://dx.doi.org/10.1016/j.future.2017.05.021 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

A System of Systems Approach for Data Centers Optimization and Integration into Smart Energy Grids Marcel Antal1, Claudia Pop1, Tudor Cioara1, Ionut Anghel1, Ioan Salomie1, Florin Pop2,3 1

Technical University of Cluj-Napoca, Baritiu 26, Cluj-Napoca, Romania {marcel.antal, claudia.pop, tudor.cioara, ionut.anghel, ioan.salomie}@cs.utcluj.ro 2

University Politehnica of Bucharest, Splaiul Independen ei 313, Bucharest, Romania [email protected] 3 National Institute for Research and Development in Informatics (ICI), Mareşal Averescu, 8-10, Bucharest, Romania [email protected] Abstract. This paper addresses the problem of proactive planning and optimizing the operation of a Systems of Systems (SoS) over a time horizon while considering the characteristics of each constituent system and complex interactions among them. We define a mathematical formalism for modelling complex systems composed of a mesh of sub-systems with linear and non-linear behaviours and abstractions like discrete time, atomic systems and interconnection of atomic system. The proposed modeling approach is simple enough to allow fast computations and simulations, and at the same time complex enough to capture the essential features of the real system thus allowing the mapping of proactive optimization problems to Mixed-Integer Optimal Control Problems. The proactive planning uses hierarchical optimization processes that compute predictions and optimization plans at various time granularities, each finer layer plan adjusting and refining the ones with higher granularity. To show case our approach we model a Data Centre which is a well-known case of a large scale complex system aiming to plan and optimize its operation to use as much as possible the locally produced renewable energy and optimize its integration in smart grid advanced context. Simulation based results show a reduction of 5% of the carbon footprint and at the same time an increase in profit of more than 14% due to flexible energy shifting. Keywords: Systems of Systems, Modelling, Multi-layer Optimization, Data Centre, Energy Flexibility

1. Introduction While the world becomes increasingly interconnected, the IT systems and processes are becoming ever more complex being exposed to the rapid changes inside the organizations as well as in market conditions and customer requirements, demanding on the fly coordination, adaptation and optimization. The Systems of System (SoS) are large scale entities that monitor and control organizational and cross organizational processes in real-time, requiring a high degree of autonomy and dependability, and aggregating different autonomous systems in various application domains. These constituent systems possess operational and managerial

independence, yet interact on various levels to give rise to an overarching SoS level capability [1]. To cope with these constraints, the organizations are facing an unprecedented process of automation and re-configuration of their core business activities making them more agile and flexible in responding to external stimuli from their business environment. Nowadays there are lots of systems in various sectors that are upgraded from industrial to the digital information age. The upgrade is driven by recent technological advancements in the areas of wireless sensors networks, IoT and mobile cloud computing which led to the development of complex IT systems for administrating heterogeneous business-related tasks and processes featuring sensors, actuators, smart devices and agents (humans or software) which interact and collaborate. A representative example in this sense is the energy management sector in which the Smart Grid is usually seen as a complex system composed of heterogeneous and independent sub-systems (i.e. consumer, producers, prosumers, storage, etc.) that interact to compete or cooperate. The increasing need for integrating larger shares of intermittent renewable energy systems pose severe problems because of their intermittent energy production nature with combined with their regionally distributed disposition require new system integration abstraction and advanced distributed methods for optimal and cost-effective management. In this rather dynamic industrial and economic context the usage of conventional integration techniques for the development of complex SoS is not possible due to their heterogeneous and large scale distributed nature making their engineering, management and horizontal integration extremely expensive. In the Smart Grid dynamic energy context, the Data Centers (DCs) are large scale prosumers which might be viewed a SoS, due to the complex mesh of interconnected components that have to operate seamlessly: the IT resources composed of servers and network elements, cooling systems and power supply and storage systems. The DCs operation is affected by both internal factors (i.e. computational resources availability) but also external factors such as the clients QoS requirements, energy availability and costs, etc. Because of the large energy footprint, energy storage and scheduling capabilities, modern DCs can be viewed as intelligent energy players deployed in Smart Grids, being able to influence the local energetic ecosystems by trading energy [45], [46], [47]. To achieve this, the DC Infrastructure Management solutions need to continuously monitor the IT hardware component while analyzing and planning control sequences for assuring the DC proper operation. Nowadays DC management software is mostly reactive and take operation planning and optimization actions only after some monitored events have been registered. This may lead to inefficient and suboptimal situations. For instance, in the case of minimizing energy consumption the reorganization of the DC’s resource is decided only after some policies have been violated, thus only after some energy was consumed in excess. Also, the DC operation re-planning can further affect the execution of the running application, especially if the optimization time is long, thus leading to several more policies violation. This paper original contribution is the definition of SoS mathematical models and an associated multi-layer proactive planning method for optimizing the operation and management of large scale distributed systems. Our modeling approach builds upon

existing state of the art by defining a diagrammatic block-based mathematical formalism for representing SoS as models of the physical world. The resulting representation features abstractions like discrete time, atomic systems and the interconnection of atomic system that leads to the SoS, being simple enough to allow fast computations and simulations, and at the same time complex enough to capture the essential features of the real system. The multi-layer proactive planning is based on layered optimization processes that compute predictions and optimization plans at various time granularities, each finer layer plan adjusting and refining the ones with higher granularity. Leveraging on the defined models, the SoS proactive optimization problem is reduced to a Mixed-Integer Optimal Control Problem, whose solution can be approximated using modern mathematical solvers. The advantage of our approach lies on forecasting problematic future states and trying to address and avoid them by triggering the optimization planning in advance. The proposed optimization methodology is validated by modeling a DC which is a large scale complex system as a SoS aiming to plan and optimize its operation to use as much as possible the locally produced renewable energy and optimize its integration with the Smart Grid while helping in achieving local sustainability goals. The rest of the paper is structured as follows: Section 2 presents the state of the art solutions for modeling and managing the operation of SoS as well existing DC energy consumption models, Section 3 defines our formalism for modeling complex distributed systems and the multi-layer proactive planning method for their operation optimization, Section 4 shows how the proposed formalism and method can be used for optimizing the operation and integration of a DC into the smart energy grid with the objectives of increasing the share of renewable energy used and decreasing the overall cost with energy, while Section 5 concludes the paper and the presents future work.

2. Related Work Even though the concept of ‘System of Systems’ has been used since 1950 for defining complex systems that are composed of other independent systems that interact and cooperate to deliver a capability that cannot be achieved individually, their systematic management and optimization is still an open research direction today. Important challenges of independence, distribution, heterogeneity, evolution and emergent behavior need to be addressed when modelling such systems [3]. There are several states of the art attempts to model complex SoS [3], [4] such as: network based techniques, general systems theory, agent based modelling, probabilistic models, object oriented (taxonomy based) modelling, state transition modelling etc. They are aiming to define a programmatic model to represent sub-systems linear and non-linear behavior, SoS interactions and to analyze by simulation the achieved accuracy or suitability for fitting the SoS challenges. Papers [5] and [6] leverage on modern network technologies abstraction for designing and modelling the complex interactions emerging in SoS and provide a formal semantic foundation to support analysis of global SoS properties. The SoS is designed using a hierarchical network

topology while mean-field theory is used to analyze and asses the network model. The ISO Network Management Model is used [7] to map the five conceptual areas for managing networks (i.e. performance, configuration, accounting, fault, and security) to the Boardman-Sauser characteristics of SoS: autonomy, belonging, connectivity, diversity, and emergence. Even though these kinds of models accurately represent the local systems integration into the global more complex one, they fail to provide decision making features targeting the SoS decentralized operation coordination. State and object-oriented modelling are also quite common approaches for engineering SoS. In [8] state modelling is used to analyze the average performance of systems over defined use conditions. The state modelling capability supports analysis and optimization of multiple systems and multiple performance metrics and is used as input for a time simulation approach. In [9] hierarchical model and decision-making process is used for enabling decision makers to rapidly and simultaneously manipulate the design space of SoS, while [10], [11] defines SoS engineering process based on UML and Systems Modelling Language consisting of domain modelling, use case development, and behavioral and structural modelling, architecting, analysis, modelling and simulation, test and evaluation. Even though these types of models manage to accurately capture the real-world characteristics and behavior they are not suited for capturing the complex interaction emerging in large scale systems and their modularity and cross-cutting concerns. In [12] SoS models for the risk analysis of industrial installations and critical infrastructures are proposed. Fault trees and hierarchical graph models are described as potential solutions to capture SoS interactions. Monte Carlo simulation and interval analysis are combined for the quantitative evaluation of the SoS models in presence of uncertainty. [13] abstracts SoS components by using Event tree methods and Bayesian Networks (BNs) are used to quantify and define interdependencies between systems. Decision making algorithms for analysis are used to provide support for decision makers to manage the development of a SoS with complex interdependencies. The solution presented in [14] offers a broad analysis of possibilities to model SoS concluding that deterministic models such as differential equations, synchronous digital logic, single-threaded imperative programs and instruction set architectures can be considered viable for such ambitious task. Few limited approaches are suggesting the use of agent based modelling for defining the SoS characteristics through formal representations and instantiate the model to form an agent-based modelling perspective. In [15] the authors leverage on set theory and first order predicate logic to give a formal validation of the defined model and using simulations they conclude that an agent based approach will suit the modelling of most SoS characteristics. [16] analyses the current status of cooperative system and argue that SOA architecture, agent oriented models and cloud models can be used to apply SoS features for large scale complex IT systems. The authors of [17] define a simplified representation of a complex system that is based on conceptual, verbal, diagrammatic, physical, or formal (mathematical) models. To construct suitable models of the physical world, observation of the real-world system is used to determine possible rules for explaining and making predictions on the systems behavior. In [18] a unified formalism to describe systems classified as complex is

proposed. The authors highlight two aspects of the systems: heterogeneity (modeled using continuous or discrete time) and integration (modeled by recursively dividing the systems and constructing them from sub-systems). From the mathematical point of view, complex systems are approached in two ways: from simple to complex and from complex to simple. The first approach understands how simple but many basic elements lead to complex overall behaviors (e.g. cellular automata) while the second approach defines precise semantics to the system and the integration of its subcomponents. Furthermore, a novel semantic formalism to define complex systems as an interaction of subsystems with respect to a discrete time is proposed, resulting a system definition given in similar to a Mealy Machine enhanced with the concept of time, and a finite set of states. The new model of SoS lays the foundations towards a new vision for radically new future technology: computational collective systems which grow themselves automatically in response to changing, unforeseeable needs, over decades and centuries to come. Such foundational research relies on concrete and interdependent breakthroughs in several relevant problems within computer science and biology. This new view that due to their interdependent nature, these breakthroughs can - and indeed can only - be solved by working in an interdisciplinary joint venture, driven by a holistic model of a long-term engineering vision. According with [56] the case studies that imply SoS focus on addressing real human needs that are approached as complex applications like smart mobility, smart energy and smart society application areas related business cases dealing with road traffic, autonomic vehicles, wireless and mobile low-power embedded ecosystems, networked appliances in buildings and immediate environment, cyber-physical systems in the university campus. As a real use-case, it is not feasible to assume that entire infrastructures that support smart cities, remote sensor operations and vast mobile networks, can be entirely redesigned and redeployed every few years, according to the latest designs and requirements. For example, the infrastructure of yesterday, such as London's tube network and Paris's sewer system were both designed and built over 100 years ago. Nevertheless, they have been adapted over time, and are still relied upon by millions of people for 21st century needs. Future infrastructure will not be of the form of pipes and tunnels, but service-based information-driven technology, embodying populations of autonomous agents, interacting as part of larger socio-technical systems. Therefore, the technology being deployed over the next 50 years will inevitably be used in many different ways and for different reasons, over the following 150 years, then can be predicted today. One naive approach to this is to design systems based on today's needs, and hope that such future repurposing can be achieved. In contrast to this passive view, we take the alternative active view that the ability of such systems to grow for themselves, can be designed in. Our vision is therefore one of systems which grow themselves automatically over the long term in an effective way, from their initially conceived, limited functionality, through foreseeable functionality, through to as yet inconceivable functionality, in a complex changing world. Lately the high energy consumption of DCs have become of great concern for their owners and operators, which are continuously seeking for energy efficient and cost-effective solutions. The DCs are large scale distributed systems characterized by

large and yet flexible electrical energy load profiles which enacts as potential solution for energy and cost reduction their voluntary participation in Demand Response (DR) programs [36]. Most state of the art energy consumption models address DC components in isolation failing to consider the complex interactions among each other and with the external smart energy systems. The energy consumption of IT servers is usually approached using queue theory or linear power models. The linear power models approximate the server power consumption as a function of the CPU utilization [35], [37], [38], [39]. The queue based models classify the workload executed by servers in real-time and delay-tolerant [32], [33]. The real-time workload consists of tasks that have strong SLA constraints and must be executed as they arrive, while delay-tolerant tasks can be scheduled for execution any time after their arrival and before a given deadline, allowing a degree of flexibility in optimizing the DC energy consumption. The power consumed by the servers is converted into heat that must be dissipated by the cooling system [34]. The cooling system energy consumption models are complex, mixing thermodynamics, fluid dynamics and mechanics being sometimes very difficult to process and compute. In [40] the energy consumption of the cooling system is approximated using a function of the energy consumed by servers and a coefficient of performance of the cooling system, denoted COP that is a quadratic function depending on the server room set-point temperature. This approximation relies on the observation that, according to the law of energy conservation, all the electrical energy consumed by the servers is transformed into heat. Another more complex approaches use heat flow models that correlate the DCs cooling system with the workload placement [34], [52], [53], [54]. The DC is viewed as a bi-dimensional matrix of servers while their power consumption model is defined as an affine equation depending on the servers’ load. An influence matrix is defined to show the thermal influence between servers under the assumption that the power consumed by the servers is transformed in heat that must be dissipated by the cooling system. Regarding the energy storage devices, batteries and thermal storage tanks are usually considered in consumption models [41], [42], [48], [49], [50], [51], [55]. Computational Fluid Dynamics is used to model the DC thermal model which integrate thermal storage elements, as well as to derive the energy losses is such a system. Also, mathematical models are usually used to represent hybrid cooling system composed of traditional CRAC cooling, free air cooling and liquid cooling.

3. SoS’s Model and Proactive Optimization We have defined and formalized a SoS model featuring the concepts of atomic system and their logical organization in SoS. The modeling approach considers a discrete time model |

,∀

0

(1)

that involves having samples of the system and its components at equidistant timestamps 0, ∈ .

An atomic system is defined by a set of inputs ∈ ,of size , a set of outputs ∈ , of size , an internal state ∈ , of size , a set of independent control inputs ∈ , of size , that can be either real numbers or ∪ , two transfer functions and and a discrete time integers ( . The function computes at each timestamp the output of the system model considering the current input set , the control variable set and its internal state at the timestamp t: :

→ ,

,

,

∈

, ∈

At each discrete timestamp the function system, preparing it for the next timestamp 1: :

→ 1

,

∈

, ∈

(2)

updates the internal state of the

,

,

,

,

∈

, ∈

,

∈

, ∈

(3)

The atomic system can be represented as a black box, (as shown in Figure 1) and moreover as a Mealy machine with infinite states, if functions and allow it.

Figure 1- Atomic System Representations.

To ease the representation and logical integration at macro SoS level, each control variable can be semantically associated with a control action defined in a human readable taxonomy as follows: ∈



∃

, ,





,

(4)

An atomic system can be either in a consistent or in an inconsistent state. The atomic system is in a consistent state at timestamp if the system defined constraints (i.e. the function) defined as: ∶

→

(5)

are met when applied on the inputs, control, outputs and state variables: ,

,

,

0

(6)

The goal of the control variables is to avoid allowing the system to reach in an inconsistent state. We define a system of systems as a direct graph of N atomic systems in a discrete time model , where the nodes represent the atomic systems and the edges the communication flow between them.

, , ,

,



. .

∈ 1. .

∈

,

∈ ∈

,



∈

,

∈



(7)

, links the output of the atomic system with the An edge input of the atomic system within the SoS. The state of the at a timestamp is defined as a tuple containing the state of all its sub-components (i.e. atomic systems) and the actions that are scheduled for execution that affect each atomic system:

∈



∈ 1. .

,

|

,

(8)

To determine the next state of the SoS corresponding to timestamp 1, all the actions scheduled for timestamp must be executed for every atomic system by applying the corresponding actions for control variables. The queue of actions scheduled for execution after the timestamp over the time window will be denoted as an action plan: ,

∈

..

, ∈ 1. .

(9)

As result by applying the action plan on the SoS at timestamp , each atomic system will reach a new state that will be aggregated to define the new state of the SoS at the timestamp 1: 1

1 ,

,

,

∈ ,

∈



∈ 1. .

∈ 1. . ,

,

,

(10)

On the above model, we define the proactive optimization of an SoS operation as the adjustment of the set of control variables of each atomic system component with the goal of fitting a target objective function defined over the SoS’s states for a time window ( ). The proactive optimization of a SoS on our defined model is reduced to a pipeline of two processes: ,

, , ,

,

, ,

(11)

The inputs of the SoS over the optimization time window needs to be predicted (see relations 2, 3 and 8) using forecasting techniques applied on the historical set of input data. Each prediction function works with the discrete time model of the

system for which predictions are made, thus computing a set of predicted inputs spanning over the equidistant timeslots 1. . of the prediction window: | ∈ 1. .

, ,



∈ 1. .

(12)

The objective function is defined on the set of control variables of the atomic systems with compose the SoS, which define the states of the SoS spanning over the optimization time window : :⋃

1. .

→

(13)

The optimization process aims at determining the control variables corresponding to the states which the SoS must reach at every timestamp ∈ 1. . on an optimization window such that the objective function is minimized and the system does not reach forbidden or inconsistent states: , ∈ 1. .



, ,



, ∈ 1. .



,

⋃ 0, ∈ 1. .

with

(14)

, ∈ 1. .

In relation (14) it can be seen that on our defined model the proactive optimization can be reduced to a classical Mixed-Integer Optimal Control Problem (MIOCP) that is a subset of the mixed-integer nonlinear optimization problems (MINLP), as shown in [26]. It must be noticed that the problem class is MINLP because the control variables can be either real values or integer values. According to [26], MIOCP is defined as:

, 0

,

,

, ∀ ∈ 0, ∈ , ∀ ∈ 0,

0

, ∀ ∈ 0,

,

(15)

, ∀ ∈ 0,

∈ , ∀ ∈ 0, where the aim is to determine a set of states : 0, → , a set of continuous controls and a set of integer controls that minimizes the function over the time window , considering that is defined by a set of ordinary differential equations. This definition can be mapped to proactive optimization of SoS if the transition functions and defined in relations (2) and (3) are twice differentiable. The quality of such a plan for proactive optimization of SoS operation is strictly correlated to the degree of accuracy of the prediction function (12) which is influenced by the length of the time window . Thus, a prediction closer to the reference time 0 will have a greater accuracy than the one further away in time: ,

1

2

|



|

(16)

To cope with this depth of time problem, we propose a multi-layer proactive optimization based on layered optimization processes that compute predictions and optimization plans at various time granularities, each finer layer plan adjusting and refining the ones with higher granularity. For the optimization process at each layer a discrete time model is defined as: |

,∀

0 , ∈ 1. .

(17)

with the property that: ∀

0 ∃ ∈





∗

(18)

meaning that the finer time granularities in lower processes layers will have the sampling frequency a division of the higher layers’ processes time granularities. Furthermore, each finer layer of the optimization will have a smaller time window than the layer above, considering the same reference frame: ∗

∗

(19)

The multi-layer proactive optimization is defined as: ,

,

∈ 1. .

(20)

where defines the layers proactive optimization process on a , each layer having its own optimization process with a finer granularity given by the discrete time model , and a shorter time prediction time window . Thus, a higher lever optimization process will have a better degree of precision because if looks closer in the future. Each level optimization process computes an action plan with the constraint that it does not violate the constraints of the plans it refines. Furthermore, if a layer 1 optimization process computes a plan over the time windows , which refines the one of the higher layer optimization process , it must meet the constraint that the state of the system at the end of the newly added optimization plan remains the same as before adding this new plan, thus allowing undisturbed execution for further moments in time:

(

∈



∈ 1. .

,

∪

(21)

4. Use Case and Results To show the effectiveness of our solution we have use it to model and optimize the operation of a DC, which is a large scale distributed system featuring both energy production and energy consumption atomic systems. Our optimization goals were to proactively plan the DC operation as a system to be able to exploit its latent flexible energy to achieve a deeper and more efficient integration with the local Smart Grid.

This worrk has been co onducted in tthe context off the EU FP7 GEYSER prroject [18] which aim ms at providin ng models forr exploiting th he DCs as tecchnological hhubs at the cross roadd of utility (ellectricity or thhermal) and daata networks (i.e. ( federationn of DCs). For moree details, pleasse refer to [19,, 20, 21]. 4.1. DC D Represen ntation as SoS S Figuree 2 shows th he DC modelled as SoS feeaturing a graaph of atomic ic systems connectedd by energy links. The attomic system ms are either energy generration and distributiion componen nts (i.e. Diessel Generatorr, Transfer Switch, S Unintterruptible Power Supply S (UPS)), Renewablee Energy Geeneration), orr energy connsumption componennts (i.e. IT Servers S and C Cooling Systtem). They are a connectedd by links showing the energy trransfer amongg them: the en nergy distribu ution links (i. e. , , , ) and th he energy conssumption link ks (i.e. , ) balanced by tthe Power Distributiion Unit (PDU U).

Figure 2 – D DC representattion as a SoS.

The IT Servers and d the Coolingg System are the t largest energy consumeer systems of the DC, D and they are poweredd through thee PDU from the UPS. Thhe UPS is equippedd with a set of o Electrical Storage Deviices (ESDs) or batteries w which can supply poower to the esssential DC syystems in casee of a power outage. o The seerver room is chilledd by the Coolin ng System thaat removes the heat produced by the IT SServers as result of executing workload. w Mosst of the elecctrical energy y consumed bby the IT Servers iss transformed d into heat, acccording to the law of energy conservatiion, so the Cooling System S should d dissipate an amount of th hermal energy approximatelly equal to the serverr electrical en nergy consumpption. Modern n Cooling Systems are equiipped with Thermal Storage Devicces (TES) [222] that store co old coolant useed to chill thee air blown by the airr conditioning g units. By taaking coolant from the TES S, the electriccal cooling system coompressor can n be turned offf thus saving energy. Howeever, when chharging the TES, the cooling system m must operaate at a higher capacity to geenerate extra ccoolant.

The DC is powered through a main transfer switch that has energy inputs from the power grid and local sources such as the diesel generator and renewable energy generation systems. Beside consuming energy from the grid, the DC can feed back and sell excess energy generated by local sources. The inputs of the DC SoS are on the energy consumption side the workload that need to be executed split on the real-time and delay-tolerant and on the energy generation side the wind speed which determines the amount of renewable energy produced. The independent control variables are mapped to the control of each atomic system operation having associated a predefined optimization actions. Table 1 summarizes the mapping between our SoS model and a DC system. Table 1 – Mapping the SoS formalism to the DC. SoS Model V – Set of Atomic Systems E – Energy Links

Inputs

Controls

Discrete Time Model

4.1.1.

DC Components  Energy production and distribution systems: Diesel generator, Transfer Switch, UPS, REN Generation  Energy consumption systems: IT Servers and Cooling System  Energy distribution links: o Diesel generator energy: o Data center energy provisioning: o Equipment energy provisioning: o REN Distribution Link:  Energy consumption links: o Server consumption: o Cooling consumption:  Thermal energy link:  Wind Speed:  Predicted IT servers energy consumption due to real-time workload:  Predicted IT servers energy consumption due to delay-tolerant workload:  Diesel generator control: ,  Cooling control:  IT Servers control: ,  UPS control:  Transfer switch control: ,  Day Ahead Time Model - one hour granularity: | 1, ∀ 0  Intra Day Time Model ½ hour granularity: | 0.5, ∀ 0

Modelling the Energy Consumption Systems

The first atomic system modelled is IT Servers that consumes energy to execute the DC workload. The mechanism for exploiting the servers and associated workload flexible power demand at each timestamp is leveraging on shifting either in time (i.e. in the same DC) and space (i.e. relocation to partner DCs) the delay-tolerant workload. As in the case of queue based models [32], [33], we differentiate between

real-time workload, which has stringent requirements on real-time execution, and delay-tolerant workload, which can be executed anytime until a given deadline. The DC power demand is reduced at timestamp with the amount of power needed to execute the delay-tolerant load that is shifted at timestamp , ∈ 1, while the DC power demand at timestamp is increased with the amount of power needed to execute the delay-tolerant load shifted from timestamp . Table 2 shows the formalism applied on the IT Servers and the associated workload based flexibility mechanisms. It has as input the predicted energy consumption at timestamp split between the one generated by the execution of real time workload ( ) and that generated by the execution of delay tolerant workload ( ). The first output is the amount of energy consumed by the IT Servers as a system at timestamp as result of rescheduling and shifting the execution of delay tolerant workload. Table 2 – IT Server modeled as Atomic System.

,

I (inputs set)

∈ ,

O (output set) ∈

C (control set)

∈

S (internal state)

∗

f (transfer function) g (state function) Constraints

∈

1 C1: ∑

∗ 1

C2:

The second output is the amount of heat generated by the IT Servers as result of workload execution. According to the law of energy conservation, most of the electrical energy consumed by the IT Servers is transformed into heat, so the cooling system has to dissipate a mass of thermal energy approximately equal to the electrical energy consumed by the servers: . The control variable ∈ is used to determine how to split and shift percentages of the delay tolerant workload energy demand to future timestamps in the time window 0. . . As result the first constraint (C1) to be meet in the optimization process is ∑ 1. The internal state of the IT Servers as a system is given by a scheduling queue that stores the percentages of delay tolerant workload re-scheduled for execution at future timestamps. This is represented as an array

∈ that holds for each discrete timestamp in the optimization time window 0. . the energy consumption associated with the workload which execution has been delayed . The transfer function computes at each timestamp the amount of energy consumed by IT Servers as the sum of the energy associated with the execution of: (i) real time workload , (ii) delay tolerant workload shifted for execution at (given by ) from previous timestamps, and (iii) delay tolerant workload originally scheduled ∗ . The state of the IT Servers is for execution at and not shifted given by updated by the state function that computes how much of the delay tolerant workload is scheduled at each future timestamps of the optimization window. In consequence the second constraint (C2) to be meet in the optimization process is to limit the workload to be executed in a certain timestamp and associated energy consumption to the maximum energy consumption limit imposed by the characteristics of the IT Servers deployed ( ). As result the energy flexibility due to workload scheduling and shifting [20] is estimated by constructing the scheduling matrix ∈ ∗ line by line from 1 up to , where 1, is the prediction and optimization time window. The delay tolerant workload shifting pattern is given by relation: ∗

(22)

The second atomic system modeled is the Cooling System, which deals with removing the heat produced (i.e. ) by the IT Servers to keep the server room temperature inside the predefined set points. The amount of electrical energy consumed by the Cooling System to dissipate an amount of thermal energy is approximated using the models from [22] and [40] where a Coefficient of Performance (COP) is used to define the relation: (23) The power consumption flexibility of the Cooling System is leveraging on the usage of non-electrical cooling systems such as the TES to precool the DC and compensating the electrical one. When charging the TES its coolant (i.e. water based thermal tanks) is overcooled by using the electrical cooling at higher capacity resulting in an increased power demand. When TES is discharged the DC is cooled down using the precooled coolant while the electrical cooling system is used at lower intensity. The DC power demand at timestamp is decreased by the amount of power compensated by discharging TES (as consequence of lowering the intensity and demand of the electrical cooling system), while the DC power demand at timestamp , ∈ 1, is increased by the amount of power needed to charge energy in TES (as a consequence of increasing the intensity of the electrical cooling system to overcool the TES). Table 3 shows the formalism applied on the Cooling System and associated flexibility mechanism. In our model the input of the Cooling System is the heat to be removed while the output is the amount of energy consumed . The system internal state variable ( ) models the capacity of the thermal storage water tank and shows the amount of potential flexible energy in each timestamp. The two control

variables, ( , ) are used to specify the actions for charging and discarding the coolant from TES. The power consumption flexibility behavior of the Cooling System is modeled by the transfer function while the state function models the updating of the system internal state variable (i.e. how much energy is actually stored in TES) based the control variables for each discrete time period. The constraints defined on this atomic system limit the amount of energy that might be stored in TES between lower and upper limits (C3) considering the device hardware specifications. In a similar way the constraints on the control variables and (C4, C5) defining the amount of energy charged and discharged from TES in a single step are specified. Finally, the charging and discharging actions cannot happen simultaneously (C6). Table 3 - Cooling System modeled as Atomic System.

∈

I (inputs set) ∈

O (output set) ,

C (control set)

∈ ∈

S (internal state) f (transfer function) g (state function) constraints

1

∗

∗

C3: 0 C4: 0 C5: 0 C6: C7: 0

4.1.2.

∗

0

Modelling the Energy Production and Distribution Systems

The Transfer Switch is responsible for transferring the energy to the DC systems that have to consume it. In our model, the DC is connected to three energy sources (see Table 4): on-site renewable energy generation ( ), diesel generator ( ) and the Smart Grid (buying energy). The connection to the smart grid is bidirectional and that represent the amount and is represented by the two control signals, of energy bought or sold. The transfer switch has no internal state and its transfer , fed into the DC. The constraints function computes the total amount of energy

defined for this atomic system operation state that and must be bounded (C8, C9) and mutual exclusive (i.e. the DC can’t buy and sell energy at the same time - C10). The Uninterrupted Power Supply takes as input the total amount energy from the Transfer System ( ) and acts as a filter on the DC power supply, regulating the ). A UPS is composed of an electrical power that fed all the DC systems (output inverter, a rectifier and a set of Electrical Storage Devices (batteries) that have to provide back-up power to the DC in case of power shortage. The flexibility mechanism for the electrical storage device (ESD) is based on reducing the DC power demand at timestamp by the amount of power discharged from batteries and increasing the DC power demand at timestamp + , ∈ 1, by the amount of power charged in batteries. State of the art batteries have a higher charge-discharge life-cycle that allows them to be used more frequently, offering a certain level of flexibility for the DC power demand. Similar models can be found in [41], [42]. Table 4 – Transfer Switch modeled as Atomic System.

,

I (inputs set)

∈ ∈

O (output set) C (control set)

,

∈

S (internal state)

None

f (transfer function) g (state function) Constraints

None C8: 0 C9: 0 C10:

∗

0

The control signals and (see Table 5) represent the commands to charge or discharge energy from the batteries. The transfer function computes the amount of energy that reaches to the DC energy consumption systems considering the batteries charge loss factor during operation ( ) and discharge loss factor during operation ( ) which are specific to device characteristics. The state function updates the internal state variable at every discrete moment of time. The constraints defined limit the batteries charge and discharge values between an upper limit and a lower considering the depth of discharge rate imposed to limit the battery

damage (C11). Furthermore, the charge and discharge control signals are also bounded (C12, C13) and mutual exclusive (C14). The Power Distribution Unit (PDU) is responsible for distributing the output filtered energy from the UPS ( ) to the main systems that consume it: IT Servers ( ) and Cooling System ( ). It has no output, control variables, and internal states, while the single constraint defined aim to equalize the energy fed in the DC with the energy consumed (i.e. C15: ). The Diesel Generator has to produce enough energy to cover the DC demand in case of an emergency or power shortage. Due to the time inertia of Diesel Generator operation, until it reaches full operational status, the DC is powered by the UPS units. We model the Diesel Generator as a system with no inputs and internal state, having a single control variable (i.e. turn on the diesel generator). The Diesel Generator needs to be run for periodic maintenance and during this process it may also energy which can be used to operate the DC. The transfer function in this case estimates the maximum power supplied by the generators and we consider that during maintenance they are running at a certain percentage of the maximum capacity. We will exploit this potential flexibility by planning the periodic maintenance in moments of renewable energy production deficits or high energy prices in the local grid. Table 5- Uninterrupted Power Supply modeled as Atomic System.

I (inputs set)

∈

O (output set)

∈ ,

C (control set)

∈ ∈

S (internal state) f (transfer function) g (state function) Constraints

1 C11:

∗

∗

∗ C12: 0 C13: 0 C14:

∗

0

The Renewable Energy Generation system features a set of wind turbines to generate electrical energy. In our modeling, we have considered that the DC has deployed homogeneous wind turbines ( ) with blade length and power coefficient . Thus, the transfer function in this case aims at estimating the output energy generated ( ) by the wind turbines during each timeslot as [31]: ∗

∗

∗

∗

∗

(24)

where represents the air density, is the area of the disk and is the system input representing the wind speed during timeslot as provided by a weather forecast service [24]. 4.1.3.

DC Multi-Layer Proactive Optimization

The multi-layer proactive optimization technique described Section 3 is mapped and used on DC model as a SoS to optimize its operation and achieve various sustainability goals. Accordingly, two optimization process layers working at different time granularities have been defined (see Figure 3):  The day-ahead optimization estimates the DC energy input from all sources for the next operational day and plans the DC operation considering the potential flexibilities at atomic systems level. It generates a plan of energy flexibility optimization actions that must be executed during day at each 24 hours. The granularity in this case is 1 .  The intra-day optimization refines, adjusts and improves the optimization actions plan determined for day by using a finer prediction on a smaller time window. In this case the granularity is ½ and considers only 4 interval 4 ∗ ,4 ∗ 1 ahead for generating refinements of the optimization actions for the next interval of 1 . The steps of multi-layer optimization process are (see Figure 3): 1. The prediction module computes the predictions for the day-ahead plan, using the discrete time model with granularity of one hour and time window length of 24. It computes the predicted values for the next 24 hours and use them for instantiating the DC SoS model. 2. The day-ahead optimization process is called to compute the best action plan 3. 4.

) for the day-ahead time window to minimize the objective function. ( The resulting action plan (and associated control variables) are saved At every 4 hours, the intra-day optimization process is called. It considers historical data to compute a finer set of predictions with only 8 values on a 4 hour time window. The predicted values are sent to the DC SoS model that uses a discrete time model with granularity ½ hour and time window length 8.

5.

6. 7. 8.

The set of actionss from the dayy-ahead plan scheduled for this intraday interval i are ex xtracted and uused to generaate new states of DC modelled atomic systeems. The intra-day i optimization proccess is called to t compute a correction of the action plan to minimize the t objective ffunction. The new n plan ∪ is saved. s Stepss 4-7 are repeaated 6 times ( every 4 hourss for 24 hours)).

Fig gure 3 - DC mu ulti-layer optim mization processses.

We haave defined tw wo optimizatioon objectives that are mutuaally exclusivee. The first f optimizattion objectivee aims at deteermining the best values fo for control variables and associateed actions at each timestam mp of the optimization o w window to exploit attomic systemss flexible eneergy for minim mizing the DC C operational cost with energy. Inn other wordss, the atomic systems operration is optim mized such thhat the DC will buy energy from the t grid whenn the prices arre low and selll the on –sitee produced energy when w prices aree high: ∗ 1. .

: ,

∑

∗

→ , ∗

(25)

The secoond optimizattion objectivee aims at dettermining thee best controol variable sequence and associateed actions a tiime window T to minimizee the Euclidean an distance between the DC eneergy consumpption curve and the on-site renewablle energy generatioon profile. In other o words thhe DC operatiion is optimizzed such that tthe on-site producedd renewable en nergy is used aas much as po ossible when available: a

:

∗ 1. .

→ ,

∑

,



(26)

4.2. Simulation Environment and Results For estimating the potential energy flexibility and economic impact of our SoS based multi-layer optimization technique we have conducted numerical simulation-based experiments considering the hardware systems characteristics and operation (see Table 6) of a reference operational medium scale DC and the workload energy consumption variation from the DC operation logs presented in [23]. Table 6 – Characteristics of the test bed DC. Atomic System

Characteristics

Upper Bounds

Cooling System

Coefficient of Performance 1.33

Maximum Cooling Capacity 4000

0.99

Charge Loss Rate

1.1

Discharge Loss Rate

Maximum Charge Rate 1000 Maximum Discharge Rate 1000 Maximum TES Capacity 3000

IT Severs

9000 Servers with characteristics of HP 360 DL

Electrical Storage System

Charge Loss Rate

= 0.8,

Discharge Loss Rate

= 1.2,

Maximum Power Consumption 3000 Maximum Charge Rate 1000 Maximum Discharge Rate 1000 , Maximum Storage Capacity 3000

Renewable Energy Sources

Air Density

1.23

Blade Length

3

20

Power Coefficient Number of Turbines

Diesel Generator

/

0.4

Maximum Energy Generation Capacity 5000

16 Maximum Generation Capacity 5000

We have constructed a DC flexibility simulation environment which is available online at http://193.226.5.79/final-prototype/#/. Figure 4 shows the main components of our simulation environment.

Fig gure 4 – Simullation environm ment components.

The Sim mulation Manaager generate s the simulattion time acccording to ouur defined discrete time t models and determinnes the impactt of action pllans executionn on each modeled atomic systeem behavior and the associated enerrgy consumpption. The Simulatioon Database sttores the energgy consumption/production n data both at individual atomic syystem level an nd aggregatedd at DC SoS level. l The Energy Budget PPrediction Module calculates c the current and fforecasted eneergy consumption/ productiion values for each atomic systeem modeled using the tw wo different time granulaarities and predictionn interval. Wee have used tiime series preediction techniques on histoorical data regardingg DC operatio on stored in tthe Simulation Database. To T implement nt the time series preediction functtionality we hhave used the R language [43], which iis strongly oriented towards statistical computting. The pro ocess of foreccasting consiists of the followingg steps: (i) gett historical daata required fo or training the prediction m model from the Simullation Databasse, (ii) set the periodicity vaalues of the neew streams daata, (iii) fit the prediction model with w the new w data and (iv v) generate predictions for the given number of o steps in the t future. T The DC Multti-Layer Optiimizer implem ments the proactivee optimization n method for SoS describeed in this pap per and uses a MINLP solver to deal with ou ur DC SoS Moodel equation ns and constraaints, while coonsidering the new data d stored in the t Simulationn Database. 4.2.11.

Energy Optimization O Results

We aim to evaluatte the effectivveness of our SoS based multi-layer m opptimization process inn improving th he modeled D DC operation to t meet differeent sustainabillity goals. Firstlyy, we have ussed our optim mization proceess to schedule the DC opeeration for the next operational o daay to consumee as much as possible locallly produced rrenewable energy. In this case th he objective fu function preseented in relation (26) has bbeen used. Figure 5 presents thee non-optimall situation siimulated in which w there iis a large differencee between th he renewable energy geneeration and th he DC energyy demand profiles. During the firrst 6 hours off the next day y it is forecasted that the D DC energy demand will exceed with up to 20% the am mount of renewable energgy locally producedd. After hour 7 due to intennsive wind th he renewable energy generration will

exceed thhe DC deman nd, a high rennewable energ gy peak being g forecasted inn the time between hours h 13 and 16.

Figu ure 5 – Initial renewable r enerrgy generation n and DC energ gy demand proofiles.

The optimization o process p is trigggered to corrrect the ineffiiciencies idenntified and the optim mization action n plan from F Figure 6 is co omputed. Thee DC energy flexibility mechanissms are used to t intelligentlyy control each h atomic system inside the D DC and as result to adapt a the DC SoS energy deemand curve to the generattion profile. Foor the first 6 hours interval when the DC energy demaand exceed the renewablle energy productioon, the energy y consumptionn has been reeduced by shifting the delaay-tolerant workloadd ahead in tim me to intervalls with less load l (i.e. hours within 7-116 interval where prroduction is actually largeer than the consumption). Furthermorre, the by discharging the batteriees or the therm mal storage ex xtra energy is made availablle.

Figurre 6 – Optimiza ation result: DC C energy dema and is followin ng the generatiion profile.

In the tim me interval beetween hours 7 and 16 where the amoun nt of renewabble energy generatedd is larger thaan the actual D DC consumption, besides executing e the additional workloadd shifted from m previous houurs, the optim mization proceess decides too recharge the batterries and TES.. By adjustingg the control of the modeled atomic sysstems, the DC can adapt a its overaall energetic ffootprint to reeach the target objective off using for its operattion all the lo ocally generateed renewable energy. As result r the DC is able to use with up u to 11% mo ore renewable energy locallly produced. Seconndly, we havee used our So S model and optimization technique forr reducing the DC costs c with energy considerinng the objectiive function defined d in relaation (25). Figure 7 shows the inittial suboptimaal situation in which the DC C energy dem mand is not correlatedd with the day y-ahead energyy prices.

Figure 7 – Initial DC en nergy demand d and energy prices profiles.

The optimization o process p will bbe triggered to o compute thee best configuuration for DC atom mic system paarameters houurly operation that will allo ow the DC aaggregated energy prrofile to be sh hifted consideering the hourrly price of en nergy in the llocal grid. Because the energy iss cheap durinng the time in nterval between hours 6 an and 15 the emergentt DC energy demand will be increased in the attemp pt of buying and using energy frrom the grid as much as possible (seee Figure 8). The T IT serverrs’ energy consumpttion is increaased by reschheduling and shifting in this interval tthe delaytolerant workload w from m the previouss time intervals with higher energy pricess (i.e. from interval between b hourss 1 and 6). Fuurthermore, th he TES is used to cool dow wn the DC thus the electrical e coolling system iss turned off during d the hou urly time interrval 1 to 5 which feature high en nergy prices. Then, when the t energy prrice is low, thhe TES is charged (its ( coolant is overcooled) uusing extra en nergy bought from the gridd. Because after the hour h 15 the en nergy price iss continuously y rising, the op ptimization prrocess will not issue any actions fo or shifting woorkload for thiis period. Finaally, as result oof running the optim mization process the DC opeerational cost with w energy was w reduced byy 14%.

Figgure 8 - Optimization action p plan for minim mizing the DC costs with ener ergy.

4.2.2. Preediction Accu uracy and Inttra-day Refin nements One of thhe main advaantages of ourr hierarchical optimization process for SSoS is the high tolerrance to prediction errors, bbecause the op ptimizations im mplemented aat different levels andd time granulaarities are ablle to limit theiir influences on o the system m emergent behavior.. For examplle, in case oof our modelled DC it is likely that dday-ahead renewable energy geneeration predicttions are not 100% 1 percent accurate and the actual generatioon profile coulld be slightlyy different. Th he predictions inaccuracies generated due to a long predictio on window (ii.e. 24 hours ahead) will affect a the quallity of the day-aheadd optimization plan and ass result the emergent DC energy profille will not match thee actual energy y generation pprofile. In thiss case the intrra-day optimizzation processses are triggerred to correct and refine the atomic system lev vel optimizati on actions pllanned for thee next day coonsidering better preediction on sm maller time fframe (i.e. on nly 4 hours ahead). Figuree 9 (TOP) presents the t initial suboptimal situattion in which there is a diffference betweeen the day ahead rennewable energ gy generationn predictions and a the actuaal production values. In consequeence in the firsst four hours of the day theere will be available with aabout 11% more renewable energy y than forecassted. The intraa-day optimizzation processs will have to adjust the initially computed c optiimization plan n in such a way that all ext xtra energy available is consumed and the overrall DC energ gy demand is increased. Coonsidering the therm mal inertia of DC D atomic syystem modeled d and the lower granularityy the intraday optim mization proccess decides to use the en nergy to charrge the TES and ESD systems for f later use (ssee Figure 9 (D DOWN)).

Figuree 9 – Intra – Day optimizatioon action plan for f interval 1 to t 4: (TOP) thee initial situation n with generattion profile misss predicted; (D DOWN) the acction plan refinnements.

Figure 100 (TOP) preseents the situatiion for the tim me interval bettween hours 5 and 9. In this case the actual eneergy generatioon profile is lo ower than the forecasted onne and the amount of o energy locaally produced to be used in n DC operation is lower thaan the one expected.. The actual difference d betw tween the foreecasted and actual renewabble energy generatioon profiles bein ng of about 200%. The intraa-day optimizaation process w will refine the day-aahead optimizzation plan inn such a way y that the oveerall DC is loowered to follow thhe actual valu ues. The elecctrical cooling g system is used u at lowerr intensity values whhile this is beiing compensatted by using the TES (see Figure F 10 (DO OWN)).

Figure 10 – Intra – Day D optimizatioon action plan for interval 5 to 8: (TOP) thhe initial situation n with generattion profile misss predicted; (D DOWN) the acction plan refinnements.

We have conducted a set of expperiments to determine d thee intra-day opptimization process potential p to co ompensate thee day-ahead reenewable enerrgy predictionn errors by refining and a improving g the optimizaation action pllan. We have considered alll six intraday optim mization processes, each o f 4 hours’ tim me interval, corresponding to a dayahead optimization plaan implementaation. Our obj bjective was to o evaluate thee intra-day optimizattion process potential p to adj djust the DC operation by ad ddressing the mismatch between the day-aheaad planed eneergy demand and the actu ual available rrenewable energy loocally produced. We have considered prediction p erro ors ranging fr from -40% (the actuaal renewable energy e generaation is with 40% 4 less than n the forecasteed one) up to +40% % (the actual renewable eenergy generation is with h 40% more than the forecastedd one). The refinement r cappacity of the intra-day opttimization is iinfluenced by the chharacteristics of DC hardw ware systemss and of enerrgy flexibilityy potential available in that time in nterval.

Figure 11 - Intra-Day po otential to: (TO OP) compensa ate renewable energy e predicti tion errors and (DOW WN) increase renewable r enerrgy usage as reesult of refining the optimizaation plans.

Figure 111 (TOP) show ws the relationn between thee energy generation predicttion errors which vaariation is shown on the hoorizontal axis and the degreee of matchingg between the day-aahead and inttra-day DC ennergy consum mption profilees on one sidde and the actual ennergy productiion profile onn the other siide. As it can n be seen thee intra-day optimizattion process iss able to imprrove the DC energy e demand curve to bet etter match and follow the generattion profile inn comparison with the day y ahead optim mization. If during thhe intra-day optimization, tthe energy geeneration proffile is less thaan the one predictedd by the day-aahead processs (i.e. left quaadrant of the chart), the reefinements brought to the optim mization plann can improv ve the match h between thhe energy consumpttion and prod duction by usiing flexibility y mechanisms such as TESS and ESD dischargee to compensaate the energy deficit. In thee other case (i.e. right quadrrant of the chart) thee energy generation profile is bigger than n the one pred dicted by the dday-ahead process thhe intra-day optimization o iss able to increease the DC energy e demandd to better match the production. As result off using the in ntra-day optim mization the aamount of renewable energy usag ge for DC opperation is increase with about a 9% durring a day which reppresents abou ut 5200kWh oof renewable energy locally y produced (ssee Figure

11 (DOW WN)). Consid dering the eneergy mix for powering thee DC this reppresents a decrease with 3.4% off DC carbon ffootprint in co omparison to the case in w which only the day-aahead planner will had beenn used. 4.2.3. Sollution Scalab bility The 3rd evaluation scenario aim ms to determinee the scalability of our SoSS modeling and multii-layer optimiization techniqque with respect to the num mber of atomiic systems and the leength of the tim me window (ssee Table 7). Table 7 – Optimization problem size w with respect to o SoS composition and time w window. No. Atoomic Systems 1

IT Serverss Yes

Cooling System No

UPS No

Diesell Generrator No

2

Yes

Yes

No

No

continuous varriables

3

Yes

Yes

Yes

No

2 continuous vaariables

4

Yes

Yes

Yes

Yes

3

Probleem size continuous variablees

continuous vvariables

and T integer variablees

Two approaches a haave been used for solving th he MINLPs op ptimization prroblems of our SoS based model:: Lingo Mathhematical solv ver [29] and a greedy solut ution using genetic heuristic h [30]. The results off the experiments run with both algorithm hms can be viewed inn Figure 12. In I Figure 12aa the two-horiizontal axis reepresent the nnumber of atomic syystem modeled in the SoS and the length h of the optim mization winddow, while the verticcal axis repreesents the num mber of iterattions performed by the alggorithm to find an approximate a solution s withh a population n of 100.000 individuals. It can be noticed thhat the largest increase in tthe number of iterations is given by thee length of the optim mization time window w (seee also Table 7 for the quadrratic relation bbetween and probllem size). a)

b)

Figure 12 1 – Multi-layeer optimization n scalability evaluation (a- relation betweenn SoS size and win ndow length; bb comparison between Lingo o and Genetic Algorithm forr T = 24).

Figure 122b shows thee gradient of the iterationss for both mathematical m ssolver and genetic heuristic h in caase of an optiimization tim me window 24. Even iif for few

atomic systems modeeled, the Linngo solver haas a leaner slope s than thhe genetic algorithm m, over the entire testingg interval itt shows a quadratic q behhavior. In comparison, the genettic heuristic eexhibits a neaar-linear behaavior. Thus, fo for a large number of o atomic systeems and contrrol variables, the t genetic heuristic scales better. To im mprove the computational time needed d to solve ou ur model matthematical optimizattion problem using solverss such as Lin ngo we have used a decoomposition techniquee similar to the t one preseented in [44]. Thus, to red duce the com mbinatorial search coomplexity we have consideered for optim mization each modeled m atom mic system individuaally and not th he entire DC. Because our optimization o problem p is NPP-hard, by splitting the t model in individual i com mponents the solving time decreases d expponentially with the drawback d thatt the computeed solution miight be a locall optimum ratther than a global onne which is alw ways found inn the case of co onsidering thee entire SoS m model.

Figure 13 – Multi-layer optimization n decomposition evaluation (T TOP - iterationns for the Lingo solver; BOTTOM- solution quality)).

To prove this, we have performed a set of experiments on 11 day-ahead optimization scenarios using our DC SoS model, but considering only the IT Server continuous and Electrical Cooling as atomic systems thus having only variables (see Table 7). The experiments conducted aimed to determine the decrease in computational complexity if the DC operation as a SoS is optimized considering each individual atomic system and the impact of the decomposition on the overall quality of the solution (see Figure 13). For Lingo, we considered the number of iterations performed to compute the optimization solution as complexity metric. Figure 13 (TOP) shows that by solving individually the optimization problems associated with IT Server and Electrical Cooling atomic systems models (as result of by decomposing the overall DC as a SoS optimization problem) the number of iterations needed by the Lingo solver to compute the solutions is reduced significantly, by about 5 times. To determine the optimization action plan solution quality, we have evaluated the DC operation cost reduction compared with the minimal baseline cost. Figure 13 (BOTTOM) shows that the SoS optimization obtains in each scenario better cost saving results reducing the overall DC operations costs with up to 7%, while the individual system optimization reduces the overall costs with roughly 5%. Thus, we can conclude that the price for reducing the solution computation time by performing individual system optimization is a degradation of around 20% of the solution quality.

5. Conclusion The paper presents a diagrammatic block-based mathematical formalism for representing and optimizing the SoS operation. Our formalism is based on the concepts of discrete time, atomic systems and the interconnection of atomic system that leads to the creation of complex SoS. On top of this we have proposed a proactive multi-layer optimization technique that aim at streamlining the operation of SoS using hierarchical optimization processes at different time granularities which mitigates the impact of prediction errors on SoS emergent behavior. The model is applied and used to represent the energy systems and flows within a modern DC, which is a case of large complex system. Furthermore, the presented proactive optimization methodology is adapted to facilitate the integration of DCs with the Smart Grid by allowing them to use as much as possible of the locally produced renewable energy and transact energy to decrease operational costs. The optimization problems are addressed using modern mathematical solvers and genetic based heuristic obtaining a reduction of DC carbon footprint with 5% and of the energy costs of 14%. The results also show that the constructed DC SoS model can be optimized using time windows to 24 hours, corresponding to an operational day, while the combinatorial optimization process complexity can be decreased using a model decomposition technique while keeping solutions quality in reasonable limits. As future work, we propose to enhance the definition of the SoS by allowing heterogeneous time models for the atomic sub-systems, such as introducing continuous time models alongside with the discrete time-models. With this

improvement, the DC could be more accurately modelled and the inertia of the subsystems could be taken into account.

6. Acknowledgment This work has been carried out in the context of the GEYSER project, co-funded by the EU Commission as part of the 7th Research Framework (FP7-SMARTCITIES2013) and it was supported by a grant of the Romanian National Authority for Scientific Research and Innovation, CNCS/CCCDI – UEFISCDI, project number PNIII-P2-2.1-BG-2016-0076, within PNCDI III. The article is also based upon work from COST Action IC1406 High-Performance Modelling and Simulation for Big Data Applications (cHiPSet), supported by COST (European Cooperation in Science and Technology).

7. References [1] Z. Fang, D. DeLaurentis, Dynamic Planning of System of Systems Architecture Evolution, Procedia Computer Science Volume 28 (2014), pp. 449-456, http://dx.doi.org/10.1016/j.procs.2014.03.055 [2] M. Menarini, F. Seracini, X. Zhang, T. Rosing, I. Krüger, Green Web Services: Improving Energy Efficiency in Data Centers via Workload Predictions, 2nd International Workshop on Green and Sustainable Software (GREENS), San Francisco, CA, 2013, pp. 8-15 http://dx.doi.org/10.1109/GREENS.2013.6606416 [3] C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock Systems of Systems Engineering: Basic Concepts, Model-Based Techniques, and Research Directions. ACM Computing Surveys 48(2) (2015), http://dx.doi.org/10.1145/2794381 [4] J. E. Campbell, D. E. Longsine, D. Shirah, and D. J. Anderson, System of Systems Modeling and Analysis, Sandia National Laboratories Report 2005, http://prod.sandia.gov/techlib/access-control.cgi/2005/050020.pdf [5] Y. Qi, Z. Liu, H. Chen, J. Han, Distributed Combat System of Systems Network Modeling, Journal of Networks. 8(8), (2013), pp. 1906-1912, http://dx.doi.org/10.4304/jnw.8.8.1906-1912 [6] COMPASS EU FP7 project, Comprehensive Modelling for Advanced Systems of Systems, http://www.compass-research.eu/ [7] A. Gorod, R. Gove, B. Sauser and J. Boardman, System of Systems Management: A Network Management Approach, IEEE International Conference on System of Systems Engineering, 2007, pp. 1-5, http://dx.doi.org/10.1109/SYSOSE.2007.4304218 [8] V. Mahulkar, S. McKay, D. E. Adams and A. R. Chaturvedi, System-of-Systems Modeling and Simulation of a Ship Environment With Wireless and Intelligent Maintenance Technologies, IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans 39 (6) (2009), pp. 1255-1270, http://dx.doi.org/10.1109/TSMCA.2009.2025140

[9] T. R. Ender, A Top-Down, Hierarchical, System-of-Systems Approach to the Design of an Air Defense Weapon, Georgia Tech Theses and Dissertations, 2006, http://hdl.handle.net/1853/11570 [10] J. S. Topper and N. C. Horner, Model-Based Systems Engineering in Support of Complex Systems Development, Johns Hopkins Apl Technical Digest 32(1) (2013), pp. 419-432 http://www.jhuapl.edu/techdigest/TD/td3201/32_01-Topper.pdf [11] AMADEOS EU FP7 Project, Architecture for Multi-criticality Agile Dependable Evolutionary Open System-of-Systems, http://amadeos-project.eu/wpcontent/uploads/2015/07/AMADEOS_D2.2_v2.6-final.pdf [12] E. Ferrario, System-of-systems modeling and simulation for the risk analysis of industrial installations and critical infrastructures. Engineering Sciences Thesis. Ecole Centrale Paris, 2014, https://tel.archives-ouvertes.fr/tel-01127194 [13] S. Y. Han, D. DeLaurentis, Development Interdependency Modeling for Systemof-Systems (SoS) using Bayesian Networks: SoS Management Strategy Planning, Procedia Computer Science 16 (2013) pp. 698-707, http://dx.doi.org/10.1016/j.procs.2013.01.073 [14] E. A. Lee, The Past, Present and Future of Cyber-Physical Systems: A Focus on Models, Sensors 15(3) (2015) pp. 4837-4869, http://dx.doi.org/10.3390/s150304837 [15] W. C. Baldwin and B. Sauser, Modeling the characteristics of system of systems, IEEE International Conference on System of Systems Engineering (SoSE), Albuquerque, NM, 2009, pp. 1-6. http://ieeexplore.ieee.org/document/5282345/ [16] L. Wang, M. Törngren, M. Onori, Current status and advancement of cyberphysical systems in manufacturing, Journal of Manufacturing Systems, 37 (2) (2015) pp. 517-527, http://dx.doi.org/10.1016/j.jmsy.2015.04.008 [17] H. Sayama, Introduction to the Modeling and Analysis of Complex Systems, Open SUNY Textbooks, ISBN: 978-1-942341-09-3, Print edition edition (August 13, 2015). [18] GEYSER EU FP7 project, Green nEtworked data centers as energY proSumers in Smart Cities: http://www.geyser-project.eu/ [19] T. Cioara, I. Anghel, M. Bertoncini, I. Salomie, D. Arnone, M. Mammina, T. H. Velivassaki, M. Antal, Optimized flexibility management enacting Data Centres participation in Smart Demand Response programs, Future Generation Computer Systems, Available online May 2016 http://dx.doi.org/10.1016/j.future.2016.05.010 [20] T. Cioara, I. Anghel, M. Antal, S. Crisan, I. Salomie, Data center optimization methodology to maximize the usage of locally produced renewable energy, Proceedings of Sustainable Internet and ICT for Sustainability (SustainIT), Madrid, 2015 http://dx.doi.org/10.1109/SustainIT.2015.7101363 [21] T. Cioara, I. Anghel, I. Salomie, M. Antal, M. Bertoncini, and D. Arnone, Optimizing the power factor of data centers connected to the smart grid, Proceedings of the 5th International Workshop on Energy Efficient Data Centres. NY, USA, Article 3, 2016, http://dx.doi.org/10.1145/2940679.2940682 [22] Y. Zhang, Y. Wang and X. Wang, TEStore: exploiting thermal and energy storage to cut the electricity bill for datacenter cooling, Proceedings of the 8th International Conference on Network and Service Management, pp. 19-27, Las Vegas, Nevada, 2012, http://dl.acm.org/citation.cfm?id=2499409

[23] C. Wang, B. Urgaonkar, Q. Wang, G. Kesidis and A. Sivasubramaniam, Data Center Power Cost Optimization via Workload Modulation, IEEE/ACM 6th International Conference on Utility and Cloud Computing, pp. 260-263, 2013, http://dx.doi.org/10.1109/UCC.2013.52 [24] Forecast for Developer API, Online at http://forecast.io/ [25] J. Moore, J. Chase, P. Ranganathan, Making scheduling cool: temperature-aware workload placement in data centers, Proceedings of the annual conference on USENIX Annual Technical Conference, Berkeley, CA, USA, 2005, http://dl.acm.org/citation.cfm?id=1247365 [26] P. Belotti, C. Kirches, S. Leyffer, J. Linderoth, J. Luedtke, and A. Mahajan, Mixed-Integer Nonlinear Optimization, Acta Numerica 22 (2013) pp. 1-131, https://doi.org/10.1017/S0962492913000032 [27] R. Kannan and C. Monma, On the computational complexity of integer programming problems. In Henn, R., Korte, B., and Oettli, W., editors, Optimization and Operations Research, vol. 157 of Lecture Notes in Economics and Mathematical Systems, 1978, pp. 161–172, http://dx.doi.org/10.1007/978-3-642-95322-4_17 [28] Time Series and Forecasting, http://www.statmethods.net/advstats/timeseries.html [29] LINGO Software for Integer Programming, http://www.lindo.com/ [30] Y.-C. Lin, K.-S. Hwang, F.-S. Wang, A mixed-coding scheme of evolutionary algorithms to solve mixed-integer nonlinear programming problems, Computers & Mathematics with Applications 47(8) (2004) pp. 1295-1307. https://doi.org/10.1016/S0898-1221(04)90123-X [31] K. Grogg, Harvesting the Wind: The Physics of Wind Turbines, Carleton College, 2005, http://apps.carleton.edu/campus/library/digitalcommons/assets/pacp_7.pdf [32] Z. Liu, A. Wierman, Y. Chen, B. Razon, N. Chen, Data center demand response: Avoiding the coincident peak via workload shifting and local generation, Performance Evaluation 70(10) (2013), pp. 770–791 http://dx.doi.org/10.1016/j.peva.2013.08.014 [33] Y. Yao, L. Huang, A. Sharma, L. Golubchik and M. Neely, Data centers power reduction: A two time scale approach for delay tolerant workloads, Proceedings of INFOCOM 2012, pp. 1431-1439. https://doi.org/10.1109/INFCOM.2012.6195508 [34] Q. Tang, T. Mukherjee, S. K. S. Gupta and P. Cayton, Sensor-Based Fast Thermal Evaluation Model For Energy Efficient High-Performance Datacenters, Fourth International Conference on Intelligent Sensing and Information Processing, Bangalore, 2006, pp. 203-208. https://doi.org/10.1109/ICISIP.2006.4286097 [35] T. Cioara, I. Salomie, I. Anghel, I. Chira, A. Cocian, E. Henis, R. Kat, A dynamic power management controller for optimizing servers’ energy consumption in service centers, Service-Oriented Computing, Lecture Notes in Computer Science 6568, pp. 158-168, 2011, http://dx.doi.org/10.1007/978-3-642-19394-1_17 [36] M. Ghamkhari and H. Mohsenian-Rad, Data Centers to Offer Ancillary Services, 2012 IEEE Third International Conference on Smart Grid Communications (SmartGridComm), Tainan, 2012, pp. 436-441, https://doi.org/10.1109/SmartGridComm.2012.6486023 [37] X. Lu, F. Kong, J. Yin, X. Liu, H. Yu and G. Fan, Geographical Job Scheduling in Data Centers with Heterogeneous Demands and Servers, IEEE 8th International

Conference on Cloud Computing, New York City, NY, 2015, pp. 413-420. https://doi.org/10.1109/CLOUD.2015.62 [38] Z. Zhang, C. C. Hsu and M. Chang, Cool Cloud: A Practical Dynamic Virtual Machine Placement Framework for Energy Aware Data Centers, IEEE 8th International Conference on Cloud Computing, New York City, NY, 2015, pp. 758765. https://doi.org/10.1109/CLOUD.2015.105 [39] I. Anghel, T. Cioara, I. Salomie, G. Copil, D. Moldovan, C. Pop, Dynamic Frequency Scaling Algorithms for Improving the CPU’s Energy Efficiency, IEEE 7th International Conference on Intelligent Computer Communication and Processing, pp. 485 – 491, 2011, http://dx.doi.org/10.1109/ICCP.2011.6047920 [40] S. Lee, R. Urgaonkar, R. Sitaraman and P. Shenoy, Cost Minimization Using Renewable Cooling and Thermal Energy Storage in CDNs, 2015 IEEE International Conference on Autonomic Computing, Grenoble, 2015, pp. 121-126 https://doi.org/10.1109/ICAC.2015.39 [41] L. Li, W. Zheng, X. Wang and X. Wang, Coordinating Liquid and Free Air Cooling with Workload Allocation for Data Center Power Minimization, Proceedings of 11th International Conference on Autonomic Computing, Philadelphia, PA, 2014, pp. 249-259. https://www.usenix.org/node/183115 [42] W. Zheng, K. Ma and X. Wang, Exploiting thermal energy storage to reduce data center capital and operating expenses, IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), Orlando, FL, 2014, pp. 132-141. https://doi.org/10.1109/HPCA.2014.6835924 [43] The R Environment, https://www.r-project.org/about.html [44] H. Stefansson, Discrete and continuous time representations and mathematical models for large production scheduling problems: A case study from the pharmaceutical industry, European Journal of Operational Research (2011) 383–392 [45] Bourdena, A., Mavromoustakis, C.X., Kormentzas, G., Pallis, E., Mastorakis, G. and Yassein, M.B., 2014. A resource intensive traffic-aware scheme using energyaware routing in cognitive radio networks. Future Generation Computer Systems, 39, pp.16-28. https://doi.org/10.1016/j.future.2014.02.013 [46] Dimitriou, C.D., Mavromoustakis, C.X., Mastorakis, G. and Pallis, E., 2013, June. On the performance response of delay-bounded energy-aware bandwidth allocation scheme in wireless networks. In Communications Workshops (ICC), 2013 IEEE, Int. Conf. on (pp. 631-636). IEEE. https://doi.org/10.1109/ICCW.2013.6649310 [47] Mavromoustakis, C.X., Bourdena, A., Mastorakis, G., Pallis, E. and Kormentzas, G., 2015. An energy-aware scheme for efficient spectrum utilization in a 5G mobile cognitive radio network architecture. Telecommunication Systems, 59(1), pp.63-75. https://doi.org/10.1007/s11235-014-9885-4 [48] Jordi Mongay Batalla, George Mastorakis and Constandinos X. Mavromoustakis and Jerzy Żurek, "On cohabitating networking technologies with common wireless access for Home Automation Systems purposes", in the Special Issue on Enabling Wireless Communication and Networking Technologies for the Internet of Things, IEEE Wireless Communication, 2016. https://doi.org/10.1109/MWC.2016.7721745

[49] Jordi Mongay Batalla, Constandinos X. Mavromoustakis, George Mastorakis, Daniel Negru, Eugen Borcoci "Evolutionary Multiobjective Optimization algorithm for multimedia delivery in critical applications through Content Aware Networks", Journal of Supercomputing (SUPE), Springer International Publishing, May 2016, pp.1-24. https://doi.org/10.1007/s11227-016-1731-x [50] Georgios Skourletopoulos, Constandinos X. Mavromoustakis, George Mastorakis, Jordi Mongay Batalla and John N. Sahalos, “An Evaluation of CloudBased Mobile Services with Limited Capacity: A Linear Approach”, accepted/to appear to the Soft Computing Journal, Springer/Berlin-Heidelberg, 2016, pp.1-8. https://doi.org/10.1007/s00500-016-2083-4 [51] Constandinos X. Mavromoustakis, George Mastorakis, Athina Bourdena, Evangelos Pallis, "Energy Efficient Resource Sharing using a Traffic-oriented Routing Scheme for Cognitive Radio Networks", IET Networks Journal, Volume 3, Number 1, March 2014 (IEEE DL), pp. 54-63. https://doi.org/10.1049/ietnet.2013.0132 [52] Vasile, Mihaela-Andreea, Florin Pop, Radu-Ioan Tutueanu, Valentin Cristea, and Joanna Kołodziej. "Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing." Future Generation Computer Systems51 (2015): 61-71. [53] Sfrent, Andrei, and Florin Pop. "Asymptotic scheduling for many task computing in big data platforms." Information Sciences 319 (2015): 71-91. https://doi.org/10.1016/j.ins.2015.03.053 [54] Castiglione, Aniello, Marco Gribaudo, Mauro Iacono, and Francesco Palmieri. "Modeling performances of concurrent big data applications." Software: Practice and Experience 45, no. 8 (2015): 1127-1144. https://doi.org/10.1002/spe.2269 [55] DArienzo, Maurizio, Mauro Iacono, Stefano Marrone, and Roberto Nardone. "Estimation of the energy consumption of mobile sensors in WSN environmental monitoring applications." In Advanced Information Networking and Applications Workshops (WAINA), 2013 27th International Conference on, pp. 1588-1593. IEEE, 2013. https://doi.org/10.1109/WAINA.2013.33 [56] Gorod, Alex, Brian E. White, Vernon Ireland, S. Jimmy Gandhi, and Brian Sauser. Case studies in system of systems, enterprise systems, and complex systems engineering. CRC Press, 2014.

Marcel Antal received his Dipl.-Ing. Degree in Computer Science in 2013 and is now a PhD student at the Technical University of Cluj-Napoca. His expertise includes artificial intelligence, green IT, optimization heuristics and knowledge engineering.

Claudia Pop received her Dipl.-Ing. Degree in Computer Science in 2014 and she is now a PhD student at the Technical University of Cluj-Napoca. Her expertise includes optimization heuristics, green IT, artificial intelligence and bio-inspired computing.

Dr. Tudor Cioara is Lecturer of Computer Science and member of the Distributed Systems Research Lab of the Technical University of Cluj-Napoca. He finalized his PhD studies with the thesis “Context aware adaptive systems with applicability in green service centres”. He is a member in EU projects on energy efficiency and Outreach Coordinator for COST Action CA15140, “Improving Applicability of NatureInspired Optimisation by Joining Theory and Practice”. His current research interest is focused on green IT, data centres optimization, nature inspired heuristics, smart grid integration and big data analytics. Dr. Ionut Anghel received the PhD degree in Computer Sciences at the Technical University of ClujNapoca in 2012 with the thesis “Autonomic computing techniques for pervasive systems and energy efficient data centres”. Currently he is a Lecturer of Computer Science with main research areas: green IT, semantic web, autonomic computing, service oriented distributed computing and context awareness.

Prof. Ioan Salomie is Professor of Computer Science at the Technical University of Cluj-Napoca, former invited professor at University of Limerick and Loyola College in Maryland and head of Distributed Systems Research Lab. His research expertise include distributed computing and systems, context-aware systems, autonomic systems, green computing, self-adaptive bio-inspired systems and knowledge engineering.

Prof. Florin Pop received his PhD in Computer Science at the University POLITEHNICA of Bucharest in 2008. He received his MSc in Computer Science in 2004 and the Engineering degree in Computer Science in 2003, at the same University. He is Associate Professor within the Computer Science Department and also an active member of Distributed System Laboratory. His research interests are in scheduling and resource management (decentralized techniques, re-scheduling), multi-criteria optimization methods, Grid middleware tools and applications development (satellite image processing an environmental data analysis), prediction methods, self-organizing systems, contextualized services in distributed systems.

• • • • •

Systems of Systems modelling and operation optimization Multi-layer proactive optimization through Mixed Integer Nonlinear Programming Use case on Data Centre as large scale complex system integrated into smart energy grids DC energy consumption and production systems modelling and operation optimization to meet sustainability goals Simulation based evaluation