Multi-Objective Communication-Aware Optimization for Virtual Machine Placement in Cloud Datacenters

Journal Pre-proof Multi-Objective Communication-Aware Optimization for Virtual Machine Placement in Cloud Datacenters Sara Farzai, Mirsaeid Hosseini Shirvani, Mohsen Rabbani

PII:

S2210-5379(19)30203-3

DOI:

https://doi.org/10.1016/j.suscom.2020.100374

Reference:

SUSCOM 100374

To appear in:

Sustainable Computing: Informatics and Systems

Received Date:

8 July 2019

Revised Date:

25 November 2019

Accepted Date:

20 January 2020

Please cite this article as: Farzai S, Shirvani MH, Rabbani M, Multi-Objective Communication-Aware Optimization for Virtual Machine Placement in Cloud Datacenters, Sustainable Computing: Informatics and Systems (2020), doi: https://doi.org/10.1016/j.suscom.2020.100374

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier.

Multi-Objective Communication-Aware Optimization for Virtual Machine Placement in Cloud Datacenters Sara Farzai1, Mirsaeid Hosseini Shirvani1⊠, Mohsen Rabbani2

1⊠- Department of Computer Engineering, Sari Branch, Islamic Azad University, Sari, IRAN.

of

2- Department of Mathematics, Sari Branch, Islamic Azad University, Sari, IRAN.

ro

Corresponding Author: [email protected] , [email protected]

-p

Abstract

Jo

ur na

lP

re

This paper formulates a new multi-objective virtual machine placement problem, which is a challenging task in cloud datacenters. In cloud environment, there are two stakeholders, namely, users and providers. Both sides try to take more benefit whereas a trade-off between conflicting benefits is crucial. From providers’ perspective, power consumption, and resource wastage are two objectives to be optimized whereas gaining high quality of service (QoS) is a critical point for users. The unpleasant issue that a user endures in the cloud environment is network delay; this is affected by common bandwidth linkage which is shared between different users’ applications; the reason for considering bandwidth usage optimization as the third objective function in users’ viewpoint. However, inefficient network bandwidth usage has drastic impact on overall performance even makes network links to get saturated and throttles communication-intensive applications. Therefore, VMs with high affinity and traffic dependency must be physically placed as close as possible so less traffic is sent on network layers. To figure out this combinatorial multiobjective problem, we extend a hybrid multi-objective genetic-based optimization solution. To evaluate this work, we conducted extensive scenarios with variable correlation coefficients between resources in requested VMs. The simulation results prove that our proposed hybrid metaheuristic algorithm outperforms against state-of-the-art ACO-based, well-known heuristic-based FFD algorithms, and random-based approach in terms of total power consumption, resource wastage, the total data transfer rate in network, and number of active servers in use. Also, the simulations in larger search space demonstrated proposed approach has high potential of scalability.

Keywords: Cloud computing, virtual machine placement, communication-awareness, multiobjective optimization, GA meta-heuristic

1. Introduction

Jo

ur na

lP

re

-p

ro

of

Nowadays, Cloud Computing (CC) has been universally accepted as an inseparable part of Information Technology (IT) [1]. Popular CC paradigm regarding to its various types of services, i.e., Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS), plays an undeniable role for hosting and delivering services over the Internet, which awards many advantages chief among, reliability, Quality-of-Service (QoS), robustness, elasticity, saving money, and reducing expenditures, to individuals and organizations [2, 3]. So, different ITbased industries are increasingly tending to CC on behalf of using its advantages to deploy their business. With pervasive usage of cloud computing which affects people’ lives in recent years, the scale of datacenters (DCs) is growing more than the past. A DC is a pool of configurable computing resources clustered together by utilizing communication networks to host applications [4]. A standard DC is modeled with a three-tier hierarchical network, includes core layer, aggregation layer and access layer, which comprises hundreds or even thousands of physical machines [5]. CC paradigm is underpinned by virtualization technology in which it virtualizes underlying computer resources between applications. Virtual machine placement (VMP) concept is to co-host number of VMs on a single PM so that the underlying PM to be efficiently utilized [6]. In this regard, high rate of power consumption in DCs is due to low resource utilization; for instance, a modern DC in US consumes 73 billion kWh, which is near to 2% of total energy consumed by the US [7]. So, DCs are going toward green computing to decrease energy cost, power consumption, and greenhouse gas emission rate. Also, imbalance usage of multi-dimensional computing resources leads to increasing power consumption and resource wastage, as well as increases service level agreement (SLA) violation rate. An intuitive solution is to apply minimum number of depreciation active servers for multi-dimensional requested VMs wherein the correlation coefficient between resources such as CPU capacity and memory needed are significantly variable. This type of problems are abstracted to famous multi-dimensional bin packing problem which is well-known as a NP-hard problem and can be solved by certain heuristics. Although energy saving is the first class concern class for energy-hungry DCs, there are several issues affecting overall performance which cannot be overlooked. In cloud environment, there are at least two stakeholders, namely, users and providers. Both sides try to take more benefit whereas a trade-off between conflicting benefits is crucial. From providers’ perspective, power consumption, and resource wastage, which lead to low utilization are very important issues whereas gaining high quality of service (QoS) is a critical point for users. In other words, energy saving and resource wastage reduction are two objectives to be optimized. Also, an unpleasant issue that a user endures in the cloud environment is network delay that is affected by common bandwidth linkage which is shared between different users’ applications, and is the reason for considering bandwidth usage optimization as the third objective function in users’ viewpoint. In addition to, as stated in [8] communication links, switching through PMs, and aggregating transferred data consume more than 30% of total energy used by DCs. Also [9] claims more than 70% of traffic in a DC is due to data transmission among VMs. However, several researches have been done to solve VMP in literature and obviate the problem, but less of them pay attention to VMs affinity, network topology, and communication traffic pattern between VMs beside reducing power consumption and resource wastage

Jo

ur na

lP

re

-p

ro

of

simultaneously. Gao et al. have formulated VMP as a bi-objective optimization problem which minimizes power consumption and resource wastage simultaneously [2]. Their work does not model network communication and traffic pattern between VMs. A topology-aware multiobjective VMP consolidation for datacenters has been presented by G. Cao [10]. He firstly introduced different DC topologies; then, he applied regression model to predict under-loaded and over-loaded servers by using global and local agent modules; afterwards, his work consolidates under-loaded servers and offloads overloaded one into suitable servers; it is done by migration techniques. Totally, his work is appropriate for dynamic workload, but in static projects which traffic patterns are determined in advance it is not suitable. Also, his work does not consider resource wastage and resource utilization to be optimized. Gopu and Venkataraman have proposed an optimal VM placement in distributed cloud environment [11]. They have formulated VMP into multi-objective optimization problem which power consumption, resource wastage, and propagation delay to be optimized. Although their work was promising in energy saving by consolidation technique, it does not consider communication traffic pattern between VMs and topology applied in underlying DCs. Consequently, it cannot improve network bandwidth usage. Study on researches in literature reveals that neglecting communication topology and VMs’ interactions may lead to network bandwidth to be saturated and communication-intensive applications to be throttled. Thus, the aim is to co-host high-affinity interactive VMs on the same physical machine or at least to place them physically as close as possible; in this way, the network traffic rate and end-to-end delay will be reduced. With this goal, traffic-dependent VMs are tried to be hosted by a same PM, and if it is not possible, preferably they are tried to be placed under zone of access layer, otherwise under aggregation layer, and under core layer in the worst case respectively. In Fig.1, VMs with the same color are interactive and traffic-dependent. As such, the end-to-end delay and traffic rate on network are significantly reduced in optimal VMP (Fig.1.b).

a. Naïve VMP

Figure 1. Communication-aware VMP vs naïve VMP

b. Communication-aware VMP

As stated earlier, a multi-partite trade-off is needed for a VMP problem form minimizing among energy consumption, resource wastage, and traffic stream on DC’s network. Therefore, this paper formulates a new mathematical multi-objective VMP optimization model with the following contributions: - Minimize power consumption, - Minimize resource wastage, - Minimize bandwidth usage by reducing traffic stream among network elements.

re

-p

ro

of

After problem statement, we solve the proposed this combinatorial NP-Hard 3-objective VMP problem by using a genetic-based meta-heuristic algorithm to overcome the time-complicated nature of the aforementioned problem from NP-hard class, and to achieve a set of promising optimal solutions within a reasonable time. The result of extensive simulations proves the effectiveness of proposed algorithm against exiting approaches. The rest of this paper is organized as follows. Section 2 has an overview on related works. A brief review of literature is presented in section 3. In section 4 problem statement is defined, followed by an illustrative example in section 5. Proposed mathematical multi-objective optimization model of this paper is depicted in section 6. Simulations results, evaluations, and time complexity are discussed in section 7., and section Finally, section 8 is dedicated to conclusion and future works directions.

lP

2. Related works

Jo

ur na

Recently, due to increasing trend in energy costs and carbon footprint, research motivations increased toward energy efficiency mechanism for datacenter management. Energy consumption as an inevitable challenge in electricity hungry DCs, widely emerged in researches with the goal of reducing reduce energy costs, namely, (hardware based techniques such as Dynamic Voltage Frequency Scaling (DVFS) [12-18], and software based techniques such as VMP (which is overviewed in following sub-sections) either face with ecological challenges and greenness issues (i.e., global warming and green-house gas emission due to fossil fuels usage) [19-26]. In addition to accepting power management as a great concern in literature, cloud optimizations studies and their solution types related to current paper topic, would be categorized as follows.

2.1. Solution types (Exact / Heuristic / Meta-heuristic)

Different types of approaches to solve VMP optimization problem have been presented in various studies that could be classified in three main categories: exact, heuristic, and meta-heuristic approaches,; each of which them has its own specifications. Exact approaches such as Linear Programming (LP) [27-30], and Constraint Programming (CP) [31, 32] provide precise solutions for VMP problem. Owing to the intractable nature of the problem, the execution time exponentially increases as the input size grows such as in real world cases. In addition to, these approaches are not applicable in multi-objective optimization domain at a same time. Several studies have focused

-p

ro

of

on heuristic approaches, among them are various Fit-works such as First Fit (FF) [33], Best Fit (BF) and Worst Fit (WF) [34], First Fit Decreasing (FFD) [35, 36], Best Fit Decreasing (BFD) [36], Modified FF and BF [37], Modified BFD [38], and pack-works such as Permutation Pack and Choose Pack [39], in which all of them employ greedy algorithm to produce the solution. Heuristics in contrast with exact approaches could provide an approximation solution near to optimal, in a reasonable time frame [40]. However, they are extremely centralized and hard to be distributed [41], as well the algorithm rapidly falls in local optimal trap [42]. Meta-heuristics has been widely used to solve VMP problems. They are a family of biological evolutionary algorithms that try to combine heuristics in higher level frameworks aimed at efficiently exploring search space for the feasible and optimal solutions [42]. Most promising meta-heuristics include, but not restricted to are Genetic Algorithm (GA) [43], Ant Colony Optimization (ACO) [44], Particle Swarm Optimization (PSO) [45, 46], and Simulated Annealing (SA) [47]. Meta-heuristics, compared to exact and heuristic approaches, are capable of solving multi-objective intractable problems in an acceptable time. However, they produce approximation solutions using randomlygenerated data, and so they do not guarantee that optimal solution will be achieved finally [42, 48].

2.2. Network- and communication- aware VMP optimization approaches

lP

re

Proposed solutions of VMP in literature can be improved after study on network connections and related elements because VMs’ mutual communication and traffic pattern have drastic impact on total energy consumption and overall performance in DCs. So, this section reviews VMPs in regard to optimization of bandwidth usage and reduction in network elements involved for users’ applications.

Jo

ur na

In [48], authors presented a 3-objective VMP optimization and consolidation model to minimize energy consumption, resource wastage, and communication energy cost, solved by ACO metaheuristic algorithm. In this work, firstly an initial mapping of VMs onto PMs has been performed followed by a complementary consolidation algorithm based on migration, build on the results of placement. However, bandwidth usage and impact of traffic rate between high-connection VMs have not been considered. Authors of [49] presented greedy and heuristic-based algorithms for VMP problem in a heterogeneous DC network with the objective of maximizing satisfaction metric, which is defined as the suitability of a PM for each of VMs assigned to it. In this paper, tendency of VMs to initiate traffic stream to be received or processed on certain network elements, both computation and connection nodes, is the main idea. Aforementioned nodes are called sink and a VM in hand attempts to acquire it. In this single objective study, inter-VM communications have been assumed negligible and omitted. Authors of [50] presented an approach named MinDistVMDataPlacement that performs VMP and data placement along with each other using ACO algorithm. The main object of this study is to reduce cross network traffic and bandwidth usage in the process of placing required number of VMs; it is done by placing data on physically closer PMs. The results show that presented algorithm by selecting PMs in close proximities had a better VM allocation in comparison with other schemes. However, this work focused on a single objective;, and nevertheless, power and resource management have not been met in that article. In [51], authors presented an approach, called VMPlanner, for network power reduction in the

Jo

ur na

lP

re

-p

ro

of

virtualization-based data centers. The basic idea was to optimize both virtual machine placement and traffic flow routing so as to turn off as many unneeded network elements as possible for power saving. VMPlanner performs in three steps: Traffic-aware VM grouping, Distance-aware VMgroup to server-rack mapping, and Power-aware inter-VM traffic flow routing. Results showed more capabilities to save network power than baseline ElasticTree [52]. In this work, resource requirements were not taken into account. A Traffic-aware Virtual Machine Placement Problem (TVMPP) has been proposed to improve the network scalability by using traffic-aware VMP [53]. In this work, placement is optimized by assigning VMs with large mutual bandwidth usage on host machines in close proximity. The proposed approach is conducted by two-tier approximation algorithm for large scale problems. Simulation results of this paper showed that the benefit of TVMPP significantly depends on the network architecture. This benefit is greater for a multi-layer architecture, such as BCube [54], and is the minimum for an architecture that employs network load balancing techniques, such as in VL2 [9]. Authors of [55] tried to improve network performance in terms of link utilization and communication traffic. Their work follows two objectives: (i) minimize the maximum link utilization (MLU) by placing VMs in a balanced way, (ii) reduce network traffic by placing inter-communicating VMs with higher traffic in the same host or at least the same network switch, implemented using a hybrid ACO combined with a 2-opt local search. To do so, they defined communication cost between PMs as the number of switches the traffic traverses until the destination. Simulation showed that MLU has been decreased by 20%, and the number of hot links is decreased by 37%. A VMP and migration policy have been presented to bring VMs closer to required data, usually located on storage area attached networks (SANs) so that all VMs can uniformly access to required Disk images [56]. The goal is to minimize data access time by placing VMs on hosts having better network condition, situation between the physical machines and the data storage, which is dynamically determined to place and migrate VMs, regarding SLA requirements. Simulation results showed that the approach is effective in optimizing the data transfer between the virtual VMs in DCs and data. A special VMP algorithm was suggested for hybrid data center networks [57]. The authors firstly designed a threshold-based, wireless link-aware flow placement algorithm. Secondly, a set of virtual machine placement algorithms suggested under the flow placement algorithm to enhance traffic locality. Simulation results in hybrid data center networks with 60 GHz wireless links showed that combination of the proposed algorithms achieved an acceptable performance in flow completion time, aggregate throughput, and computational complexity. Nevertheless, their work has not taken energy saving and resource dissipation into consideration.

2.3. Other VMP optimization studies

Here, we open new room to classify other approaches toward VMP. In [2], authors proposed a multi-objective ant colony system algorithm, called VMPACS for the virtual machine placement problem to efficiently obtain a set of non-dominated solutions, the Pareto set that simultaneously minimizes total resource wastage and power consumption. Solution performance is compared to multi-objective grouping genetic algorithm (MGGA) [58], FFD [44], and single-objective ACO (SACO) [59]. The results showed that the proposed algorithm is more

Jo

ur na

lP

re

-p

ro

of

promising than others. Authors of [60] proposed an algorithm, called Multiplayer Random Evolutionary Game Algorithm (MREGA) for VMP. It exploits the evolutionary game theory to solve the dynamic VMs placement problem, with the objective to optimize energy consumption. Simulation results compared to FF, BFI, BFD, greedy and Load Balancing (LB) approaches, showed that MERGA totally outperformed against others in term of energy consumption reduction. A novel algorithm known as Predictive Anti-Correlated Placement Algorithm (PACPA) has been proposed, which considers both CPU and bandwidth resource consumption [61]. Empirical results showed PACPA reduces energy consumption by 18% while also reducing service violations by over 34% compared to FF, BF, and Power Aware Best Fit Decreasing (PABFD) [38]. In [62], a cloud brokering approach has been proposed that optimizes placement of virtual infrastructures across multiple clouds and also abstracts the deployment and management of infrastructure components in these clouds. Experimental results confirmed that multi-cloud deployment provides better performance and lower costs compared to the usage of a single cloud only. A Clustering based virtual machines placement algorithm, called Clustering Based Min Diameter (CBMinDia), was proposed in [63]. CBMinDia keeps the 2-approximation property developed by taking full use of the topology and the density of cloud network. For distributed cloud computing, the placement of VMs of one big task or of one organization focuses on minimizing the distances and bandwidths between DCs. The purpose is to minimize the maximum inter-DC distance. At first, a data center cluster should be found to accommodate the requested VMs. Then, the requested VMs should be partitioned to the DC cluster, so that the expensive inter-DC bandwidth is saved and the availability is improved. To do so, with the introduction of a half communication model, a novel heuristic algorithm which further cuts down the used bandwidths is presented to partition VMs. The simulation showed the proposed algorithm is appropriate for very large scale problems. Moreover, its time complexity is reduced to 𝑂(𝑛2 ) by a factor of 𝑂(log 𝑛) and it runs 3 times faster than the existing method. Authors of [23] proposed a VMP algorithm to increase the environmental sustainability by taking into account distributed data centers with different carbon footprint rates and power usage effectiveness. This paper’s outcome illustrated that the proposed algorithm reduced the CO2 emission and power consumption, while maintaining the level of QoS compared to other related algorithms. A Modified Discrete Particle Swarm Optimization (MDPSO) was proposed for VMs placement which is based on the characteristic of particle swarm optimization [64]. Also, the selection algorithm was propounded for optimizing the current allocation based on memory utilization, bandwidth utilization, and size of the VMs. The outcome of implementation indicates that combination of proposed allocation and selection algorithms leads to reduction in energy consumption in a cloud data center up to 32%, compared to [38, 65-67]. Authors of [68] proposed a task to VM and VM to PM algorithm, called Energy-aware Task-based Virtual Machine Consolidation (ETVMC). The proposed algorithm reduces the energy consumption by depreciating the number of active PMs, while also minimizes the makespan and task rejection rate. Simulation results demonstrated the effectiveness of the ETVMC over FCFS, Round-Robin, and EERACC proposed in [69]. A two-level control system is proposed in [58] to manage the mappings of workloads to VMs and VMs to physical resources. It is formulated as a multi-objective optimization problem which minimizes total resource wastage, power consumption, and thermal dissipation costs using an improved GGA with fuzzy multi-objective evaluation for searching the large solution space and combining possibly conflicting objectives. A simulation-based evaluation

ro

of

demonstrates good performance, scalability, and robustness of proposed approach when compared with well-known FFD, BFD, SGGA, and MGGA algorithms. In [70], a multi-objective formulation of VMP problem considering SLA was proposed and was solved by a memetic algorithm. Simulations proved the correctness of the proposed memetic algorithm by calculating the complete Pareto front (100%) and its scalability considering different experimental scenarios. A multi-objective evolutionary algorithm (MOEA/D) is proposed in [11] to find non-dominated solutions which minimizes resource wastage, power consumption, and network propagation delay. The statistical analysis of the simulation results proves that MOEA/D outperforms against the related existing algorithms in distributed cloud VMP. A hybrid genetic algorithm using BFD has been presented to deal with the infeasible solutions due to the bin-used representation [71]. It conducts several operators to amend infeasible chromosomes; it also improves quality of obtained solutions. The outcome for solving VMP by proposed algorithm is promising since the algorithm has optimized the bin-packing problem by applying hybrid approach, so it could be deduced that it is able to minimize the energy dissipation by DCs.

re

-p

As mentioned earlier, energy management has been emerged as the first-class concern for datacenter owners, regarding to expenditure and green computing goals. However, the impact of network topology and VMs’ communications on overall performance has not been discussed in reviewed studies of this sub-section.

2.4.Discussion

Jo

ur na

lP

There are numerous communication-intensive applications running on cloud DCs that are as great as not possible to be placed on a single PM, such as MapReduce-based projects. These types of applications are generally split in form of many high-connectivity VMs and data chunks, placed on different PMs in cloud DCs to be processed in parallel. Inter-VM data congestion in these applications heavily involves switches and communication links in DC’s network. This dilemma can significantly increase energy consumption and highly saturates network bandwidth, so it may lead to performance degradation for other applications and also it may seriously lead to DC’s overall performance reduction. Investigation on recent studies reveals that many researches focused on energy and resource management, but less concentrated on VMs’ mutual data transfer and traffic stream over communication links. Therefore, in this paper we propose a 3-objective optimization model to optimize energy consumption, resource allocation, and bandwidth usage simultaneously, in which high-connectivity VMs are tried to be co-hosted or at least to be placed in physical vicinity as close as possible. According to the best of our knowledge, there are no related researches that propose a multi-objective optimization model such as our interest in this work, especially from bandwidth usage calculation perspective.

3. Literature Review 3.1. VMP

Jo

ur na

lP

re

-p

ro

of

Assigning user applications as VMs to PMs is called VMP (Fig.2). The strategy of solving VMP problem directly affects cost, power, and resource usage in DCs. VMP belongs to NP-hard time complexity class, because several conflicting objectives should be met in this problem. VMP has been formulated as a multi-dimensional bin packing problem in related studies, in which different VMs as objects should be placed in a number of PMs as bins with variable sizes and prices. The goal is to pack VMs in the minimum number of available PMs [6]. The VMP problem can be generally classified into two categories: static VMP and dynamic VMP. Static VMP is usually performed when a system starts up, or when a number of new VMs are created and they should be placed to available PMs, while it is not needed that any of existing placed VMs to be replaced from their current host. Therefore, the static VMP is normally used for initial stages. Static VMP techniques are often done in off-line manner, in which all VM requests are known in advance. In dynamic VMP, some unstable factors such as states of PMs and new VMs arrivals, cause adjusting the mapping process between VMs and PMs, often reallocation is required. Dynamic VMP techniques are performed on-line, that allows VM requests arrive dynamically and be hold in the system until be allocated and processed [48, 60]. In this paper, we focus on static VMP, where all requested VMs and related characteristics are determined in advance.

Figure 2. VMP scheme in DCs

3.2. Datacenter network topology

a. Tree

c. VL2

lP

b. Fat-Tree

re

-p

ro

of

In DCs, the concept of topology may be used in a dual meaning: (i) it refers to applications communications, which depicts a pattern of communications between VMs that may be mined from system log or history. In this paper, this type of topology is named communication, (ii) it also may be used to point to the network elements architecture in a DC. By considering the number of switches, links, and their arrangement, physical distance between PMs, and consequently between different VMs located on these PMs can be measured. Three famous three-tier DC network topologies are shown in Fig.3. Core switches in the highest layer (core layer) deliver network traffic between switches of the lower levels and also deliver traffic between DC network and the Internet. Aggregation switches that are in the intermediate layer (aggregation layer) connect the core switch with Top of Rack (ToR) switches, also called access switches, in the lowest layer (access layer). Each ToR switch forwards traffic for its underlied PMs in the rack. PMs have no local disks, instead, share SAN storage devices to manage VMs [10].

ur na

Figure 3. Three standard DCs’ network topologies [10]

Jo

The simplest way to scale up a topology is the manner that happens in Fat-Tree. The Fat-Tree topology uses identical 4-port switches in all layers which totally supports 16 PMs. To scale up the Fat-Tree network fabric, 4-port switches can be substituted with 𝑝-port identical switches (whereas 𝑝 = 8, 16, …), to form a Portland topology [72]; Portland topology is built based on the concept of pod: a unit of switches in access and aggregation layers that makes a complete bipartite graph, and is connected with all the core switches. Generally, in Fat-Tree-based topologies, if there are 𝑝-port switches, then there are 𝑝 pods each of which contains 𝑝⁄2 access and 𝑝⁄2 aggregation switches. Each pod is connected with 𝑝2 ⁄4 core switches and 𝑝2 ⁄4 PMs. Therefore, there are totally 5𝑝2 ⁄4 switches and 𝑝3 ⁄4 PMs [53]. Fig.4 shows such a network with 𝑝 = 8 in comparison with Fat-Tree (𝑝 = 4).

ro

of

a. Fat-Tree (𝑝 = 4)

-p

b. PortLand (𝑝 = 8)

3.3. Multi-objective optimization

re

Figure 4. Pods in Fat-Tree and PortLand topologies

ur na

lP

There are several real world problems need to be optimized in order to satisfy multiple objectives simultaneously, sometimes to make a trade-off between two or more conflicting objectives. Assume vector 𝑋⃗ = [𝑥1 , 𝑥2 , … , 𝑥𝑛 ] is a set of n feasible decision variables, and there are M objective functions to be minimized (optimized). Mathematically, a multi-objective optimization problem would be formulated as equation (1): min 𝑓⃗(𝑋⃗) = [𝑓1 (𝑋⃗), 𝑓2 (𝑋⃗), … , 𝑓𝑀 (𝑋⃗)] 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜: 𝑔𝑖 (𝑋⃗) ≤ 0, 𝑖 = 1, … , 𝐼 ℎ𝑗 (𝑋⃗) = 0,

(1)

𝑗 = 1, … , 𝐽

Jo

Where 𝑔𝑖 (𝑋⃗) and ℎ𝑗 (𝑋⃗) are problem constraints. In a multi objective optimization problem, the ⃗⃗⃗⃗⃗∗ for which 𝑓⃗(𝑋 ⃗⃗⃗⃗⃗∗ ) is minimized. In multi-objective optimization, there are goal is to find vector 𝑋 more than one objective so that objectives may have conflicting. So, generally more than one solution is achieved which make sub-optimal solutions. Whereas whereas a single-objective optimization results in a single optimal solution. To evaluate multiple sub-optimal solutions, ⃗⃗ = [𝑏1 , 𝑏2 , … , 𝑏𝑀 ], and 𝐶⃗ = [𝑐1 , 𝑐2 , … , 𝑐𝑀 ] be two Mdominance concepts should be defined. Let 𝐵 ⃗⃗ dominates 𝐶⃗, or 𝐶⃗ is dominated by 𝐵 ⃗⃗ (denoted by 𝐵 ⃗⃗ ≤ 𝐶⃗ for dimensional vectors. It is said: 𝐵 minimizing problem), iff: ∀𝑖 ∈ {1, … , 𝑀}, 𝑏𝑖 ≤ 𝑐𝑖 𝑎𝑛𝑑 𝑓𝑜𝑟 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑖 ∈ {1, … , 𝑀}, 𝑏𝑖 < 𝑐𝑖 .

(2)

For multi-objective functions, a set of non-dominated solutions, called Pareto optimal solutions or Pareto Set (PS), is formed, in which there exists no solution dominated by any other solutions. In the other word we have [73]: 𝑃𝑆 = {𝑆⃗1 , 𝑆⃗2 , … , 𝑆⃗𝑀 }, ∄𝑖, 𝑗 ∈ {1, … , 𝑀}, 𝑆⃗𝑖 ≤ 𝑆⃗𝑗 , (𝑖 ≠ 𝑗)

(3)

4. Problem statement

ro

of

For constructing the multi-objective static VMP model for a DC with n requested VMs and m existing PMs in this paper, firstly the mathematical models corresponding to each objective, namely, resource wastage, power consumption, and bandwidth usage along with related constraints have been are formulated in 3 following sub-sections. In the last sub-section, the complemented multi-objective optimization model is proposed. Also, Table 1 depicts a nomenclature of notations and used parameters in current and forthcoming sections. Table 1. Nomenclature, notation, and parameter description

Description

ToR 𝑝

Top of Rack Number of switch ports Storage Attached Network Pareto Set i th Virtual Machine j th Physical Machine Million Instruction Per Second Cpu utilization for PMj Memory utilization of PMj Threshold for CPU capacity Threshold for Memory capacity CPU capacity request for VMi Memeory capacity request for VMi Residual CPU capacity for VMj Residual Memory capacity for VMj Power consumption PMj in full load state Power consumption PMj in idle mode Normalized resource wastage of PMj Number of communication links between PMs Total data transfer Total resource wastage Traffic pattern Matrix Total Power Consumption Population size Spacing parameter

MIPS 𝐶𝑝𝑢

𝑈𝑗

ur na

𝑈𝑗𝑀𝑒𝑚 CpuThr MemThr CpuReqi MemReqi

re

VMi PMj

lP

SAN PS

𝐶𝑝𝑢

𝑅𝑗

𝑅𝑗𝑀𝑒𝑚 𝑃𝑗𝐹𝑢𝑙𝑙

Jo

𝑃𝑗𝐼𝑑𝑙𝑒 Wj

hops TDT TRW TPM TPC PopSize SP ONVG

-p

Parameter

Overall non-dominated vector generation Iteration

Number of iteration; it is user defined parameter

n

Number of requested VMs

m

Number of Physical machine in Datacenter

k

Number of items in Pareto Set (number of non-dominated solutions)

4.1. Server resource wastage model

ro

of

Utilization of resources in a DC is highly influenced by various VMP strategies. When a VM with a certain requirements of CPU and memory is hosted by a PM, it is desired that the probably residual portions of PM’s available resources to be assigned to other existing VMs,; so, that the wasted portions to must be minimized. Utilization models of CPU and memory of PMj are represented in equations (4) and (5) respectively: 𝑛

= ∑ 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 × 𝑦𝑖𝑗

-p

𝐶𝑝𝑢 𝑈𝑗

𝑖=1 𝑛

(5)

re

𝑈𝑗𝑀𝑒𝑚 = ∑ 𝑅𝑒𝑞𝑀𝑒𝑚𝑖 × 𝑦𝑖𝑗 𝑖=1

(4)

𝐶𝑝𝑢

𝑅𝑗

ur na

lP

Where 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 and 𝑅𝑒𝑞𝑀𝑒𝑚𝑖 are the CPU and memory required by VMi respectively, and 𝑦𝑖𝑗 ∈ {0,1} is a binary decision variable which equals to 1 if VMi has been placed on PMj, otherwise equals to 0. Though it seems 100% of utilization is desirable, achieving to such level may lead to falling in an operational swamp. Hence, it is not so practical. So, usually a threshold level is defined for these elements. Therefore with considering thresholds, the remained capacity amounts of CPU and memory of PMj are formulated as equations (6) and (7) show respectively: 𝐶𝑝𝑢

= 𝐶𝑝𝑢𝑇ℎ𝑟 − 𝑈𝑗

(6)

𝑅𝑗𝑀𝑒𝑚 = 𝑀𝑒𝑚𝑇ℎ𝑟 − 𝑈𝑗𝑀𝑒𝑚

(7)

Jo

Where 𝐶𝑝𝑢𝑇ℎ𝑟 and 𝑀𝑒𝑚𝑇ℎ𝑟 denote CPU and memory thresholds respectively. Accordingly, two-dimensional resource wastage for PMj can be calculated by equation (8) [2]: 𝐶𝑝𝑢

𝑊𝑗 =

|𝑅𝑗

− 𝑅𝑗𝑀𝑒𝑚 | + 𝜀

𝐶𝑝𝑢

𝑈𝑗

(8)

+ 𝑈𝑗𝑀𝑒𝑚 𝐶𝑝𝑢

Where 𝑊𝑗 is the resource wastage of PMj, 𝑈𝑗

and 𝑈𝑗𝑀𝑒𝑚 are the ratio of used resource to total 𝐶𝑝𝑢

available resource, representing the normalized CPU and memory usage. 𝑅𝑗 and 𝑅𝑗𝑀𝑒𝑚 denote the normalized remaining capacity of CPU and memory resources, and 𝜀 is a very small positive real number, which is considered 0.0001 in this paper. Note that, we take both resources CPU and

Memory having the same importance in the system for applications. Total resource wastage (TRW) in a DC with m PMs is calculated by equation (9): 𝑚

𝑇𝑅𝑊 = ∑ ( 𝑗=1

𝐶𝑝𝑢

|𝑅𝑗

− 𝑅𝑗𝑀𝑒𝑚 | + 𝜀

𝐶𝑝𝑢

𝑈𝑗

+ 𝑈𝑗𝑀𝑒𝑚

) . 𝑥𝑗

(9)

ur na

lP

re

-p

ro

of

Where 𝑥𝑗 ∈ {0,1} is a binary decision variable which equals to 1 if the PMj is on, and otherwise equals to 0. Note that, variable TRW is a multiplicative of a full server utilization. To illustrate the concept, a two-dimensional resource wastage scheme (, as a two-dimensional bin packing sample, ) is shown in Fig.5. We do not consider disk as a PM’s resource, because DCs take benefit of SAN storage technology to data store and retrieve data. So, the local disks in PMs are disregarded in calculation.

Figure 5. Two-dimensional resource wastage scheme

4.2. Datacenter power consumption model

Jo

Obviously, we know all the physical elements in a DC contain electrical and electronics devices that each of which consumes power electric. But the main amount of overall power consumption in a DC is resulted from running PMs [60]. Other involved facilities such as cooling equipment, switches, routers, and communication links which have less portion in this metric. Therefore, we focus on PMs’ power consumption in two states: full, (when it is assigned to VMs which occupy all PM’s CPU capacity), and idle, (when it is not assigned to any VM), and other devices’ usage is omitted. Then, we can calculate PM’s current power consumption based on its CPU utilization. Note that, there exist a linear relationship between the power consumption and CPU utilization [74]. So, the power consumption of the PMj can be formulated by equation (10) [2]:

(𝑃 𝑃𝑗 = { 𝑗 0

𝑓𝑢𝑙𝑙

𝐶𝑝𝑢

− 𝑃𝑗𝑖𝑑𝑙𝑒 ) × 𝑈𝑗

+ 𝑃𝑗𝑖𝑑𝑙𝑒

𝐶𝑝𝑢

𝑖𝑓 𝑈𝑗 >0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

(10)

𝑓𝑢𝑙𝑙

In equation (10), which 𝑃𝑗 and 𝑃𝑗𝑖𝑑𝑙𝑒 denote the average values of power consumption when the PMj is fully utilized and idle, respectively. Total power consumption (TPC) in a DC with m PMs is calculated by equation (11): 𝑚

𝑇𝑃𝐶 = ∑ ((𝑃𝑗

𝑓𝑢𝑙𝑙

𝐶𝑝𝑢

− 𝑃𝑗𝑖𝑑𝑙𝑒 ) × 𝑈𝑗

+ 𝑃𝑗𝑖𝑑𝑙𝑒 ) . 𝑥𝑗

(11)

𝑗=1

ro

of

Where 𝑥𝑗 ∈ {0,1} is a binary decision variable which equals to 1, if the PMj is on, and otherwise equals to 0.

4.3. Datacenter bandwidth usage model

ur na

lP

re

-p

In this paper, we focus on Fat-Tree topology (Fig.3.b). However, without Without loss of generality, the proposed bandwidth usage model can be extended to other topologies. Fat-Tree utilizes identical 4-port switch devices, and homogeneous communication bandwidth which is mutually exploited between each layer (core, aggregation, and access layers) [10]. In VMP, maybe each VM to be is placed on each arbitrary PM.; So so, the physical distance between PMs, which are hosting traffic-dependent VMs, on a network can be measured through counting the number of hops,. Namely The term hops is number of communication links that should be passed for routing data from source PM to destination PM [53]. On the other hand, different VMs may have data connectivity or not. Consider VMi and VMj which are placed on PMk and PMℓ respectively, have data connectivity with 𝑑𝑣 average rate. Definitely, the ideal state is that PMk and PMℓ to be the same, i.e., 𝑘 = ℓ. Otherwise, if PMk and PMℓ are connected via: - Access switch, 2 hops are needed for data transfer, - Aggregate switch, 4 hops are needed for data transfer, - Core switch, 6 hops are needed for data transfer.

Jo

Therefore, DC network can be segmented in 4 virtual zones as it has been defined in Table 2 based on the position of PMs, containing dependent VMs, against each other. Fig.6 shows the area of each zone. The set of all the zones is defined via equation (12): 3

𝑍 = ⋃ 𝑍𝑞 𝑞=0

(12)

Table 2. Virtual zones of DC network

Zone Position of PMk & PMℓ same (𝑘 = ℓ) under access switch under aggregate switch under core switch

Data rate on links 0 × 𝑑𝑣 = 0 2 × 𝑑𝑣 4 × 𝑑𝑣 6 × 𝑑𝑣

0 2 4 6

lP

re

-p

ro

of

Z0 Z1 Z2 Z3

Hops

Figure 6. Virtual zones areas in a Fat-Tree DC network

Jo

ur na

For each relation from VMi placed on PMk to VMj placed on PMℓ, a connection 𝐶𝑘,ℓ ∈ 𝑍𝑞 is defined. The traffic volume transferred between VMi and VMj, and generally for each arbitrary pair of VMs can be extracted from Traffic Pattern Matrix (TPM). A sample TPM matrix is shown in equation (13) in which all the indexes are in term of Mega Byte per second (MBps). The TPM can be achieved in many ways, such as investigating DC’s profile or discovering from VMs behaviors using data mining techniques, that both are not in the scope of this paper. As we assumed static VMP in this paper, we are given a static TPM in advance in which it has been derived from data history and has been determined as VMs behaviors. 𝑉𝑀1 𝑉𝑀2 𝑇𝑃𝑀 = ⋮ 𝑉𝑀𝑛

𝑉𝑀1 000 [125 ⋮ 400

𝑉𝑀2 250 000 ⋮ 112

⋯ ⋯ ⋯ ⋱ ⋯

𝑉𝑀𝑛 320 260] ⋮ 000

(13)

Regarding to earlier definitions, data transfer (DT) between VMi placed on PMk and VMj placed on PMℓ can be represented via equation (14):

𝐷𝑇(𝐶𝑘,ℓ ) = ℎ𝑜𝑝(𝐶𝑘,ℓ ) × 𝑇𝑃𝑀(𝑉𝑀𝑖 , 𝑉𝑀𝑗 )

(14)

Where ℎ𝑜𝑝(𝐶𝑘,ℓ ) is calculated from virtual zone containing PMk and PMℓ according to algorithm 1 depicted in Fig.7.

ur na

lP

re

-p

ro

of

Algorithm 1. Hop Counter Input: p, k, ℓ: Number of ports, Indices of PMs hosting VMi, and VMj respectively. Output: Count of hops from PMk to PMℓ. Begin 1: if p = = 4 then 2: if ((mod (k , 2) = = 1) and (ℓ = = k + 1)) or 3: ((mod (ℓ , 2) = = 1) and (k = = ℓ + 1)) then hop = 2; 4: else if (abs (k - ℓ) < 4) and (floor ((k - 0.5) / 4) = = floor ((ℓ - 0.5) / 4)) then hop = 4; 5: else hop = 6; 6: end if 7: end if % ------------------------------------------------------------------------------------------------------8: if p = = 8 then 9: if ((abs (k - ℓ) < 4) and (floor ((k - 0.5) / 4) = = floor ((ℓ - 0.5) / 4))) then hop = 2; 10: else if (floor ((k - 0.5) / 16 ) = = floor ((ℓ - 0.5) / 16)) then hop =4; 11: else hop=6; 12: end if 13: end if 14: return hop; End {Hop Counter} Figure 7. Algorithm 1. Hop Counter

Regarding to Based on Fig.6, for example, consider VM1 placed on PM2 has data transfer to VM2 placed on PM5. and take in this Whereas in Fat-tree, PM2 and PM5 are connected in zone Z4 Z3 (under core switch),; therefore, here we have 6 hops. So, ℎ𝑜𝑝(𝐶2,5 ) = 6. Therefore Consequently, the data transferred between VM1 and VM2 equals:

Jo

𝐷𝑇(𝐶2,5 ) = ℎ𝑜𝑝(𝐶2,5 ) × 𝑇𝑃𝑀(𝑉𝑀1 , 𝑉𝑀2 ) = 6 × 250 = 1500 𝑀𝐵𝑝𝑠 The total data transfer (TDT) in a DC with n VMs and m PMs is calculated via equation (15): 𝑛

𝑛

𝑇𝐷𝑇 = ∑ ∑ ℎ𝑜𝑝(𝐶𝑘,ℓ ) × 𝑇𝑃𝑀(𝑉𝑀𝑖 , 𝑉𝑀𝑗 ) ,

𝑘, ℓ ∈ {1, … , 𝑚}

𝑖=1 𝑗=1

4.4.

Multi-objective communication-aware optimization model

(15)

In this section, we present our main optimization model along with related constraints. According to mathematical models denoted in sub-sections 4.1 – 4.3, our multi-objective optimization model is defined as in equations (16-24): 𝑚

𝑚𝑖𝑛 ∑ 𝑊𝑗 × 𝑥𝑗 =

(16)

𝑗=1 𝑚

𝑗=1

|(𝐶𝑝𝑢𝑇ℎ𝑟 − ∑𝑛𝑖=1 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 × 𝑦𝑖𝑗 ) − (𝑀𝑒𝑚𝑇ℎ𝑟 − ∑𝑛𝑖=1 𝑅𝑒𝑞𝑀𝑒𝑚𝑖 × 𝑦𝑖𝑗 )| + 𝜀 × 𝑥𝑗 ∑𝑛𝑖=1 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 × 𝑦𝑖𝑗 + ∑𝑛𝑖=1 𝑅𝑒𝑞𝑀𝑒𝑚𝑖 × 𝑦𝑖𝑗

𝑚

𝑚

min ∑ 𝑃𝑗 × 𝑥𝑗 = 𝑗=1 𝑛 𝑛

𝑛

𝑓𝑢𝑙𝑙 ∑ ((𝑃𝑗 𝑗=1

−

𝑃𝑗𝑖𝑑𝑙𝑒 )

× ∑ 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 × 𝑦𝑖𝑗 + 𝑃𝑗𝑖𝑑𝑙𝑒 ) × 𝑥𝑗 𝑖=1

𝑚𝑖𝑛 ∑ ∑ ℎ𝑜𝑝(𝐶𝑘,ℓ ) × 𝑇𝑃𝑀(𝑉𝑀𝑖 , 𝑉𝑀𝑗 )

(18)

ro

𝑖=1 𝑗=1

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜:

𝑚

∑ ∑ 𝑥𝑖𝑗 = 1 , 𝑤ℎ𝑒𝑟𝑒 𝑥𝑖𝑗 ∈ {0,1}

re

𝑖=1 𝑗=1 𝑛 𝑚

-p

𝑥𝑗 ∈ {0,1} 𝑘, ℓ ∈ {1, … , 𝑚} 𝑛

∑ ∑ 𝑦𝑖𝑗 = 1 , 𝑤ℎ𝑒𝑟𝑒 𝑦𝑖𝑗 ∈ {0,1}

lP

𝑖=1 𝑗=1 𝑛 𝑚

∑ ∑ 𝑅𝑒𝑞𝐶𝑝𝑢𝑖 × 𝑦𝑖𝑗 ≤ 𝐶𝑝𝑢𝑇ℎ𝑟 × 𝑥𝑗 𝑖=1 𝑗=1 𝑛 𝑚

∑ ∑ 𝑅𝑒𝑞𝑀𝑒𝑚𝑖 × 𝑦𝑖𝑗 ≤ 𝑀𝑒𝑚𝑇ℎ𝑟 × 𝑥𝑗 𝑖=1 𝑗=1

(19) (20) (21) (22) (23)

(24)

ur na

{

(17)

of

∑

To solve aforementioned combinatorial multi-objective optimization problem, which needs to optimize triple equally important optimization functions simultaneously, we extend a hybrid metaheuristic algorithm.

Jo

5. An illustrative example

In this section, we exemplify how our smart algorithm differs from a naïve approach, which uses much more resources, dissipate more energy, and transfer more data on network links. In DCs, the placement of VMs must be done swiftly to ensure response speed [75]. A quick approach which only focuses on users’ SLA and neglects underlying objectives are known as a naïve approach. Refer to Fig.1, take requested VMs and underlying PMs are as Fig.1 shows. Systems’ specifications, VMs requirements and the results before and after VMP optimization are depicted in Fig.1.a and Fig.1.b respectively. Also, Table 3 elaborates the numerical data relevant to triple

objectives. It proves the effectiveness of proposed algorithm. Note that, all calculations have been attained via equations (9), (11), and (15).

Table 3. Illustrative example of Fig.1 VMs Requirements

Fig.1.a Naïve VMP

Fig.1.b Optimal VMP Power 295 0

Wastage 0.0410 0

290

0.140

250

0.1110

835

0.1660

CPU 500 MIPS 1800 MIPS

Mem 2 GB 7 GB

PM1 PM2

Power 225 290

Wastage 0.0002 0.0140

Mem: 8 GB

VM3

1400 MIPS

5 GB

PM3

270

0.0570

VM4

1000 MIPS

5 GB

PM4

250

0.1110

DC

1035

: 200 W TPM

000 000 000 400

400 000 000 000

000 400] 000 000

0.1822

4 × (4 × 400) = 6400 𝑀𝐵

TDT

2 × (0 × 400) + 2 × (2 × 400) = 𝟏𝟔𝟎𝟎 𝑴𝑩

re

PowerFull : 300 W

000 [000 400 000

-p

Power

Idle

ro

CPU: 2000 MIPS

VM1 VM2

of

PMi (𝒊 = 𝟏, … , 𝟒) Specifications:

lP

6. Proposed Multi-objective communication-aware optimization geneticbased algorithm

ur na

In this section, we explain the proposed multi-objective genetic-based solution for combinatorial VMP problem. Canonical GA was originally designed to optimize one objective function. Therefore, we improve GA in such a way that it covers all equally important objective functions. Also, it is needed to explain about gens and chromosomes definitions for problem encoding, besides data structures which are used for servers in this paper. As gens and chromosomes are cornerstone of GA, in forthcoming sub sections these concepts relevant to improved GA are explained.

6.1. Gens and Chromosome

Jo

Since n requested VMs to be hosted on m PMs, the chromosome length is n, where the gens are integer numbers in {1, … , 𝑚} such as gene representation in [76]. We exemplify that there are a request for 10 VMs, namely, VM1 through VM10 to be hosted on 8 PMs. Consider the chromosome ch: 1

ch:

2

2 2

3

4

5

1 4 3

6

7

2 1

8

9 10

4 1 3

The numbers of upper row are VMs indices and the numbers of downer row demonstrate the PMs’ indices. That means VM3, VM7 and VM9 are placed on PM1; VM1, VM2 and VM6 are placed on PM2; VM5 and VM10 are placed on PM3; VM4 and VM8 are placed on PM4 respectively.

6.2. Data structures

of

In addition to Besides, each physical server PMj has a data structure (DSj) as format below:

ro

Struct server specification for PMj is called DSj { Status ∈ {on Active, off Hibernate} 𝐶𝑝𝑢 𝐶𝑗 : Cpu capacity of PMj ; 𝐶𝑗𝑀𝑒𝑚 : Memory capacity of PMj ; 𝐶𝑝𝑢

𝑈𝑗

: Cpu utilization of PMj ;

-p

𝐶𝑝𝑢

re

𝑈𝑗 : Memory utilization of PMj ; List : List of PMj VMs which are co-hosted on PMj ; } // end of structure

𝑙𝑒𝑛𝑔𝑡ℎ(𝑃𝑀𝑗 .𝐿𝑖𝑠𝑡)

=∑

𝑈𝑗𝑀𝑒𝑚

𝑖=1

𝑙𝑒𝑛𝑔𝑡ℎ(𝑃𝑀𝑗 .𝐿𝑖𝑠𝑡)

=∑

𝐶𝑝𝑢

𝑉𝑀𝑖

𝑉𝑀𝑖𝑀𝑒𝑚

ur na

𝐶𝑝𝑢

𝑈𝑗

lP

Note that, server status can be in two states Active/Hibernate; the last status is in low power state and it can suddenly back to data center to improve the system productivity. Note that In this structure, processing and memory utilization are attained via equations (25) and (26) respectively.

𝑖=1

(25) (26)

Jo

To determine an overflowed server, we define a Threshold parameter. We considered Threshold to have small bandwidth for context switch and H/W polling of a server. As we normalized resource request, it is set to 90% or 0.90 in this paper. For the sake of simplicity we take static Threshold for both CpuThr and MemThr because we normalize all parameters. So, the server PMj 𝐶𝑝𝑢 having 𝑈𝑗 > 𝑇ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 or 𝑈𝑗𝑀𝑒𝑚 > 𝑇ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 is considered overflow. To generate initial population, we define two dimensional matrix 𝑃𝑜𝑝(1: 𝑃𝑜𝑝𝑆𝑖𝑧𝑒, 1: 𝑛) 𝑃𝑜𝑝[1. . 𝑃𝑜𝑝𝑆𝑖𝑧𝑒][ 1. . 𝑛] where PopSize is a fixed number for length of population; each individual 𝑃𝑜𝑝(𝑖, 1: 𝑛) 𝑃𝑜𝑝[i][1. . n] representative of a chromosome Chi which is a random solution. Therefore, in a solutioni , 𝑃𝑜𝑝(𝑖, 𝑗) = 𝑘 𝑃𝑜𝑝[𝑖][𝑗] = 𝑘 means that VMj is placed on PMk. Note that, each PMj has a data structure DSj, then all individual in Pop has data structure; consequently, two-dimensional matrix DS(1: 𝑃𝑜𝑝𝑆𝑖𝑧𝑒, 1: 𝑚) DS[1. . 𝑃𝑜𝑝𝑆𝑖𝑧𝑒][1. . 𝑚] can be attained via 𝑃𝑜𝑝(1: 𝑃𝑜𝑝𝑆𝑖𝑧𝑒, 1: 𝑛) 𝑃𝑜𝑝[1. . 𝑃𝑜𝑝𝑆𝑖𝑧𝑒][ 1. . 𝑛] vice versa. Moreover, pseudo-code below determines whether each PMk in solutioni is overflow or not; where (𝑖, 𝑗) = 𝑘 𝑃𝑜𝑝[𝑖][𝑗] = 𝑘.

𝐶𝑝𝑢

if (DS (i,k) DS[i][k]. 𝑈𝑗 > Threshold) OR (DS (i,k) DS[i][k]. 𝑈𝑗𝑀𝑒𝑚 > Threshold) then Server PMk is overflow end if Algorithm 2 in Fig.8 elaborates proposed improved multi-objective GA. The final output is nondominated solutions regarding to triple objectives energy consumption, resource wastage, and network bandwidth usage.

Jo

ur na

lP

re

-p

ro

of

Algorithm 2. Hybrid meta-heuristic GA-based VMP Input: PopSize: Size of population and problem characterization, and fixed value q0 ϵ (0..1) Output: Pareto set of non-dominated solutions Initialize population phase: 1: for i = 1 to PopSize do 2: Pop (i , 1 : n) Pop[i] [1..n] ← spread randomly VM1 to VMn into PM1 through PMm 3: end for /* end of initialization 4: for i = 1 to PopSize do 5: Pop (i , 1 : n Pop[i] [1..n]) =  Check & Correct (Pop (i , 1 : n) Pop[i] [1..n]) 6: Pop (i , 1 : n) Pop[i] [1..n] =  Exploitation (Pop (i , 1 : n) Pop[i] [1..n]) 7: Fit1 (i) [i] =  Fitness1 (Pop (i , 1 : n) Pop[i] [1..n]) 8: Fit2 (i) [i] =  Fitness2 (Pop (i , 1 : n) Pop[i] [1..n]) 9: Fit3 (i) [i] =  Fitness3 (Pop (i , 1 : n) Pop[i] [1..n]) 10: end for 11: /* Create Pareto Set PS from Pop regard to multi-objective */ 12: PS ← non-dominated (Pop) 13: while termination criteria is not met do 14: for i = 1 to PopSize do 15: j =  Roulettewheel (Pop) based on Fit1 16: Mate1 (i) [i] =  j where i ≠ j 17: k =  Roulettewheel (Pop) based on Fit2 18: Mate2 (i) [i] =  k where i ≠ k 19: m =  Roulettewheel (Pop) based on Fit3 20: Mate3 (i) [i] =  m where i ≠ m 21: end for 22: Pop1 =  Crossover Pop members with Mate1 elements with probability 23: Pop2 =  Crossover Pop members with Mate2 elements with probability 24: Pop3 =  Crossover Pop members with Mate3 elements with probability 25: Pop4 =  Mutation Pop with probability 26: Pop1 =  Check & Correct (Pop1) 27: Pop2 =  Check & Correct (Pop2) 28: Pop3 =  Check & Correct (Pop3) 29: Pop4 =  Check & Correct (Pop4) 30: Pop5 ← Pop1 ∪ Pop2 ∪ Pop3 ∪ Pop4 ∪ PS 31: PS =  non-dominated (Pop5) 32: Pop ← Add 10% top most individuals of Pop5 according to Fit1 33: Pop ← Add 10% top most individuals of Pop5 according to Fit2 34: Pop ← Add 10% top most individuals of Pop5 according to Fit3 35: Pop ← Add 10% top most individuals of Pop5 according to Fit1+Fit2 36: Pop ← Add 10% top most individuals of Pop5 according to Fit2+Fit3 37: Pop ← Add 10% top most individuals of Pop5 according to Fit1+Fit3 38: Pop ← Add 40% top most individuals of Pop5 according to Fit1+Fit2+Fit3 39: /* Exploitation Phase 40: Draw randomly q ϵ [0..1] 41: if q > q0 then

42 Pop  Call Exploitation(Pop) 43: end if 39: Randomly apply Pop ← Exploitation (Pop) 44: end while return the PS as non-dominated solution Figure 8. Algorithm 2. Hybrid meta-heuristic GA-based VMP

re

-p

ro

of

As Algorithm 2 starts with casual input which line #1 through line #3 show, it may lead imbalanced load between PMs or overflow one. This is a reason to design check&correct function, which investigates a solution is feasible or not. If not so, it offloads VMs from PM to put its utilization less than predetermined upper threshold. Algorithm 3 in Fig.9 illustrates this function in details. Algorithm 3 uses circular queue to find PM for offloading overloaded one. When it finds a PM, which has enough CPU and Memory capacities; it deletes VM(s) from source PM and add VM(s) to destination PM. Note that it tries to find active PM for destination otherwise it launches new PM and changes its status from off Hibernate to on Active mode. After the feasible initial population is made, the rudimentary Pareto set (PS) is then constructed based on dominance concept.; it is illustrated in line #12 of main algorithm. However, PS would be improved during the algorithm iterations. Since a meta-heuristic algorithm does not terminate until reaching predetermined iterations or getting predetermined fitness value, we take 50 times for iteration as termination criteria. It can be set in line #13 of main algorithm. The number 50 generations is set experimentally because setting greater iterations has not led to better performance. Then, the average of 20 independent executions of different scenarios are reported in section 7.

Jo

ur na

lP

Algorithm 3. Check & Correct (Pop) Input: Population Pop with PopSize as length, n: number of VMs, m: number of PMs Output: Population Pop with PopSize as length, if any change needed Begin 1: DS (1 : PopSize , 1 : m) [1..PopSize][1 .. m] = Derive from (Pop (1 : PopSize , 1 : n) [1..PopSize][1.. n]) 2: for i = 1 to PopSize do 3: for j = 1 to m do 4: if (DS (i , j)[i][j].UjCPU > Threshold) OR (DS (i , j)[i][j].UjMem > Threshold) then 5: k =  DS (i , j)[i][j].List (1)[1] ; (* k is VM’s index which is placed on PMj *) 6: Index = False 7: ℓ =  (j + 1) mod m ; 8: if ℓ = 0 then ℓ = 1 ; end if 9: while ℓ ≠ j do 10: if (DS (i , ℓ)[i][ℓ].Status = on Active) and 11: ((DS (i , ℓ)[i][ℓ].CℓCPU - DS (i , ℓ) [i][ℓ].UℓCPU) ≥ Request (k)[k].CPU) and 12: ((DS (i , ℓ)[i][ℓ].CℓMem – DS(i , ℓ) [i][ℓ].UℓMem) ≥ Request (k)[k].Mem) then 13: (* drop VMk from PMj *) 14: DS (i , j)[i][j].UjCPU =  DS (i , j)[i][j].UjCPU - Request (k)[i].CPU 15: DS (i , j)[i][j].UjMem =  DS (i , j)[i][j].UjMem - Request (k)[i].Mem 16: DS (i , j)[i][j].List =  [DS (i , j)[i][j].List (2 : end)[2][end]]; 16: drop the first virtual machine as VMk from DS[i][j].List 17: (* Add VMk to PMℓ *) 18: DS (i , ℓ)[i][ℓ].UℓCPU =  DS (i , ℓ) [i][ℓ].UℓCPU + Request (i)[i].CPU 19: DS (i , ℓ) [i][ℓ].UℓMem =  DS (i , ℓ) [i][ℓ].UℓMem + Request (i)[i].Mem 20: DS (i , ℓ) [i][ℓ].List =  [DS (i , ℓ) [i][ℓ].List , k]; 20: Add VMk to the end list of DS (i , ℓ) [i][ℓ].List 21: Index = True ; 22: break ;

ur na

lP

re

-p

ro

of

23: end if 24: ℓ =  (ℓ + 1) mod m; 25: if ℓ = 0 then ℓ = 1 ; end if 26: end while 27: if Index = False then ℓ =  (j + 1) mod m; 28: if ℓ = 0 then ℓ = 1 ; end if 29: while ℓ ≠ j do 30: if (DS (i , ℓ)[i][ℓ].Status = off Hibernate) then 31: (* drop VMk from PMj *) 32: DS (i , j)[i][j].UjCPU =  DS (i , j)[i][j].UjCPU - Request (k)[k].CPU 33: DS (i , j)[i][j].UjMem =  DS (i , j)[i][j].UjMem - Request (k)[k].Mem 34: DS (i , j)[i][j].List =  [DS (i , j)[i][j].List(2 : end)[2][end]]; 34: drop the first virtual machine as VMk from DS[i][j].List 35: (* Add VMk to PMℓ *) 36: DS (i , ℓ)[i][ℓ].Status =  on Active; 37: DS (i , ℓ)[i][ℓ].UℓCPU =  DS (i , ℓ)[i][ℓ].UℓCPU + Request (k)[k].CPU 38: DS (i , ℓ)[i][ℓ].UℓMem =  DS (i , ℓ)[i][ℓ].UℓMem + Request (k)[k].Mem 39: DS (i , ℓ)[i][ℓ].List =  [DS (i , ℓ)[i][ℓ].List , k]; 39: Add VMk to the end list of DS (i , ℓ) [i][ℓ].List 40: Index =  True ; 41: break ; 42: end if 43: ℓ =  (ℓ + 1) mod m; 44: if ℓ = 0 then ℓ = 1 ; end if 45: end while 46: end if 47: end if 48: end for 49: for j = 1 to m do 50: Len =  length (DS (i , j)[i][j].List) ; 51: for k = 1 to Len do 52: Pop (i ,DS (i , j).List (k) = j 53 IDXDS [i][j].List[k] 54 Pop [i][IDX]  j 55: end for 56: end for 57: end for End {Algorithm 3} Figure 9. Algorithm 3. Check & Correct (Pop)

Jo

In Algorithm 2, the Fitness function is determined based on objective function. For instance, Fitness 1, Fitness 2 and Fitness 3 calculate chromosome’s value regarding to the first, the second, and the third objective functions. The Roulettewheel function is used to select an individual with good fitness value to be selected with more chance such as in [1, 77]. By this way, Roulettewheel function finds a mate for current individual to generate new children. The selected individual is then used in crossover and mutation operations with predetermined probabilities. In this paper, crossover and mutation rates are experimentally considered 90% and 5% respectively. These operations are applied to generate intermediate populations Pop1 through Pop4. In each crossover operation, two chromosomes which have been selected by Roulettewheel and are conducted by their fitness function are mated to generate one child. Crossover operator is randomly biased by fitness function of two parents; here, we don’t apply one- or two-point crossover instead we construct a uniform mask; then, we make new child based on uniform mask. For mutation, two

random numbers are drawn to put random VM on random PM. Fig.10 and Fig.11 illustrate crossover and mutation functions respectively. Note that, after each Crossover and Mutation operation, check & correct function is called to verify changes and amends it if necessary.

lP

re

-p

ro

of

Algorithm 4. Crossover Input: Chromosome Chr1 = [a1a2...an], Chr2= [b1b2...bn] , with n as length of chromosomes, objective functions obj Output: NewChr = c1c2...cn as new chromosome 1: f1 = Fitness (Chr1) based on objective function obj 2: f2 = Fitness (Chr2) based on objective function obj 3: Mask = zeros (1 , n) [1..n] ; 4: Token =  f1 / (f1 + f2) 5: for i = 1 to n do 6: draw random r so that 0 ≤ r ≤ 1 7: if r < Token then Mask (i)[i] =  0 ; 8: else Mask (i)[i] =  1 ; 9: end if 10: end for 11: for i = 1 to n do 12: if Mask (i)[i] = 0 then NewChr (i)[i] =  Chr1 (i)[i] ; 13: else NewChr (i)[i] =  Chr2 (i)[i] ; 14: end if 15: end for return NewChr (1:n)[1..n]; Figure 10. Algorithm 4. Crossover

ur na

Algorithm 5. Mutation Input: A chromosome Chr = [c1c2...cn] , m: number of PMs, n: number of VMs Output: A chromosome NewChr = [c1ʹc2ʹ...cnʹ ] Begin 1: put NewChr (1:n)[1..n] =  Chr (1:n)[1..n] 2: draw two random numbers 1 ≤ i ≤ n , 1 ≤ j ≤ m ; 3: NewChr (i)[i] =  j 4: return NewChr (1:n)[1..n]; End Figure 11. Algorithm 5. Mutation

Jo

After Pop1, Pop2, Pop3 and Pop4 are calculated by crossover and mutation operators, and also checked and corrected, new Pareto set is calculated from sum of Pop1 through Pop4 plus last PS members. For the next round, new Pop is filled by portion of other generated population base on one or mixed of objective functions. The number 10% and 40% were experimentally attained. Afterwards, randomly Exploitation function is called to improve the algorithm because continuous calling Exploration function leads getting stuck in local optimum and rapid convergence. For this reason we can set fixed value q0=0.5 as another input of Algorithm 2; then, we draw a random number q which belongs to [0..1]; only if the value of q is greater than fixed value q0, then, the Exploitation procedure is called to improve exploration performance. This routine is done at the end of each iteration which can be seen in line #40 through line #43 of Algorithm 2. The randomness makes balance between exploration and exploitation in search space; consequently,

the hybrid multi-objective approach has better performance against pure multi-objective metaheuristic. Fig.12 depicts Exploration Exploitation function as complementary of hybrid algorithm. As exploration may lead sub-optimal solutions, the exploitation function can improve overall performance. This function is a heuristic algorithm to decline the number of active servers; then, it tries to find a PMj which has the capacity more or equal than all of another co-hosted VMs on current PMi; so, if it finds such a server PMj, it offloads all co-hosted VMs from PMi to PMj; then, it switches PMi’s status off to Hibernate mode. In this way, it can reduce power consumption by using less active servers, potentially resource wastage and possibly reduction in bandwidth usage.

Jo

ur na

lP

re

-p

ro

of

Algorithm 6. Exploitation (Pop) Input: Pop as population with PopSize individuals, n: number of VMs, m: number of PMs Output: The optimal population with minimum number of used PMs Begin 1: for i = 1 to PopSize do 2: for j = 1 to m do 3: if DS (i , j) (i , j).Status = on Active do 4: ℓ=j+1 5: while ℓ ≤ m do 6: if (DS (i , ℓ) [i][ℓ].Status = on Active) and 7: (DS (i , ℓ) [i][ℓ].CℓCPU - DS (i , ℓ) [i][ℓ].UℓCPU ≥ DS (i , j) [i][j].UjCPU ) and 8: (DS (i , ℓ) [i][ℓ].CℓMem - DS (i , ℓ) [i][ℓ].UℓMem ≥ DS (i , j) [i][j].UjMem ) then 9: (* Add VMj to PMℓ *) 10: DS (i , ℓ) [i][ℓ].UℓCPU = DS (i , ℓ) [i][ℓ].UℓCPU + DS (i , j) [i][j].UjCPU 11: DS (i , ℓ) [i][ℓ].UℓMem = DS (i , ℓ) [i][ℓ].UℓMem + DS (i , j) [i][j].UℓMem 12: DS (i , ℓ) [i][ℓ].List =  Merge[DS (i , ℓ) [i][ℓ].List , DS (i , j) [i][j].List] ; 13: (* Remove VMj from PMj *) 14: DS (i , j) [i][j].UjCPU =  0 15: DS (i , j) [i][j].UjMem =  0 16: DS (i , j) [i][j].List =  [ ] 17: DS (i , j) [i][j].Status =  off Hibernate 18: break ; 19: end if 20: ℓ =  (ℓ + 1) 21: end while 22: end if 23: end for 24: for j = 1 to m do 25: Len =  length (DS (i , j) [i][j].List) 26: for k = 1 to Len do 27: Pop (i , DS (i , j).List (k)) = j 27 IDXDS [i][j].List[k] 28 Pop [i][IDX]  j 29: end for 30: end for 31: end for return Pop (1 : PopSize , 1 : n) [1..PopSize][1..n] End Figure 12. Algorithm 6. Exploitation (Pop)

7. Simulation, evaluation, and time complexity

Here, we assess proposed solution by definition extensive scenarios. Forthcoming sub-sections are devoted to scenarios, datasets, and parameter setting definitions along with final evaluation. We also allege the proposed time complexity. 7.1 Scenarios, datasets, and parameter setting

Jo

ur na

lP

re

-p

ro

of

This section clarifies scenarios, used dataset and parameters, which are applied for proposed algorithm evaluation. The performance of proposed algorithm must be verified in different circumstances to rely on its robustness. Since its performance depends on the size and the number of both VMs and PMs, extensive scenarios are defined. Then, we investigate algorithm outcomes in terms of predetermined relevant parameters, when the number of requested VMs increases whereas the number of PMs is considered fixed, and the observation of algorithm changes when both number of VMs and PMs increase gradually. Also, we consider resource variations in requested VMs. The second part is taken into consideration to investigate potential scalability of different algorithms. To do so, we consider Dell server families with 6000 MIPS and 8 GB as CPU and Memory capacities respectively in homogenous datacenter, where with which each server’s 𝑓𝑢𝑙𝑙 power usage is Emin=162 𝑃𝑗𝐼𝑑𝑙𝑒 = 162 and Emax=215 𝑃𝑗 = 215 in idle and full load conditions respectively [2]. Note that, cloud datacenters take benefit of SAN storage and broker module; all of PMs can uniformly access to SAN storage. This is the reason for not taking disk into consideration. Also, SAN storage contains different VMs’ images which are used for VMs launches and executions of cloudlet assigned to them wherever the broker module assign VMs to PMs. For the sake of simplicity, we normalize resource usage. For instance, the requested VM, which needs 1500 MIPS and 4 GB of resource bandwidth is considered CpuReq=0.25 and MemReq=0.5. Without loss of generality and for the sake of simplicity, we take Fat-Tree topology and homogeneous datacenter, where the switch scales are 𝑝 = 4 with covering 16 physical servers and 𝑝 = 8 with covering 128 physical servers at most (it can be observed in Fig.4.a and Fig.4.b respectively). However Also, we can extend this algorithm to heterogeneous datacenter and other topologies. Even even for other bigger larger scales. To reach concrete results, we have considered different scenarios. Since the users request VMs so that which may have variable correlation in resource usage such as CPU and RAM usage commensurate with their needed applications, we define new dataset with most relevant usage pattern (correlation coefficient = +0.85), rather relevant usage pattern (correlation coefficient = -0.07), and adverse relevant usage pattern (correlation coefficient = -0.85) between resources. Here, we introduce 10 different scenarios, where each scenario is run on 3 different datasets having different correlation coefficient between resource vectors. Table 4, illustrates determined scenarios. Moreover, scenarios #6 through #10 are devoted to prove algorithm scalability. Table 4. Scenarios specifications

Scenario No.

1

2

3

4

5

6

7

8

9

10

Requested VMs

15

20

25

30

35

80

90

100

110

120

Number of PMs

16

16

16

16

16

30

40

50

60

120

Switch Port

p=4

p=8

In this paper, resource request for VMs have been considered in such a way that it includes variety correlation spectrums. To generate dataset with desired correlation coefficient, we applied pseudo code in Algorithm 7, which Fig.13 depicts. By considering appropriate variable 𝑝𝑎𝑟 ϵ [0. .1.0] we can attain resource vectors with correlation coefficients belonging to {−0.85, −0.07, +0.85}. Also, variables ̅̅̅̅̅̅ 𝑅𝐶𝑝𝑢 and ̅̅̅̅̅̅̅ 𝑅𝑀𝑒𝑚 are set to 30% so that vectors CpuRequest and MemRequest of a requested VM belong to [0...60%) of a full server resource. Moreover, we take 90% as a server resource threshold of a physical server to adopt VMs. Overall, we define 10 different scenarios each of which with 3 variable resource needs, which are 30 total scenarios.

re

-p

ro

of

Algorithm 7. Dataset Generation Input: n, par, ̅̅̅̅̅̅ 𝑅𝐶𝑝𝑢 and ̅̅̅̅̅̅̅ 𝑅𝑀𝑒𝑚 Output: two vectors CpuRequest and MemRequest representative of VMs resource request 1: for i=1 to n do 2: CpuRequesti = rand (2 * ̅̅̅̅̅̅ 𝑅𝐶𝑝𝑢 ); ̅̅̅̅̅̅̅ 3: MemRequesti = rand (𝑅 𝑀𝑒𝑚 ); 4: r = rand (1.0); 5: if ( r < par and CpuRequesti ≥ ̅̅̅̅̅̅ 𝑅𝐶𝑝𝑢 ) OR ( r ≥ par and CpuRequesti < ̅̅̅̅̅̅ 𝑅𝐶𝑝𝑢 ) then 6: end if 7: end for Figure 13. Algorithm 7. Pseudo-code for generating dataset

ur na

7.2 Evaluation

lP

In addition to, traffic pattern matrix (TPM) variable determines the data streaming volume between each pair of (VMi ,VMj); this volume can belong to [25-250MB], whereas for independent VMs, this amount is considered zero. Note that, this matrix’s value can be attained via switches’ and datacenter’s profile or data mining techniques.

Jo

As we have mentioned earlier, we defined different scenarios with variable resource demand for requested VMs. To evaluate our proposed multi-objective genetic-based (MOGA) algorithm, we compare it with Multi-objective ACO-based (MOACO) VM placement [48], well-known heuristic-based FFD [35] and Random VMP approaches in terms of total power consumption, total resource wastage, and total bandwidth usage for DCs. Note that, for the sake of gaining robust results, the average outcome of 20 independent executions are reported in Fig. 14 through Fig. 23 respectively. In this regard, Fig.14 through Fig.23 separately illustrate comparison between different scenarios in terms of aforementioned metrics. Since our proposed algorithm is multiobjective in nature, the feasible solutions are in the form of non-dominated points. In this line, we utilize two important parameters, namely convergence and diversity to contrast with other approach, which generates Pareto curve. To analyze the performance of our proposed multiobjective algorithm versus the state-of-the art ACO-based VMP approach, we compare scenarios’ results in terms of convergence and diversity parameters of solutions in search space. To do so, for diversity we use overall non-dominated vector generation (ONVG) parameter [78] and for divergence we use Spacing (SP) parameter [79]. For this reasons, parameters ONVG and SP can be calculated by equations (27) and (28) respectively.

𝑂𝑁𝑉𝐺 = |𝑌𝑃𝑆 | 𝑆𝑃 = √|𝑌

1 𝑃𝑆 |−1

(27) |𝑌

|

𝑃𝑆 ∑𝑖=1 (𝑑̅ − 𝑑𝑖 )

2

(28)

of

𝑚 Where |𝑌𝑃𝑆 | implies the number of solutions in non-dominated set PS and 𝑑𝑖 = min𝑗 (∑𝑚 𝑘=1‖𝑓𝑖 − 𝑓𝑗𝑚 ‖), 𝑖, 𝑗 = 1, … , |𝑌𝑃𝑆 |, the terms 𝑓𝑗𝑚 , m and 𝑑̅ are j-th objective function, number of objectives and mean value of all of 𝑑𝑖 . We also evaluate our proposed algorithm results in comparison with other approaches in terms of total energy consumption, total resource wastage, total bandwidth usage, total active servers in use. Note that, for the sake of gaining robust results, the average outcome of 20 independent executions are reported in Fig. 14 through Fig. 23 respectively. Also, Fig.14 through Fig.23 separately illustrate comparison between different scenarios in terms of power consumption, resource wastage and total data transfer rate.

Jo

ur na

lP

re

-p

ro

Scenario 1 through Scenario 5 are considered for fixed number of PMs and gradually increasing number of requested VMs. Figure 14 depicts the performance comparison of different approaches for the first scenario, as the simplest scenario, where requested 15 VMs to be placed on 16 PMs. In term of resource wastage, our algorithm has significantly dominance against others. Only MOACO can compete with MOGA in terms of power consumption and bandwidth usage. In term of power consumption when correlation coefficient is -0.85, MOGA beats MOACO marginally, but in other resource variation both MOGA and MOACO have the same behavior. MOGA acts more efficient than MOACO in term of bandwidth usage when correlation coefficient is -0.85, but in other cases it has marginally dominance. Both FFD and Random search have inferior performance against MOGA and MOACO in all cases. On the other hand, both FFD and Random search have near treatment in terms of power consumption and bandwidth usage, but in term of resource wastage Random search is dominated by FFD.

Figure 14. Scenario 1: 15 VMs on 16 PMs

lP

re

-p

ro

of

Figure 15 depicts the performance comparison of different approaches for the second scenario. MOACO and MOGA have the same behavior in term of power consumption in all cases in different resource variation, but MOGA outperforms against MOACO in all cases in terms of resource wastage and total bandwidth usage. FFD and random search have the same treatment in power consumption in correlation coefficient -0.07 and 0.85, but in other cases they do not have clear dominance against each other. Only FFD outperforms against other approaches in bandwidth usage merely in correlation coefficient -0.07. It may origin from its special VMs data relationship in dataset with aforesaid correlation because FFD which concentrates on the first two objectives cannot have any triumph in other circumstances.

ur na

Figure 15. Scenario 2: 20 VMs on 16 PMs

Jo

Figure 16 depicts the performance comparison of different approaches for the third scenario, where requested 25 VMs to be placed on 16 PMs. In this scenario, MOGA, MOACO, FFD, and Random search has the best to the worst rankings based on objective functions in the most cases. Note that, in power consumption comparison in correlation 0.85, all approaches have the same behavior.

of ro

Figure 16. Scenario 3: 25 VMs on 16 PMs

Jo

ur na

lP

re

-p

Figure 17 depicts the performance comparison of different approaches for the fourth scenario, when requested 30 VMs to be hosted on 16 PMs. Such as other pervious scenarios FFD and Random search are the worst, but MOACO has marginally dominance against MOGA in term of resource wastage for correlation coefficients {-0.85, 0.85} and has the same treatment in correlations {-0.07, 0.85} in term of power consumption versus MOGA. Note that, high rate of resource wastage is not necessarily bad feature. For instance for the same problem, an algorithm A may lead 70% resource wastage but with 4 servers in use; on the other hand, another algorithm B my lead 60% resource wastage but with 5 servers in use. Consequently, the resource wastage is meaningful with regards to the server in use; this is the reason to bring server in use parameter in Table 5. A part from power wastage calculation where FFD and Random search have the same performance in other cases they do not have any meaningful dominance against each other.

of ro -p

Figure 17. Scenario 4: 30 VMs on 16 PMs

lP

re

In Fig.14 through Fig.17, which are relevant to scenarios1 scenario 1 through scenario4 scenario 4, it can be observed that our proposed algorithm outperforms against other approaches in terms of total power consumption, resource wastage and total bandwidth usage for all correlation coefficients except for in correlation coefficient {−0.07, 0.85} so that only power consumption reported by proposed algorithm and ACO are the same.

Jo

ur na

In scenario 5 where the search space is larger than the four previous one, our algorithm superiority is more tangible versus other approaches except for in correlation coefficient -0.07 where our GAbased solution works the same as MOACO in term of power consumption; moreover, MOACO algorithm has marginally dominance against our algorithm in term of resource wastage for the case of correlation coefficient 0.85. Recall that, high rate of resource wastage is bad feature when the number of active server in use is high too. In this scenario, the number of active servers is 6 for MOGA algorithm whereas it is 7 for MOACO; consequently, high rate of resource wastage which was reported by MOGA is not necessarily bad feature against ACO-based algorithm in this episode. It can be observed in Table 5 in details.

of ro -p

Figure 18. Scenario 5: 35 VMs on 16 PMs

ur na

lP

re

In scenario 5 where the search space is bigger than the four previous one, our algorithm superiority is more tangible versus other approaches except for in correlation coefficient -0.07 where our GAbased solution works the same as ACO in term of power consumption; moreover, ACO-based algorithm marginally has dominance against our algorithm in term of resource wastage for the case of correlation coefficient 0.85. Note that, high degree of resource wastage is bad feature in the case that the number of active server in use is high too. In this scenario, the number of active servers is 6 for GA-based algorithm whereas it is 7 for ACO-based; consequently, high rate of resource wastage which was reported by GA is not bad feature against ACO-based algorithm in this episode. Scenario 6 through Scenario 10 are considered for the cases that both number of requested VMs and underlying PMs gradually have increasing trend. It is the case to investigate the scalability potential of different approaches.

Jo

Fig. 19, for scenario 6, proves that MOGA outperforms against others in terms of all objectives in all cases. After MOGA, MOACO beats others in all objectives except for in bandwidth usage where MOACO, FFD, and Random search have the marginal competition.

of ro

Figure 19. Scenario 6: 80 VMs on 30 PMs

Jo

ur na

lP

re

-p

Fig. 20, for scenario 7, proves that MOGA outperforms against others in terms of all objectives in all cases. Also, MOACO beats others in all objectives in all cases after MOGA. Moreover, FFD and Random search do not have dominance against themselves.

Figure 20. Scenario 7: 90 VMs on 40 PMs

Fig. 21, for scenario 8 where requested 100 VMs to be hosted on 50 PMs has the same behavior in comparison with Fig.20 which is drawn for scenario 7.

of ro

Figure 21. Scenario 8: 100 VMs on 50 PMs

Jo

ur na

lP

re

-p

Fig. 22, for scenario 9 where requested 110 VMs to be hosted on 60 PMs has the same behavior in comparison with Fig.20 and Fig. 21 which are drawn for scenario 7 and scenario 8 respectively, but the only difference is related to dominance of FFD against Random search approach in the most cases of scenario 9.

Figure 22. Scenario 9: 110 VMs on 60 PMs

Also, in Fig. 22 for scenario 10, MOGA has superiority in all objectives with different correlations. It simply proves the high potential of scalability for MOGA by tracking the amount of discrepancies results from different objectives against other approaches.

of ro -p

Figure 23. Scenario 10: 120 VMs on 120 PMs

Jo

ur na

lP

re

By considering larger search space, in scenarios 6 through 10, as Fig.19 through Fig.23 demonstrate, it can be observed that our proposed GA-based solution has obvious dominance in comparison with state-of-the art ACO-based algorithm, well-known FFD heuristic and Randombased approaches in terms of total power consumption, resource wastage and total bandwidth usage especially in all of correlation coefficients. It also proves that our algorithm is more scalable against others in which it can traverse large search space efficiently. Since our proposed algorithm is multi-objective in nature, the feasible solutions are in the form of non-dominated points. On the other hand, several meta-heuristics may be conducted in such a way that they can constitute Pareto set of non-dominated solutions; but a big question arises. The question is: which of meta-heuristics has superiority against others in solving multi-objective problems? In this line, we utilize two important parameters, namely, convergence and diversity to contrast with other approach, which generates Pareto curve. To analyze the performance of our proposed multi-objective algorithm versus the state-of-the art ACO-based VMP approach, we compare scenarios’ results in terms of convergence and diversity parameters of solutions in search space. To do so, for diversity we use overall non-dominated vector generation (ONVG) parameter [78] and for divergence we use Spacing (SP) parameter [79]. For this reasons, parameters ONVG and SP can be calculated by equations (27) and (28) respectively. Note that, a good solution set has SP value close to 0; also, the solution which efficiently traverse search space gains greater value in ONVG parameter. Therefore, the small value for SP and the bigger value for ONVG parameters are favorable when you compare different algorithms solving the same multi-objective problem. This is the reason to bold better value in Table 5.

𝑂𝑁𝑉𝐺 = |𝑌𝑃𝑆 | 𝑆𝑃 = √|𝑌

1 𝑃𝑆 |−1

(27) |𝑌

|

𝑃𝑆 ∑𝑖=1 (𝑑̅ − 𝑑𝑖 )

2

(28)

Where |𝑌𝑃𝑆 | implies the number of solutions in non-dominated set PS and 𝑑𝑖 = 𝑚 𝑚 𝑚 ̅ min𝑗 (∑𝑚 𝑘=1‖𝑓𝑖 − 𝑓𝑗 ‖), 𝑖, 𝑗 = 1, … , |𝑌𝑃𝑆 |, the terms 𝑓𝑗 , m and 𝑑 are j-th objective function, number of objectives and mean value of all of 𝑑𝑖 . We also evaluate our proposed algorithm results in comparison with other approaches in terms of total energy consumption, total resource wastage, total bandwidth usage, total active servers in use. To illustrate analytical data analysis and numerical investigations, Table 5 demonstrates our approach superiority against other heuristics and meta-heuristics in terms of determined parameters in details.

PMs

Dataset

16

-0.07

0.85

-0.85

2

20

16

-0.07

-0.85

25

16

30

-0.07

0.85

Jo 4

Bandwidth

Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD

613.5497 627.8030 789.8030 789.8030 616.9292 616.9292 778.9292 778.9292 588.3252 588.3252 616.9292 616.9292 798.1011 798.2607 1122.300 960.2607 1011.300 1011.300 1011.300 1011.300 800.0486 800.0486 800.0486 800.0486 969.7314 979.4034 1141.400 1141.400 973.9280 973.9280 1135.900 1135.900 1015.400 1015.400 1015.400 1015.400 1173.800 1183.800 1345.800 1345.800 1202.800 1202.800 1364.800 1364.800 1175.600 1175.600 1175.600

0.5884 0.8748 1.3232 1.4966 0.1772 0.1829 1.1621 1.9460 0.0308 0.1481 0.1945 1.1937 0.3990 0.8957 2.6362 1.3515 1.1752 1.2318 1.2244 2.6865 0.1786 0.1831 0.3278 0.9405 0.5482 0.5739 2.4695 3.1861 0.9006 1.1832 1.9347 3.3494 0.1701 0.3484 0.2940 1.1261 0.4917 0.4511 2.7947 3.1198 0.3197 0.8726 2.3195 2.3968 0.5446 0.5264 1.1906

63800 70900 71200 72400 67100 68500 73150 73650 56700 57150 57150 58400 121450 134400 171850 164050 151100 155550 148700 163600 114500 116800 119050 124150 271500 276400 286800 317800 292800 297600 293100 314350 246000 259250 290500 277650 418500 427100 466950 454900 410950 445500 448050 480750 376000 377800 439800

ur na

0.85

3

Wastage

-0.85

16

-0.07

0.85

Used server

ONVP

SP

3 3 4 4 3 3 4 4 3 3 3 4 4 6 6 6 5 5 5 6 4 4 4 4 5 6 6 7 5 6 6 7 5 5 5 6 6 6 8 8 6 6 8 8 6 6 9

17 14 23 15 11 10 16 10 32 18 18 12 11 10 10 10 14 12 17 15 25 15 13 11 -

0.0851 0.2284 0.1711 0.2223 0.3588 0.4132 0.1524 0.1896 0.2278 0.6143 0.1454 0.2626 0.3610 0.9857 0.3565 0.4663 0.0886 0.1083 0.2640 0.5156 0.0689 0.4912 0.0914 0.1772 -

ro

15

Energy

-p

-0.85

1

Algorithm

Correlation

re

VMs request

lP

Scenario number

of

Table 5. Evaluation results

0.85

-0.85

6

80

30

-0.07

0.85

-0.85

7

90

40

-0.07

0.85

-0.85

100

50

-0.07

ur na

8

0.85

-0.85

110

Jo

9

60

-0.07

0.85

9 6 7 8 8 7 7 8 9 6 7 8 10 15 16 19 20 15 16 20 20 13 14 14 16 17 18 23 25 18 18 23 23 17 19 21 21 20 21 31 33 19 20 25 27 18 18 19 21 19 21 25 27 19 21 26 27 21 22 25 25

20 18 22 17 13 10 23 19 22 18 25 19 28 22 26 25 28 27 27 25 23 23 23 20 32 28 29 27 30 28 -

0.1245 0.2312 0.0756 0.9802 0.7255 0.8478 0.0443 0.4624 0.1257 0.7983 0.0914 0.2071 0.0765 0.1446 0.4658 0.6720 0.1278 0.3173 0.0892 0.4150 0.1984 0.2984 0.3195 0.8664 0.0496 0.3620 0.1823 0.2941 0.0511 0.0905 -

of

-0.07

423400 639900 655200 662650 680500 628150 653000 656300 660250 565650 588200 646250 653250 3859500 4195400 4173950 4261350 3799500 4024300 4096050 4140100 3779000 3995000 4023750 4058150 4936300 5287400 5411700 5416150 4989000 5279900 5412050 5411700 4841200 5116400 5200800 5313200 6223700 6579050 6759450 6708300 6224400 6572400 6658800 6701350 6052300 6420000 6532050 6574100 7590500 8024700 8178850 8248800 7418800 7842900 8102350 8128400 7515300 7923550 8037300 8166750

ro

16

2.2523 0.6037 1.2923 3.2431 2.1658 0.3802 1.2170 2.3709 2.3853 0.2077 0.2032 0.9898 1.2434 0.9321 1.9250 2.9032 3.4929 0.8011 2.6434 3.4364 3.4313 0.4775 0.7462 0.9853 1.2516 0.5046 1.7697 3.4813 7.8982 1.0300 1.3867 3.9367 4.3105 0.9426 1.4206 1.9657 1.4244 1.3143 2.7692 4.2195 4.6458 0.9154 2.7203 3.0096 5.3841 0.6265 0.6442 0.7053 1.1707 0.9669 1.7196 3.7145 3.9564 0.5834 0.9419 3.7835 4.9012 0.9131 0.9801 1.9843 1.9951

-p

35

1175.600 1426.900 1442.100 1766.100 1604.100 1398.300 1398.300 1560.300 1560.300 1241.800 1403.800 1403.800 1565.800 2924.290 3091.500 3577.500 3739.500 2950.415 3120.400 3282.443 3444.443 2747.780 2915.496 2915.496 3239.496 3148.204 3475.900 4123.895 3961.900 3289.860 3486.600 3825.984 3961.000 3077.619 3254.000 3253.980 3416.000 3604.774 3804.500 4304.272 4790.272 3614.085 3808.400 4232.412 4394.400 3359.864 3698.700 3698.731 3860.700 3853.769 4074.200 4722.242 5046.242 3498.089 3701900 4349.237 4349.237 4100.960 4318.100 4331.715 4817.715

re

5

RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD

lP

-0.85

-0.85

10

120

120

Proposed ACO FFD RD Proposed ACO FFD RD Proposed ACO FFD RD

-0.07

0.85

4206.789 4435.900 5083.851 5731.851 3913.620 4131.900 4638.950 4962.950 4123.010 4353.500 4530.127 4854.127

1.0970 2.0805 3.9225 6.0860 0.9516 1.9199 2.3745 8.2882 1.0628 0.8718 1.8746 2.2734

9179500 9679850 9829750 9940050 9006000 9501900 9726500 9761850 9057200 9540850 9648850 9745050

22 23 30 34 20 22 28 33 22 23 31 33

31 27 25 25 33 25 -

0.1314 0.4942 0.2481 0.5029 0.1802 0.2817 -

of

The columns ONVP and SP in Table 5 confirm that the proposed algorithm made a good balance between exploration and exploitation in search space especially for larger search space.

ro

7.3 Time Complexity

Table 6. Parameter description

re

-p

To get time complexity of Hybrid meta-heuristic GA-based VMP presented in Algorithm 2, we use bring a the nomenclature illustrated in Table 1 to introduce input parameters relevant to procedures which are called in the main algorithm; then, time complexity can be stated based on input parameters because the algorithm treatment is directly dependable on these input parameters. Input parameters are tabulated in Table 6.

Description

PopSize

Population size

iteration

Number of iteration; it is user defined parameter

n

Number of requested VMs

m

Number of Physical machine in Datacenter

k

Number of items in Pareto Set (number of non-dominated solutions)

ur na

lP

Parameter

Jo

Firstly, we introduce time complexity of sub-procedures which are called in the main algorithm; then, we allege the time complexity related to the main proposed algorithm. Check&Corrct procedure takes n, m and PopSize as input; it can be simply calculated that its time complexity is O(𝑚2 × 𝑃𝑜𝑝𝑆𝑖𝑧𝑒) because of nested loops applied in the context. Time complexity of Exploitation procedure is the same as Check&Correct procedure. Time complexity of crossover and fitness functions are O(𝑚 + 𝑛) where n, m are known as input parameters. Mutation is very fast and simple; so, it belongs to O(1). Our Roulettewheel function such as in [1] has time complexity O(𝑃𝑜𝑝𝑆𝑖𝑧𝑒). If k is the number of current Pareto set members, the time complexity for nondominated (.) function is O(𝑘 2 ). Finally, the time complexity for proposed GA-based solution is O(𝑖𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 × 𝑚2 × 𝑃𝑜𝑝𝑆𝑖𝑧𝑒 + 𝑚2 × 𝑃𝑜𝑝𝑆𝑖𝑧𝑒 + 𝑘 2 ) by tracking counting execution time of its statements. As parameter k is smaller than PopSize in practice, it is simplified in O(𝑖𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 × 𝑚2 × 𝑃𝑜𝑝𝑆𝑖𝑧𝑒 + 𝑚2 × 𝑃𝑜𝑝𝑆𝑖𝑧𝑒 2 ).

8. Conclusion and future direction

re

-p

ro

of

In this paper, a new combinatorial virtual machine placement problem has been modeled and formulated. This multi-objective optimization model not only focuses on energy savings, but also pays attention to reduction in resource wastage and decline in high rate of data transfer over DCs’ shared network bandwidth. Because there are several parallel projects such as Hadoop MapReduce in which it needs many VMs with high rate of data dependency to figure out the complicated projects; therefore, the naïve algorithm leads throttling for communication-intensive applications the reason that our paper places VMs in physical vicinity so less traffic is sent on network layers. To solve the combinatorial multi-objective problem, a hybrid evolutionary algorithm has been proposed. The algorithm has been validated by extensive scenarios and experiments. Also, the performance of proposed algorithm has been compared with a state-of-theart ACO-based, heuristic-based FFD, a random-based approach. The result of simulations proves that proposed hybrid algorithm significantly outperforms against aforementioned approaches in terms of power saving, resource wastage reduction, and total data transfer rate on shared channels in DCs. Also, the simulations in larger search space demonstrated proposed approach has high potential of scalability. For future work, we intend to extend exact VMP by studying VMs’ behavior along with articulating a comprehensive energy model, which considers all energy consumer elements in datacenter such as all network devices. Then, it smartly places VMs based on their behavior and system capabilities.

lP

Conflict_of_interest

Jo

ur na

There is no conflict of interest in authors

References

Jo

ur na

lP

re

-p

ro

of

[1] Hosseini Shirvani, M., Rahmani, A. M., & Sahafi, A. (2018). An iterative mathematical decision model for cloud migration: A cost and security risk approach. Software: Practice and Experience, 48(3), 449-485. https://doi.org/10.1002/spe.2528. [2] Gao, Y., Guan, H., Qi, Z., Hou, Y., & Liu, L. (2013). A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. Journal of Computer and System Sciences, 79(8), 1230-1242. https://doi.org/10.1016/j.jcss.2013.02.004. [3] Randles, M., Lamb, D., Odat, E., & Taleb-Bendiab, A. (2011). Distributed redundancy and robustness in complex systems. Journal of Computer and System Sciences, 77(2), 293-304. https://doi.org/10.1016/j.jcss.2010.01.008. [4] Bilal, K., Khan, S. U., Kolodziej, J., Zhang, L., Hayat, K., Madani, S. A., ... & Chen, D. (2012, May). A Comparative Study Of Data Center Network Architectures. In ECMS (pp. 526-532). http://dx.doi.org/10.7148/2012-0526-0532. [5] Tian, H., Wu, J., & Shen, H. (2017, December). Efficient algorithms for VM placement in cloud data centers. In 2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT) (pp. 75-80). IEEE. https://doi.org/10.1109/PDCAT.2017.00021. [6] Hosseini Shirvani, M., Rahmani, A. M., & Sahafi, A. (2018). A survey study on virtual machine migration and server consolidation techniques in DVFS-enabled cloud datacenter: taxonomy and challenges. Journal of King Saud University-Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2018.07.001. [7] Shehabi, A., Smith, S., Sartor, D., Brown, R., Herrlin, M., Koomey, J., ... & Lintner, W. (2016). United states data center energy usage report. https://escholarship.org/content/qt84p772fc/qt84p772fc.pdf. [8] Kliazovich, D., Bouvry, P., & Khan, S. U. (2012). GreenCloud: a packet-level simulator of energyaware cloud computing data centers. The Journal of Supercomputing, 62(3), 1263-1283. https://doi.org/10.1007/s11227-010-0504-1. [9] Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., . & Sengupta, S. (2009, August). VL2: a scalable and flexible data center network. In ACM SIGCOMM computer communication review (Vol. 39, No. 4, pp. 51-62). ACM. https://doi.org/10.1145/1592568.1592576. [10] Cao, G. (2019). Topology-aware multi-objective virtual machine dynamic consolidation for cloud datacenter. Sustainable Computing: Informatics and Systems, 21, 179-188. https://doi.org/10.1016/j.suscom.2019.01.004. [11] Gopu, A., & Venkataraman, N. (2015). Optimal VM placement in distributed cloud environment using MOEA/D. Soft Computing, 1-20. https://doi.org/10.1007/s00500-018-03686-6. [12] Krzywda, J., Ali-Eldin, A., Carlson, T. E., Östberg, P. O., & Elmroth, E. (2018). Powerperformance tradeoffs in data center servers: DVFS, CPU pinning, horizontal, and vertical scaling. Future Generation Computer Systems, 81, 114-128. https://doi.org/10.1016/j.future.2017.10.044. [13] Rizvandi, N. B., Taheri, J., & Zomaya, A. Y. (2011). Some observations on optimal frequency selection in DVFS-based energy consumption minimization. Journal of Parallel and Distributed Computing, 71(8), 1154-1164. https://doi.org/10.1016/j.jpdc.2011.01.004. [14] Stavrinides, G. L., & Karatza, H. D. (2019). An energy-efficient, QoS-aware and cost-effective scheduling approach for real-time workflow applications in cloud computing systems utilizing DVFS

Jo

ur na

lP

re

-p

ro

of

and approximate computations. Future Generation Computer Systems, 96, 216-226. https://doi.org/10.1016/j.future.2019.02.019. [15] Safari, M., & Khorsand, R. (2018). Energy-aware scheduling algorithm for time-constrained workflow tasks in DVFS-enabled cloud environment. Simulation Modelling Practice and Theory, 87, 311-326. https://doi.org/10.1016/j.simpat.2018.07.006. [16] Wu, T., Gu, H., Zhou, J., Wei, T., Liu, X., & Chen, M. (2018). Soft error-aware energy-efficient task scheduling for workflow applications in DVFS-enabled cloud. Journal of Systems Architecture, 84, 12-27. https://doi.org/10.1016/j.sysarc.2018.03.001. [17] Wu, C. M., Chang, R. S., & Chan, H. Y. (2014). A green energy-efficient scheduling algorithm using the DVFS technique for cloud datacenters. Future Generation Computer Systems, 37, 141-147. https://doi.org/10.1016/j.future.2013.06.009. [18] Guérout, T., Monteil, T., Da Costa, G., Calheiros, R. N., Buyya, R., & Alexandru, M. (2013). Energy-aware simulation with DVFS. Simulation Modelling Practice and Theory, 39, 76-91. https://doi.org/10.1016/j.simpat.2013.04.007. [19] De Courchelle, I., Guérout, T., Da Costa, G., Monteil, T., & Labit, Y. (2019). Green energy efficient scheduling management. Simulation Modelling Practice and Theory, 93, 208-232. https://doi.org/10.1016/j.simpat.2018.09.011. [20] Ilic, M. D. (2010). Dynamic monitoring and decision systems for enabling sustainable energy services. Proceedings of the IEEE, 99(1), 58-79. https://doi.org/10.1109/JPROC.2010.2089478. [21] Garg, S. K., Yeo, C. S., & Buyya, R. (2011, August). Green cloud framework for improving carbon efficiency of clouds. In European Conference on Parallel Processing (pp. 491-502). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23400-2_45. [22] Bhattacherjee, S., Das, R., Khatua, S., & Roy, S. (2019). Energy-efficient migration techniques for cloud environment: a step toward green computing. The Journal of Supercomputing, 1-29. https://doi.org/10.1007/s11227-019-02801-0. [23] Khosravi, A., Garg, S. K., & Buyya, R. (2013, August). Energy and carbon-efficient placement of virtual machines in distributed cloud data centers. In European Conference on Parallel Processing (pp. 317-328). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40047-6_33. [24] Usman, M. J., Ismail, A. S., Chizari, H., Abdul-Salaam, G., Usman, A. M., Gital, A. Y., ... & Aliyu, A. (2019). Energy-efficient Virtual Machine Allocation Technique Using Flower Pollination Algorithm in Cloud Datacenter: A Panacea to Green Computing. Journal of Bionic Engineering, 16(2), 354-366. https://doi.org/10.1007/s42235-019-0030-7. [25] Jing, S. Y., Ali, S., She, K., & Zhong, Y. (2013). State-of-the-art research study for green cloud computing. The Journal of Supercomputing, 65(1), 445-468. https://doi.org/10.1007/s11227-0110722-1. [26] Gu, C., Fan, L., Wu, W., Huang, H., & Jia, X. (2018). Greening cloud data centers in an economical way by energy trading with power grid. Future Generation Computer Systems, 78, 89101. https://doi.org/10.1016/j.future.2016.12.029. [27] Lin, J. W., Chen, C. H., & Lin, C. Y. (2014). Integrating QoS awareness with virtualization in cloud computing systems for delay-sensitive applications. Future Generation Computer Systems, 37, 478-487. https://doi.org/10.1016/j.future.2013.12.034. [28] Chaisiri, S., Lee, B. S., & Niyato, D. (2009, December). Optimal virtual machine placement across multiple cloud providers. In 2009 IEEE Asia-Pacific Services Computing Conference (APSCC) (pp. 103-110). IEEE. https://doi.org/10.1109/APSCC.2009.5394134. [29] Bichler, M., Setzer, T., & Speitkamp, B. (2006). Capacity planning for virtualized servers. In Workshop on Information Technologies and Systems (WITS), Milwaukee, Wisconsin, USA. https://ssrn.com/abstract=1025862.

Jo

ur na

lP

re

-p

ro

of

[30] Speitkamp, B., & Bichler, M. (2010). A mathematical programming approach for server consolidation problems in virtualized data centers. IEEE Transactions on services computing, 3(4), 266-278. https://doi.org/10.1109/TSC.2010.25. [31] Van, H. N., Tran, F. D., & Menaud, J. M. (2010, July). Performance and power management for cloud infrastructures. In 2010 IEEE 3rd international Conference on Cloud Computing (pp. 329336). IEEE. https://doi.org/10.1109/CLOUD.2010.25. [32] Hermenier, F., Lorca, X., Menaud, J. M., Muller, G., & Lawall, J. (2009, March). Entropy: a consolidation manager for clusters. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (pp. 41-50). ACM. https://doi.org/10.1145/1508293.1508300. [33] Yue, M. (1991). A simple proof of the inequality FFD (L)≤ 11/9 OPT (L)+ 1,∀ L for the FFD bin-packing algorithm. Acta mathematicae applicatae sinica, 7(4), 321-331. https://doi.org/10.1007/BF02009683. [34] Grit, L., Irwin, D., Yumerefendi, A., & Chase, J. (2006, November). Virtual machine hosting for networked clusters: Building the foundations for autonomic orchestration. In Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing (p. 7). IEEE Computer Society. https://doi.org/10.1109/VTDC.2006.17. [35] Baker, B. S. (1985). A new proof for the first-fit decreasing bin-packing algorithm. Journal of Algorithms, 6(1), 49-70. https://doi.org/10.1016/0196-6774(85)90018-5. [36] Kao, M. Y. (Ed.). (2008). Encyclopedia of algorithms. Springer Science & Business Media. ISBN: 978-0-387-30162-4. [37] Cardosa, M., Korupolu, M. R., & Singh, A. (2009, June). Shares and utilities based power consolidation in virtualized server environments. In 2009 IFIP/IEEE International Symposium on Integrated Network Management (pp. 327-334). IEEE. https://doi.org/10.1109/INM.2009.5188832. [38] Beloglazov, A., Abawajy, J., & Buyya, R. (2012). Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future generation computer systems, 28(5), 755-768. https://doi.org/10.1016/j.future.2011.04.017. [39] Leinberger, W., Karypis, G., & Kumar, V. (1999, September). Multi-capacity bin packing algorithms with applications to job scheduling under multiple constraints. In Proceedings of the 1999 International Conference on Parallel Processing (pp. 404-412). IEEE. https://doi.org/10.1109/ICPP.1999.797428. [40] Coffman Jr, E. G., Csirik, J., Galambos, G., Martello, S., & Vigo, D. (2013). Bin packing approximation algorithms: survey and classification. Handbook of combinatorial optimization, 455531. https://doi.org/10.1007/978-1-4419-7997-1_35. [41] Feller, E. (2012). Autonomic and energy-efficient management of large-scale virtualized data centers (Doctoral dissertation). https://tel.archives-ouvertes.fr/tel-00785090. [42] Festa, P. (2014, July). A brief introduction to exact, approximation, and heuristic algorithms for solving hard combinatorial optimization problems. In 2014 16th International Conference on Transparent Optical Networks (ICTON) (pp. 1-20). IEEE. https://doi.org/10.1109/ICTON.2014.6876285. [43] Mi, H., Wang, H., Yin, G., Zhou, Y., Shi, D., & Yuan, L. (2010, July). Online selfreconfiguration with performance guarantee for energy-efficient large-scale cloud computing data centers. In 2010 IEEE International Conference on Services Computing (pp. 514-521). IEEE. https://doi.org/10.1109/SCC.2010.69. [44] Feller, E., Rilling, L., & Morin, C. (2011, September). Energy-aware ant colony based workload placement in clouds. In Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing (pp. 26-33). IEEE Computer Society. https://doi.org/10.1109/Grid.2011.13.

Jo

ur na

lP

re

-p

ro

of

[45] Jeyarani, R., Nagaveni, N., & Ram, R. V. (2013). Self adaptive particle swarm optimization for efficient virtual machine provisioning in cloud. In Organizational Efficiency through Intelligent Information Technologies (pp. 88-107). IGI Global. https://doi.org/10.4018/978-1-4666-20476.ch006. [46] Jeyarani, R., Nagaveni, N., & Ram, R. V. (2012). Design and implementation of adaptive poweraware virtual machine provisioner (APA-VMP) using swarm intelligence. Future Generation Computer Systems, 28(5), 811-821. https://doi.org/10.1016/j.future.2011.06.002. [47] Liao, X., Jin, H., & Liu, H. (2012). Towards a green cluster through dynamic remapping of virtual machines. Future Generation Computer Systems, 28(2), 469-477. https://doi.org/10.1016/j.future.2011.04.013. [48] Malekloo, M. H., Kara, N., & El Barachi, M. (2018). An energy efficient and SLA compliant approach for resource allocation and consolidation in cloud computing environments. Sustainable Computing: Informatics and Systems, 17, 9-24. https://doi.org/10.1016/j.suscom.2018.02.001. [49] Ilkhechi, A. R., Korpeoglu, I., & Ulusoy, Ö. (2015). Network-aware virtual machine placement in cloud data centers with multiple traffic-intensive components. Computer Networks, 91, 508-527. https://doi.org/10.1016/j.comnet.2015.08.042. [50] Shabeera, T. P., Kumar, S. M., Salam, S. M., & Krishnan, K. M. (2017). Optimizing VM allocation and data placement for data-intensive applications in cloud using ACO metaheuristic algorithm. Engineering Science and Technology, an International Journal, 20(2), 616-628. https://doi.org/10.1016/j.jestch.2016.11.006. [51] Fang, W., Liang, X., Li, S., Chiaraviglio, L., & Xiong, N. (2013). VMPlanner: Optimizing virtual machine placement and traffic flow routing to reduce network power costs in cloud data centers. Computer Networks, 57(1), 179-196. https://doi.org/10.1016/j.comnet.2012.09.008. [52] Heller, B., Seetharaman, S., Mahadevan, P., Yiakoumis, Y., Sharma, P., Banerjee, S., & McKeown, N. (2010, April). Elastictree: Saving energy in data center networks. In Nsdi (Vol. 10, pp. 249-264). https://www.usenix.org/legacy/event/nsdi10/tech/full_papers/heller.pdf. [53] Meng, X., Pappas, V., & Zhang, L. (2010, March). Improving the scalability of data center networks with traffic-aware virtual machine placement. In 2010 Proceedings IEEE INFOCOM (pp. 1-9). IEEE. www.cmlab.csie.ntu.edu.tw/~freetempo/files/Traffic-aware_VM_Placement.pdf. [54] Guo, C., Lu, G., Li, D., Wu, H., Zhang, X., Shi, Y., ... & Lu, S. (2009). BCube: a high performance, server-centric network architecture for modular data centers. ACM SIGCOMM Computer Communication Review, 39(4), 63-74. https://doi.org/10.1145/1594977.1592577. [55] DONG, J. K., WANG, H. B., LI, Y. Y., & CHENG, S. D. (2014). Virtual machine placement optimizing to improve network performance in cloud data centers. The Journal of China Universities of Posts and Telecommunications, 21(3), 62-70. https://doi.org/10.1016/S1005-8885(14)60302-2. [56] Piao, J. T., & Yan, J. (2010, November). A network-aware virtual machine placement and migration approach in cloud computing. In 2010 Ninth International Conference on Grid and Cloud Computing (pp. 87-92). IEEE. https://doi.org/10.1109/GCC.2010.29 [57] Roh, H., Jung, C., Kim, K., Pack, S., & Lee, W. (2017). Joint flow and virtual machine placement in hybrid cloud data centers. Journal of Network and Computer Applications, 85, 4-13. https://doi.org/10.1016/j.jnca.2016.12.006. [58] Xu, J., & Fortes, J. A. (2010, December). Multi-objective virtual machine placement in virtualized data center environments. In 2010 IEEE/ACM Int'l Conference on Green Computing and Communications & Int'l Conference on Cyber, Physical and Social Computing (pp. 179-188). IEEE. https://doi.org/10.1109/GreenCom-CPSCom.2010.137.

Jo

ur na

lP

re

-p

ro

of

[59] Ajiro, Y., & Tanaka, A. (2007, December). Improving packing algorithms for server consolidation. In int. CMG Conference (Vol. 253, pp. 399-406). https://pdfs.semanticscholar.org/3cd9/5539f604af450a8b81b82aed186467f22257.pdf. [60] Xiao, Z., Jiang, J., Zhu, Y., Ming, Z., Zhong, S., & Cai, S. (2015). A solution of dynamic VMs placement problem for energy consumption optimization based on evolutionary game theory. Journal of Systems and Software, 101, 260-272. https://doi.org/10.1016/j.jss.2014.12.030. [61] Shaw, R., Howley, E., & Barrett, E. (2019). An energy efficient anti-correlated virtual machine placement algorithm using resource usage predictions. Simulation Modelling Practice and Theory, 93, 322-342. https://doi.org/10.1016/j.simpat.2018.09.019. [62] Tordsson, J., Montero, R. S., Moreno-Vozmediano, R., & Llorente, I. M. (2012). Cloud brokering mechanisms for optimized placement of virtual machines across multiple providers. Future generation computer systems, 28(2), 358-367. https://doi.org/10.1016/j.future.2011.07.003. [63] Zhang, J., Wang, X., Huang, H., & Chen, S. (2017). Clustering based virtual machines placement in distributed cloud computing. Future Generation Computer Systems, 66, 1-10. https://doi.org/10.1016/j.future.2016.06.018. [64] Reddy, V. D., Gangadharan, G. R., & Rao, G. S. V. (2019). Energy-aware virtual machine allocation and selection in cloud data centers. Soft Computing, 23(6), 1917-1932. https://doi.org/10.1007/s00500-017-2905-z. [65] Wu, G., Tang, M., Tian, Y. C., & Li, W. (2012, November). Energy-efficient virtual machine placement in data centers by genetic algorithm. In International conference on neural information processing (pp. 315-323). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-344879_39. [66] Kumar, D., & Raza, Z. (2015, February). A PSO based VM resource scheduling model for cloud computing. In 2015 IEEE International Conference on Computational Intelligence & Communication Technology (pp. 213-219). IEEE. https://doi.org/10.1109/CICT.2015.35. [67] Dashti, S. E., & Rahmani, A. M. (2016). Dynamic VMs placement for energy efficiency by PSO in cloud computing. Journal of Experimental & Theoretical Artificial Intelligence, 28(1-2), 97-112. https://doi.org/10.1080/0952813X.2015.1020519. [68] Mishra, S. K., Puthal, D., Sahoo, B., Jayaraman, P. P., Jun, S., Zomaya, A. Y., & Ranjan, R. (2018). Energy-efficient VM-placement in cloud data center. Sustainable computing: informatics and systems, 20, 48-55. https://doi.org/10.1016/j.suscom.2018.01.002. [69] Vakilinia, S., Heidarpour, B., & Cheriet, M. (2016). Energy efficient resource allocation in cloud computing environments. IEEE Access, 4, 8544-8557. https://doi.org/10.1109/ACCESS.2016.2633558. [70] Pires, F. L., & Barán, B. (2013, December). Multi-objective virtual machine placement with service level agreement: A memetic algorithm approach. In Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing (pp. 203-210). IEEE Computer Society. https://doi.org/10.1109/UCC.2013.44. [71] Kaaouache, M. A., & Bouamama, S. (2015). Solving bin packing problem with a hybrid genetic algorithm for VM placement in cloud. Procedia Computer Science, 60, 1061-1069. https://doi.org/10.1016/j.procs.2015.08.151. [72] Niranjan Mysore, R., Pamboris, A., Farrington, N., Huang, N., Miri, P., Radhakrishnan, S., ... & Vahdat, A. (2009, August). Portland: a scalable fault-tolerant layer 2 data center network fabric. In ACM SIGCOMM Computer Communication Review (Vol. 39, No. 4, pp. 39-50). ACM. https://doi.org/10.1145/1592568.1592575. [73] Deb, K. (2001). Multi-objective optimization using evolutionary algorithms (Vol. 16). John Wiley & Sons. ISBN: 978-0-471-87339-6.

[74] Fan, X., Weber, W. D., & Barroso, L. A. (2007, June). Power provisioning for a warehousesized computer. In ACM SIGARCH computer architecture news (Vol. 35, No. 2, pp. 13-23). ACM. https://doi.org/10.1145/1273440.1250665.

[75] Jian-ping, L., Li, X., Min-rong, C. (2014). Hybrid shuffled frog leaping algorithm for energy-efficient dynamic consolidation of virtual machines in cloud data centers. Expert Syst. Appl. 41 (13), 5804–5816.

of

[76] Hosseini Shirvani, M., Gorji, A. B. (2019). Optimization of Automatic web services composition using genetic algorithm. International journal of cloud computing, 2019 (8)(1). Accepted, In press. [77] Hosseini Shirvani, M. (2018, July). Web Service Composition in multi-cloud environment: A bi-objective genetic optimization algorithm. In 2018 Innovations in Intelligent Systems and Applications (INISTA) (pp. 1-6). IEEE. https://doi.org/10.1109/INISTA.2018.8466267. [78] Van Veldhuizen, D. A. (1999). Multiobjective evolutionary algorithms: classifications, analyses, and new innovations (No. AFIT/DS/ENG/99-01). AIR FORCE INST OF TECH WRIGHT-PATTERSONAFB OH SCHOOL OF ENGINEERING.

re

-p

ro

[79] Schott, J. R. (1995). Fault tolerant design using single and multicriteria genetic algorithm optimization (No. AFIT/CI/CIA-95-039). AIR FORCE INST OF TECH WRIGHT-PATTERSON AFB OH

Authors Biography

ur na

lP

Sara Farzai received her B.Sc. and M.Sc. degrees in Computer Engineering-Software from IAU Sari-branch (Iran) in 2011 and 2013 respectively. She is Ph.D candidate in Software Systems in IAU Sari-branch, Iran, and also she is an instructor in department of Computer Engineering, Maziar Higher Education Institute, Royan, Iran. Her research interests include cloud and fog computing, and software quality analysis.

Jo

Mirsaeid Hosseini Shirvani received his B.Sc., M.Sc., and PhD all in Computer Software Engineering Systems at Universities in Tehran, IRAN. He has been teaching miscellaneous computer courses in several universities in Mazandaran province since 2001. He also published several papers in authentic journals. Currently, he is Assistant Professor in Computer Engineering Department at IAU (Sari-Branch). His research interests are in the areas of cloud- and fog- computing, IoT, distributed systems, parallel processing and evolutionary computing.

Jo

ur na

lP

re

-p

ro

of

Mohsen Rabbani was born in 1967 in Iran. He received the B.S degree in applied mathematics (operation research) from the Tehran University in 1991, the M.S degree in applied mathematics (Numerical analysis) from Iran university science & technology (I.U.S.T), Tehran, Iran, in 1998 and Ph.D. degree in applied mathematics (Integral equations) from I.U.S.T , Tehran, Iran , in 2008. Now, he is an Associate Professor in department of applied mathematics in I.A university of Iran. His favorite fields are integral equations, wavelet, numerical solution of ordinary and partial differential equations.