Traffic matrix estimation with software-defined NFV: Challenges and opportunities

Accepted Manuscript Title: Traffic matrix estimation with Software-Defined NFV: challenges and opportunities Author: Ugo Fiore Paolo Zanetti Francesco Palmieri Francesca Perla PII: DOI: Reference:

S1877-7503(17)30240-5 http://dx.doi.org/doi:10.1016/j.jocs.2017.03.001 JOCS 623

To appear in: Received date: Revised date: Accepted date:

4-11-2016 19-2-2017 1-3-2017

Please cite this article as: Ugo Fiore, Paolo Zanetti, Francesco Palmieri, Francesca Perla, Traffic matrix estimation with Software-Defined NFV: challenges and opportunities, (2017), http://dx.doi.org/10.1016/j.jocs.2017.03.001 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

•

Traffic matrix estimation from link-level measurements is a central issue in modeling complex networking systems.

• In modern networks, the offloading many functions to a cloud, proactive caching, dynamic network reconfiguration, may boost in the ability of controlling traffic and the stringent need to protect assets and guarantee isolation are among the factors that will affect traffic matrices

ip t

cr us an M d te ce p

•

We listed the principal transformations induced by virtualization technologies NFV and SDN and their potential effect for Traffic matrix estimation, the problem is analyzed from several points of view, including modeling engineering and econometric perspectives.

Ac

•

Page 1 of 31

ip t

Traffic matrix estimation with Software-Defined NFV: challenges and opportunities Ugo Fiorea,∗, Paolo Zanettib , Francesco Palmieric , Francesca Perlab a Department

of Molecular Medicine and Medical Biotechnologies, Federico II University of Business and Quantitative Studies, Parthenope University c Department of Computer Science, University of Salerno

us

cr

b Department

Abstract

an

Traffic matrices, abstract representations of demand, are essential for network operators endeavoring to model, measure, maintain, and improve the efficiency of their complex and heterogeneous architectures. Traffic matrix estimation

M

consists in inferring a traffic matrix from link-level measurements. Provoked by the need to enable agile deployment of new services while, at the same time, slashing operating expenditure and energy consumption, the trend in telecom-

d

munications is to shift functionality from physical appliances to virtualized ser-

te

vices. We analyze the effects of this landscape change on traffic matrices, their dynamics, and their estimation, indicating some new challenges and problems

Ac ce p

that will arise in all the associated modeling, analysis and evaluation activities. Keywords: traffic matrix, network function virtualization, complex systems, virtual network embedding, modeling

1. Introduction

In an increasingly network-centric connected world, knowledge of traffic is

essential for network operators that need to effectively and efficiently satisfy current demands, as well as prepare for future needs, adjusting the architecture

5

of their complex networks and planning the procurement of additional capacity. Entries of traffic matrices represent the volume of traffic flowing between ∗ Corresponding

author Email address: [email protected] (Ugo Fiore)

Preprint submitted to Journal of Computational Science

February 19, 2017

Page 2 of 31

an origin and a destination, abstract network entities (usually nodes or sets thereof), hence, an accurate estimation of traffic matrices is a prerequisite of

10

ip t

paramount importance for reliably modeling complex network organizations for both simulation and traffic engineering purposes. An important feature of traf-

cr

fic matrices is that, as an abstract representation of consumer demand, they are

expected to be largely invariant with respect to network topology, in the sense

us

that they should be insensitive to small topological changes that do not cause significant diversions in network paths [1]. Conversely, the topological structure 15

of a network and its routing policies can be expected to have been tailored to the

an

traffic matrix. From the point of view of an operator that builds and operates its network, the guiding principle is to carry in the most efficient way the traffic that is (or is forecast) to be serviced. The anticipated traffic matrix is therefore

20

M

supposed to have determined and oriented many, if not all, of the decision in the design of advanced network architectures. The same is true in general for all complex, large-scale communication networks.

d

Traffic can be described at various levels of aggregation, and between logical

te

or physical endpoints. The simplest and straightforward approach for building a traffic matrix is direct measurement, by reliably collecting flow-level statistics at the network ingress and egress points. However, direct measurement of point-to-

Ac ce p

25

point traffic flows, while technically feasible in principle, requires the deployment of complex and expensive equipment at the infrastructure level and may affect forwarding performance on the involved nodes. Consequently, this is seldom implemented in practice due to the performance degradation it would cause.

30

Estimating the traffic matrix by starting on some available link-level statistics and leveraging reliable traffic models will be a much better solution. The key insight of traffic matrix estimation (TME) is that a traffic matrix

can be inferred from link-level measurements, which are readily available to network operators [2]. If the traffic matrix were known, it would determine the link 35

counts through the routing matrix, which describes the utilization of links by Origin-Destination (OD) flows. The inverse inference (sometimes known as deconvolution) is difficult, because the number of links is usually much lower than 2

Page 3 of 31

the number of OD pairs, resulting in an ill-posed, highly undetermined, system of equations. Estimation techniques include gravity models, network tomography, and hybrid methods (Section 3). Models such as the gravity one are based

ip t

40

on the assumption of independence between the volumes of traffic entering or

cr

exiting a network at each of its endpoints. Such an assumption has been shown

to not hold in practice for Internet traffic, according to several considerations

45

us

[3]. These include the connection-based, asymmetric nature of protocols such as HTTP, and the presence, at an Internet-wide scale, of content-heavy Internet Service Providers (ISPs) and access-heavy (Eyeball) ISPs with different traffic

an

profiles [4]. Changing trends in consumer demands, such as online video coupled with the emergence of Content Distribution Networks (CDNs), cloud services, and the shift of prevalence from wired traffic to mobile, also produce effects that should be accounted for in adapting existing models and designing new ones.

M

50

Recently, a new paradigm is emerging in the telecommunications industry. Motivated by the need to cut on operating expenditure and energy consump-

d

tion, operators consider moving functionality from physical network appliances

55

te

to virtualized services. The proprietary nature and diversity of purpose-built middle-boxes that operators used to rely upon have been hampering the flex-

Ac ce p

ibility to adapt to quickly-changing demand, also making deployment of new services slow and costly (network ossification). With Network Function Virtualization (NFV) [5], which leverages upon virtualization technology, the functions realized by middle-boxes are implemented by software subsystems running

60

on commodity hardware, allowing a reduction in costs along with agile provisioning, deployment, and centralized management of virtual network functions. A virtual network function1 is the software instance in NFV that consists of one or more Virtual Machines (VMs) running different processes supporting a network function. Virtual network functions can be instantiated when needed,

65

using VMs, without having to install new hardware. In synthesis, functionality 1 The

acronym VNF is sometimes used. We preferred not to adopt that convention in order

to avoid possible confusion with NFV.

3

Page 4 of 31

is thus decoupled from location, and software is decoupled from the hardware platform. Operators get the ability to rapidly introduce—and dismiss—highly

ip t

tailored services while reaping the benefits of reduction in costs due to consolidation of hardware.

Latency and throughput, and especially the stability of these performance

cr

70

indicators, are issues that have often been raised with NFV. Certainly, NFV

us

would be losing much of its attractiveness, had its adoption brought a slump in performance respect to the one achievable with specialized, purpose-built, hardware. Another major aspect that needs careful consideration by operators is how to smoothly migrate from their existing legacy infrastructure to an NFV-enabled

an

75

one. In addition, the possibility of instantiating virtual appliances dynamically, and placing them optimally, requires the network infrastructure to support this

M

ability. These reflections suggest the integration of NFV with Software-Defined Network (SDN), an emerging framework focused on decoupling the network 80

control plane from the data plane, providing programmable connectivity and

d

centralized control. SDN complements NFV by providing traffic engineering

te

and flexible connectivity between virtual network functions, while NFV can serve SDN by virtualizing the SDN controller (which can be regarded as a net-

Ac ce p

work function) to run on cloud, thus allowing dynamic migration of controllers 85

to optimal locations.

Besides the objectives detailed above, ISPs might leverage upon SDN and

NFV to transfer some traffic off their network, and in general to increase their control over the inbound and outbound traffic volumes exchanged at interconnection points. In the complex context of peering agreements, traffic ratios

90

(between the volumes of incoming and outgoing traffic) are of the essence when determining whether settlement-free peering agreements are viable or paid peering is to be sought [6]. Imbalances between incoming and outgoing traffic has, in fact, led to sharp contrasts between interconnected ISPs. The economic effects of changes in traffic volumes may be felt at two different timescales. When pro-

95

visioning, operators may need to augment the capacity of their network to meet a raising quantity of traffic, while traffic declines would induce them to devise 4

Page 5 of 31

ways to prevent the resulting unused (but paid for) capacity from turning into a loss. On a faster timescale, congestion due to increments in traffic volume will

100

ip t

drive operating and management costs up. In this framework, operators may

wish to exert as much control as possible over their traffic. A relevant part can

cr

be played by the movement of content caches to locations closer to end users, so as to reduce the associated inbound traffic.

us

A factor which has to be accounted for is how users will respond to pricing changes, and to changes in the cost of forwarding. Pricing models may need 105

revision, in order to reflect the changes brought about by NFV. Finally, new

an

paradigms emerging in the Internet scenario, such as next generation mobile communication systems (5G) or fog computing [7] may also have an impact on the estimation of traffic matrices. All these factors produce interesting effects

110

M

on the estimation of traffic matrices, which we describe and analyze. After a review of related work in Sec. 2, some detail about the traffic matrix, the problem of its estimation, and an outline of NFV, SDN, and peering practices

d

are given in Sec. 3. The effects of SDN and NFV are discussed in Sec. 5.

te

Conclusions, along with directions for future work, are reported in Sec. 6.

Ac ce p

2. Related Work 115

The TME problem has been studied in the literature in two main different

flavors, according to the considered time scale: for long time intervals, a purely spatial, static notion, modeling the average traffic demand, and a temporal sequence of values, describing the dynamical evolution of traffic over short time frames.

120

However, a limited number of techniques are available, until now, for the

traffic matrix estimation in large-scale network infrastructures. Since the TME problem is characterized by a set of linear relationships, the basic problem of maximizing the total traffic routed throughout the network, constrained by the trivial flow conservation criterion, can be easily formulated

125

using a Linear Programming model and standard techniques can be used to

5

Page 6 of 31

solve it. This problem was first formulated as such in [8]. Another approach, proposed in [9], relies on Bayesian methods to determine

ip t

the conditional probability distributions for elements in the traffic matrix, given the observed link loads. This approach typically uses Markov-chain Monte Carlo simulation techniques for calculating the posterior probabilities needed.

cr

130

Alternatively, the Expectation Maximization (EM) approach [10] is able to

us

consider multiple sets of link-level measurements and is based on an expectation maximization algorithm for calculating the maximum likelihood estimate for the traffic matrix based on the link loads.

Time series traffic matrices show both spatial and temporal correlation, and

an

135

good models have exploited both [11]. Methods that tackle the dimension of the traffic estimation search space by leveraging upon their spatio-temporal

estimation of TM elements [13]. 140

M

structure (e.g. [12]) often have high complexity, due to simultaneous constrained

When particular network environments are considered, their specific charac-

d

teristics affect traffic matrices and their estimation in subtle ways. At a global

te

scale, the traffic exchanged between pairs of Autonomous Systems (ASes) is elusive, due to the reluctance of ASes to reveal data that might be useful to

Ac ce p

their competitors, motivating researchers to adopt model-based approaches [14] 145

and look for surrogate measurements that could be used to obtain reasonable estimates for AS-level traffic matrices [15]. A wide survey of research about Software Defined Optical Networks (SDONs), outlining ten directions for further study, can be found in [16]. The estimation of traffic flows in data center networks (DCNs) has been undertaken by Hu et al. [13]. In DCN, there can be

150

many different, redundant paths between node pairs, and the choice of a specific path can be determined by the need to avoid congestion. By decomposing the DCN topology into several clusters based on local structure, the complexity of the inference problem is reduced, and intra-cluster as well as inter-cluster traffic can be estimated. A different approach to the complexity of the estima-

155

tion problem was proposed by Airoldi and Blocker [17], who explicitly modeled traffic dynamics, so as to constrain the solution space to be explored. Basi6

Page 7 of 31

cally, the projection of aggregate traffic volumes onto the latent space of OD traffic is made under a given probabilistic model, i.e., the expectation of OD

160

ip t

traffic is estimated given the observed measurements and their distributions. The resulting multilevel model for traffic combines a heavy-tailed process, in

cr

which the amount of traffic on each OD route is proportional to its variability up to a scaling factor shared by all OD routes, and error process for better

us

capturing near-zero traffic volumes. The problem of sampling from the resulting highly constrained solution space is tackled via model-based regularization 165

and a sequential sample-importance-resample-move (SIRM) particle filter that

an

is statistically efficient, numerically stable, and scales well. One might expect that blind-source separation methods such as Independent Component Analysis (ICA) [18] might shed some light on the estimation of traffic matrices, since the

170

M

observed traffic volumes are the superposition of different OD flows. However, as pointed out by Airoldi and Blocker, strong correlation can be observed in

d

traffic volumes and ICA is of little use in this context [17].

te

3. Model

The fundamental concept behind to the generation of a traffic matrix is

Ac ce p

communication demand, representing the amount of traffic potentially flowing 175

between any pair (i, j) of network endpoints, i.e. entering the network at i and leaving it at j.

We are interested in a set V of nodes, labeled {1, . . . , n}. Suppose that two

n-vectors, d = (di ) and s = (si ) specify the overall demand and supply, i.e. the aggregate amount of traffic exiting (resp., entering) node i. Assume that traffic P P is conserved, that is, i∈V di = i∈V si = X tot , with X tot denoting the overall amount of traffic in the network. All nodes act as source or sink. Denote as X = (xij ) the traffic matrix. Thus, xij is the amount of traffic that goes from node i to node j. Hence, X

xij = si and

j∈V

X

xij = dj

(1)

i∈V

. 7

Page 8 of 31

3.1. Gravity model Under the gravity model, the source and destination are assumed to be

ip t

independent [19]. Traffic between any two nodes is proportional to the total

traffic flowing out of the source node and to the total traffic going into the

xij =

cr

destination node. Entries of the traffic matrix have the form si dj fij

(2)

us

where the friction factors fij express the different conditions that may exist among node pairs. The form of (2) resembles Newton’s law of gravitation, where

an

the gravitational force between two bodies is proportional to the product of their

the fanouts are obtained: si pi = P

M

masses divided by their distance squared. In computer networks, friction factors si dj are usually taken as constant, and it is convenient to write (2) as xij = tot , X so that (1) will be verified. Normalizing the aggregate demands and supplies,

sk

dj and qj = P

d

k∈V

dk

(3)

k∈V

180

te

Fanouts have been shown to exhibit a more stationary behavior as compared to traffic volumes. By dividing traffic in classes and substituting the notion of

Ac ce p

independence with conditional independence, a generalized gravity model has been obtained [12] that outperforms the regular gravity model. Precisely the assumption from which the gravity model derives its principal

strength, i.e., the independence of source and destination nodes, has elicited

185

some criticism [3]. The session-oriented nature of TCP, for example, as well as the dropping of packets because of congestion at routers, induce observable dependency between nodes. In addition, traffic is not necessarily equal in both directions. Some applications, notably HTTP traffic, are inherently asymmetric, with small requests in one direction and large responses in the opposite direction.

190

A consequence is that forward and reverse traffic are not independent. While the reasoning above applies locally, to a single client-server pair, the presence of popular content providers and, on an Internet-wide scale, of Content ISPs and Eyeball ISPs [4], makes the dependence evident also at the aggregate level. 8

Page 9 of 31

Other factors that conflict with the gravity model include the effects of traffic 195

ratio thresholds in peering agreements [20], and asymmetries due to Hot-potato

ip t

routing. Peering is an interconnection whereby two networks agree to mutually

carry traffic destined to each other’s network, as opposed to Transit, a customer-

cr

supplier arrangement where an ISP buys from another ISP the connectivity to the global Internet. Peering is typically done at an Internet Exchange Point (IXP), which provides a shared switching fabric to enable the mutual inter-

us

200

connection of ISPs. Settlement-free peering involves no payments, while paid peering is a customer-provider business relationship where the customer pays

an

the provider in order to gain access to the networks of the provider and of its customers. Not much details are publicly known about paid peering, as such 205

contracts are often subject to nondisclosure agreements. Central to peering is

M

the notion of mutual benefit. On the assumption that the costs an ISP has to carry are proportional to the volume of traffic on its network, the ratio between inbound and outbound traffic has often been taken as a measure of fairness in

centiles on 5-minute traffic distributions (this is true in particular for Tier 1 ISPs [21]).

te

210

d

the sharing of costs in peering [20]. Volumes are usually taken as 95th per-

Ac ce p

When two ISPs peer, each of them has a selfish interest to limit the resources consumed to forward the traffic destined to the peer’s network. Each ISP has thus an incentive to force traffic into its destination network as quickly and

215

cheaply as possible. When there are multiple interconnection points (as it often is the case) packets are then delivered to the peer at the nearest peering interface, which might not be the optimal one from the point of view of the overall network. The expected deviation from the globally optimum routing (the cost of anarchy in game theory parlance) has been shown to be a factor of three [22].

220

3.2. Network tomography Network tomography infers the internal structure of a network using information derived from end nodes [1]. Active methods use probes, whereas passive techniques do not require network elements to perform additional actions other 9

Page 10 of 31

than the usual packet forwarding. The phrase network tomography was coined 225

by Vardi [23], who started the first rigorous study about topology inference, by

ip t

analogy with tomographic methods used in medicine. Network tomography is

by no means specific to computer networks: It has also been studied in the con-

cr

text of transportation, and it still receives consistent attention (see, e.g., [24]).

From an information-theoretic point of view, the improvements achievable in network tomography through Network Coding have been studied. Interested

us

230

readers may refer to [25] for a survey.

The vector of link loads y can be obtained from the vectorized2 traffic matrix

an

x = vec X and the routing matrix A

y = Ax

(4)

235

M

The network tomography problem consists in estimating x given y. Note that typically the measured link counts are low-dimensional while the latent

d

point-to-point traffic volumes are high-dimensional. Network tomography is not relying on the assumption that the route along which traffic on a given

te

origin-destination (OD) pair is directed is unique [26]. Entries of the routing matrix can be imagined as modeling the fraction of traffic belonging to the OD pair k that is routed to link h, or as binary values ahk ∈ {0, 1} as in [27],

Ac ce p

240

expressing the circumstance that traffic relative to the OD pair k is routed onto the link h.

Figure 1 shows the load yAB over the link AB and the traffic matrix entries

xAB and xAC relative to the traffic exchanged between A and B and between

245

A and C. In addition, traffic which is local to A, denoted as xAA , is also shown. Note that literals have been used instead of number for the sake of clarity. A general and accessible formulation of regularized models for the tomogra-

phy as optimization problem was given in [1]. The best estimate of the vectorized 2 The

vec operator acts on a matrix by stacking its columns onto one another.

10

Page 11 of 31

ip t cr us an

M

Figure 1: A simple sketch of traffic matrix entries

d

traffic matrix is given by

x∗ = arg min2 R(x, y) + λd(x, M)

(5)

te

x∈Rn

where R(·, ·) is the deviation of a feasible solution x from the observations

Ac ce p

y, d(·, ·) is the deviation of the solution from the model M, and λ ≥ 0 is a regularization parameter penalizing solutions which deviate from the model. A

250

good choice for the prior model x(0) is the generalized gravity model, whereas alternatives for the penalty function include the mutual information between the estimated and the prior models.

4. The importance of traffic matrices in modern networks A reliable traffic matrix is able to explain us why the distribution of traffic

255

within the network behaves in a certain way, and more useful, it allows us to perform some kind of prediction on the evolution of the network, in presence of changes in its topology, traffic patterns and functional roles in network nodes. Topological changes may be caused by failure events, routing metric changes, 11

Page 12 of 31

creation of traffic engineering tunnels by using Multi-Protocol Label Switching 260

(MPLS) label switched paths (LSPs) [28], as well as by the introduction of new

ip t

network entities or the disposal of already existing ones. Modifications in traffic patterns may be due to the growth or reduction of the users’ volumes in specific

cr

areas within the network, whereas the change of roles within the network may

reflect the relocation of transit peering connections or specific network services. Clearly, in lack of a clear and stable formulation of the traffic matrix, it is not

us

265

easy to perform any kind of prediction on the network’s response to the above changes, since in order to gain a good understanding of the underlying network

an

dynamics we need estimate in advance what will be the effects on the traffic distribution when modifications occur in the network topology. For example, 270

when a communication link goes out of service some amount of traffic, previously

M

flowing through it, may move on a specific alternate path and some other may be diverted through another one, depending on the source and destination points characterizing such traffic.

also for the optimal placement of specific functional components within the net-

te

275

d

Accurate estimation of a traffic matrix becomes of paramount importance

work, mainly in presence of network virtualization practices. In such scenarios,

Ac ce p

traffic engineered tunnels can be set-up to encapsulate traffic between different hypervisors empowering virtualization services on different sites by allowing the realization of fully virtualized network infrastructures between virtual machines

280

that are independent on the underlying physical network. In addition, some basic networking components needed to support a modern infrastructure can be further consolidated and delivered in a virtualized way by decoupling the network functions, such as network address translation (NAT), firewalling, intrusion detection, domain name (DNS), load balancing and caching services, from

285

proprietary hardware appliances so that they can be fully managed as software applications and dynamically moved from site to site according to evolving traffic demands. Virtualized network-level services, supporting control plane function in Software Defined Networks, as well as security and content distribution have to be located in strategic places within the communication infrastructure and 12

Page 13 of 31

290

dynamically moved, thanks to the enhanced mobility features provided by stateof-the-art virtualization technologies, according to sudden changes in topology

ip t

and traffic dynamics. For each specific traffic-demand matrix, we can determine

a near-optimal virtual network topology and virtual network service placement

295

cr

that minimizes communication resource waste by fairly balancing the network

load towards virtualized network function provisioning points while satisfying

us

the performance and flexibility requirements of a modern virtual network infrastructure.

an

5. NFV and its effects on traffic 5.1. NFV

Competition in the telecommunications market is a powerful driver for op-

M

300

erators to reduce costs and improve agility. Traditional carrier networks rely upon dedicated hardware middleboxes, often provided by a single vendor, with

d

standardized network interfaces and behavior. These boxes implement network functions on top of proprietary operating systems and specialized hardware platforms. leading to a complex, rigid, and costly lifecycle management. With NFV,

te

305

a service component can be decomposed into a set of virtual network functions,

Ac ce p

which could then be separately implemented as software instances on commercial off-the-shelf (COTS) hardware. To adapt to changes in traffic, operators will no longer necessarily need to provision and deploy new middleboxes. They

310

only have to focus on the resources needed to run network functions adequate to support user demand. Carrier-grade networks can be thus built flexibly, provisioning connectivity as a service [29]. Seven telco operators formed an European Telecommunications Standards Institute (ETSI) specifications group for NFV [30]. NFV is expected to cut the total cost of ownership (TCO) by reducing

315

both equipment cost (CAPEX) and operational cost (OPEX). Architecturally, SDN can be viewed as an enabler to NFV. The two technologies complement each other, especially within a widely-spanned mobile network. NFV involves the separation of network functions from the underly-

13

Page 14 of 31

ing proprietary appliances, running network functions as software instances on 320

commodity equipment or in the cloud, whereas SDN support the decoupling

ip t

of control-plane and data-plane functions, enabling logically centralized control

over multi-layered, multi-vendor, networks. These concepts are being considered

cr

as promising tools towards a programmable, flexible, and cost-efficient mobile network architectures, the Software Defined Mobile Network (SDMN)[31]. De-

spite that, not excessive attention has been drawn to the network load and

us

325

data-plane delay implied by introducing NFV and SDN [32]. Considerable volumes of traffic will be involved and models for TME will need to reflect the

an

new context, considering the role played by the entities involved. In addition, during the adoption of NVF, it is reasonable to expect an extended migration 330

stage, where virtualized functions coexist with legacy hardware (so as to pre-

M

serve the return on investment on traditional systems). The shape of traffic during migration should reflect a steady transition from the legacy situation to the NFV-enabled one, since the effects of the transition add up linearly.

test network environments where new ideas can be safely tested in realistic

te

335

d

Entities with various roles in the business model may also wish to realize

conditions without causing any harm to customers [33]. The ease with which

Ac ce p

such laboratories can be set up and deployed will make likely that the impact of them on traffic matrices will be non-negligible. The IaaS (Infrastructure as a Service) business model decouples the roles of

340

Service Providers (SPs), who offer customer services and deploy the necessary support systems, and Infrastructure Providers (InPs), who are in charge of provisioning, operating, and maintaining physical resources. Within the business model for NFV (described e.g., in [34]), new roles are introduced. In particular, Virtual Network Function Providers (VnfP) can be expected to offer virtual

345

network functions. For the latter role, traditional vendors of network equipment are in the advantageous position of being able to evolve their portfolio, providing software implementations of their products, provided that they pay special attention to the adaptation of components originally designed for a single-tenant, static network, to the dynamic and scaling carrier cloud. NFV also opens the 14

Page 15 of 31

350

opportunity for other entities to specialize with innovative, niche services (which may include, e.g., tailored content filtering or malware protection) and act as

ip t

advanced service providers for those specific services. The virtualized network structure of NFV will create a network of relationships between stakeholders.

355

cr

Some of these relationships are many-to-many, in the sense that, for example, a virtual network operator (VNO) can rely on resources from different InPs,

us

and the infrastructure of a single InP can host multiple independent virtual networks. All the involved parties will want to protect their respective assets. An InP will have an interest in ensuring that malfunctions (or deliberate at-

360

an

tacks) within a single VNO are not disruptive of the smooth functioning of the entire infrastructure. On the other hand, a VNO will likely distrust its competitors and request that the separation with other tenants operating on the

M

same infrastructure be as thorough as possible. In addition, VNOs will regard the details of their traffic measurements and estimates as sensitive data and will attempt to prevent competitors to have access to them. They might be even attempt to interfere with the collection of measurement data, by either blocking

d

365

te

it, transmitting false readings, or injecting fake traffic for the sole purpose of masquerading the correct volumes.

Ac ce p

5.2. Evolved Packet Core (EPC)

Mobile networks have naturally been in the focus of research work inves-

370

tigating the introduction of NFV, SDN, and both [35]. The EPC is the core transport network for Long Term Evolution (LTE) networks as specified by 3GPP. The EPC performs essential functions that can be implemented as virtual network functions, including subscriber tracking, mobility management and session management.

375

Placement of components such as Serving Gateways (S-GWs) is usually done

statically, with reference to the forecast traffic demand, with about 10 S-GWs serving 106 subscribers [36]. The flexibility brought in by NFV will enable, however, on-demand adaptation to quickly variable demand, also with potentially large traffic volumes. For example, to provide convenient service in case 15

Page 16 of 31

380

of events with massive attendance, new SGWs could be activated and operated for the duration of the event. Basta et al. [35] investigated the performance

ip t

effect, in terms of network load and delay, brought about by virtualizing the

S-GW and Packet Gateway (P-GW) functions. A fully virtualized EPC would

385

cr

involve the instantiation of all virtual network functions into one or more data centers and the use of SDN-enabled switches to handle data traffic. Service

us

requirements (e.g., time-criticality or desire to guarantee tenant isolation) may, however, induce operators to deploy virtual network functions to locations close to end users, or even at customer premises. In such cases, the data traffic could

390

an

be offloaded to dedicated nodes across the core transport network, while the control plane logic could be centralized [32]. This way, the advantages of a global, unified view of the network would be retained. Substantial differences (up to

M

8x) were found between separation of control plane and data plane and full virtualization [32]. The authors also proposed a function placement scheme that minimizes the SDN-induced overhead. Issues opened to further study include the need to ensure that the centralized control in the carrier cloud complies with

d

395

bottleneck.

te

the regulatory delay requirements and to avoid that it becomes a performance

Ac ce p

5.3. 5G and Cloud Radio Access Network (C-RAN) Next generation mobile networks (5G) are envisioned to increase area capac-

400

ity by three orders of magnitude with respect to 4G, and be extremely reliable [37]. Their peak rates should be in the range of tens of Gbps, with millisecond round-trip latency, and they are expected to support the connection of a trillion devices, supporting the Internet of Things (IoT) paradigm. To reduce network costs and meet the future requirements of high spectral efficiency coupled

405

with a low power footprint, advanced wireless network architectures are sought. Since Base Stations consume substantial energy, virtualizing their functions in a Cloud Radio Access Network (C-RAN) is an appealing perspective [37]. Important aspects of C-RAN that can generate significant traffic and need thus to be considered for accurate TME include the placement of controller(s) and 16

Page 17 of 31

410

the edge cache. Since latency and other network performance metrics depend significantly on the location of controllers, their placement should be carefully

ip t

looked after. With edge caching, content that is determined as likely to be mas-

sively requested is downloaded to the periphery and cached there. Edge caching

415

cr

also needs to deal with the problems raised by encrypted content. 5.4. Network function placement

us

With NFV, network functions can be instantiated in the locations which are most convenient for the operator. Eligible locations include datacenters,

an

network nodes, specific geographic locations, and end-user premises [34], with effects on the distribution of traffic flows which need to be reflected in estimation 420

models.

M

In a bird’s eye view, two conflicting groups of objectives affect the placement of virtual network functions: reduction of costs by consolidation of hardware and workforce competence calls for centralization, while decentralization

functions at customer premises can be motivated by subscribers asking for ded-

te

425

d

is motivated by performance and security reasons [38]. Placing virtual network

icated service, requiring that their functions are protected and isolated from the ones dedicated to other customers. In addition, the need to efficiently manage

Ac ce p

the Network Function Virtualization Infrastructure (NFVI) and protect it from the delivered subscriber services is likely to introduce a further layer of functions

430

(mostly firewalls) and associated traffic. Placement of virtual network functions in distinct fault domains to increase resilience is also likely with NFV, also inducing possible alterations in traffic patterns. When the temporal dimension is taken into account, virtual network functions may be dynamically relocated and instantiated to meet the demand with fine granularity.

435

Network functions cannot be embedded freely, because of strict chaining

requirements that exist for service components. Localization of the virtual network functions supporting service components needs to ensure compliance with chaining requirements, as well as be feasible given the network topology and the desired performance. Sequencing requirements in virtual network functions 17

Page 18 of 31

440

are expressed with the Virtual Network Functions Forwarding Graph (VNFFG)

able to an InP is an interesting research problem in itself.

ip t

[34]. Inferring an unknown topology given the VNFFG and information avail-

Allocating virtual network functions into an entirely operator-owned cloud

445

cr

is a solution bringing many advantages, including cost containment, dynamic migration, flexible upgrades, and complete control over the infrastructure. How-

us

ever, not all operators are already equipped with their own cloud infrastructure or are willing to invest to build one. Therefore, as a consequence of NFV, an increase of traffic exchanged between different entities (operators and cloud

450

an

providers) is expected, with many issues and open problems still to be solved, including the need for new dedicated links and peerings as well as more awkward aspects such as lawful interception that is likely to cause a further growth

M

in traffic volumes.

The optimal placement of NFV components has been the subject of several research works. More generally, the problem of embedding virtual networks in a substrate network is usually referred to as the Virtual Network Embedding

d

455

te

(VNE) problem [39]. The VNE problem is NP-hard and many heuristics have been put forward to address it. Also, the notion of optimality has been de-

Ac ce p

clined in several ways. Criteria that have been suggested to precisely specify optimal allocation in the VNE problem include setup cost, latency, economical

460

profit, resilience, risk exposure, energy-related considerations, and acceptance rate (defined as the fraction of virtual network requests that are accepted). As VNE deals with the allocation of the nodes and links of a virtual network, two subproblems can be identified: Virtual Node Mapping (VNoM), concerning the allocation of virtual nodes into physical ones, and Virtual Link Mapping

465

(VLiM), which is about the choice of paths in the substrate network to implement the virtual links. These subproblems can be addressed separately or in a combined way.

18

Page 19 of 31

5.5. Monitoring and security

470

ip t

A logically centralized network architecture facilitates monitoring and analysis, as well as prompt reaction to failures and attacks, provided that the control point has accurate, complete, and up to date information about the network

cr

state. Centralized control and global view of the network, two features of SDN, may also be leveraged upon to retrieve and gather measurements of network traf-

475

us

fic, to aid in traffic engineering. Software Defined Monitoring (SDM) consists of harvesting information from high-speed networks in an efficient and dynamical way. According to the SDM paradigm, SDN network elements report packet

an

and flow data and metadata, along with failure status, to the controller [31], which may in turn offload them to big data analytics systems aimed at correlating heterogeneous and complex data, profiling them and analyzing them, in support of both the regulation of network operation and anomaly detection

M

480

[40]. The adequacy and effectiveness of security policies can be effectively verified and the network can be quickly reprogrammed to reflect policy updates.

d

Having said that, management and synchronization of security policies and traf-

485

te

fic forwarding policies at a network-wide scale requires a global analysis of all elements within the network and of the mutual relationships between them, to

Ac ce p

identify and correct inconsistencies, conflicts, or performance bottlenecks [31]. The overhead generated by such activities may be not negligible, especially for forensic applications. As far as filtering is concerned, the sheer amount of traffic to be inspected with high-speed networks implies the recourse to specialized

490

hardware. Field-Programmable Gate Arrays (FPGAs) combine power and flexibility and might be advantageously used to this purpose [41]. This will create additional constraints for network function placement which can be exploited to reduce the complexity of TME. 5.6. Peering and topology

495

The Internet is the interconnection of over 40,000 sovereign entities known as Autonomous Systems (ASes). Traffic is routed in accordance with the complex routing policies of each participating network, along with the topological 19

Page 20 of 31

structure of the interdomain network. It should be kept in mind that the abstraction of ASes as atomic nodes in a network hides a lot of significant detail about the real routing structure, which depends on finer-grained aspects [42].

ip t

500

The different connectivity structures associated with virtual or logical connec-

cr

tions are shaped by technological, economic, and societal forces and evolve in response to external and internal signals and responses. Obtaining a complete

505

us

picture of such elements is difficult, because ASes are reluctant to divulge what they consider privileged information providing competitive advantages, including both technological choices (the internal topological structure, availability of

an

backup paths, routing strategies) and commercial details (customers and their contracts).

An important goal that ISPs may wish to pursue is control and optimization of routing in order to allow them to shift the traffic inside and outside

M

510

their network in the most efficient way. Lutu et al. [43] showed that customer ISPs can control traffic by strategic disaggregation, that consists in selectively

d

announcing different fragments of the assigned address block to disjoint subsets

515

te

of neighbors. Due to asymmetries in traffic sources and to the 95th percentile billing, customer ISPs can even reduce the monthly bill paid to their transit

Ac ce p

providers.

NVF and SDN increase the degree of control that operators may exercise

over their traffic, even though offloading it to other network may raise security concerns. The diffusion of NVF-based SDN-enabled networks may affect the

520

global network topology. In this respect, a reflection on a trend that is being observed in the Internet may be insightful. The evolution in the Internet economic ecosystem includes two trends: One is toward a surge in the recourse to peering and the other is toward a reduction in the number of intermediary organizations on Internet paths (a phenomenon known as flattening). The re-

525

lated issues have been raised in the context of peering, in a thorough discussion of remote peering [44], where a network whose infrastructure does not reach a shared location for interconnection uses a layer-2 intermediary to connect and peer with other networks. When layer-2 connections are used to remotize links, 20

Page 21 of 31

it is worth noticing that pure layer-3 modeling ignores the presence of some 530

of the intermediaries, although explicit discovery mechanism may be used to

ip t

reveal otherwise hidden Multiprotocol Label Switching (MPLS) tunnels [45].

Among the considerations made by Castro et al. in this respect, we report here

cr

two, which are relevant to the context discussed here. First, the hidden layer-2 intermediaries might not be easily accountable for delaying or dropping traffic,

monitoring of it, attempt to retrieve privileged information about the structure

us

535

and organization of tenants’ networks, or, much worse, expose traffic to other entities. Moreover, the analysis at layer 3 makes difficult to assess the real de-

an

gree of reliability of a network structure, since the same resources may be used to provide interconnection at both layers. 5.7. Traffic matrix completion

M

540

The notion of independence of traffic matrix entries, implicit in gravity models, has been questioned in real interdomain traffic. A line of research has used

d

the observation that static traffic matrices have been empirically shown to have

545

te

low effective rank, and hence high spatial correlation. In other words, knowledge of a few entries may be sufficient to infer the entire matrix. Consequently, techniques such as Principal Components Regression could be advantageously

Ac ce p

used in the inference. In particular, an interesting application involves inferring characteristics of traffic that an ISP cannot observe (because it flows entirely outside its network), starting from link measurements [46]. This problem is

550

termed traffic matrix completion [47]. This is especially important since many of the factors an ISP has to evaluate when deciding whether a new peering agreement should be initiated involve knowledge of traffic that the ISP is unable to see [6]. The extent to which the introduction of NFV and SDN will influence the degree of independence between traffic matrix entries is a significant factor

555

worth of investigation. As an example of the aspects related to NFV which should be taken account when estimating a traffic matrix, Fig. 2 portrays two different scenarios for Customer Premises Network Virtualization, where the traffic matrix will most likely have different characteristics in the two imple21

Page 22 of 31

mentations. In the upper scenario, VNFs are allocated at the SP level, whereas 560

in the lower scenario, VNFs are allocated entirely within customer premises.

ip t

The traffic exchanged will be different in quantity and quality, because in the

first case filters will operate upstream while in the second case all traffic will

LAN

us

cr

cross the link.

NID

VNF VNF Branch

an

VNF VNF

VNF VNF

VNF

M

LAN

Service Provider

VNF VNF

d

Branch

te

Figure 2: Two possible implementations for Customer Premises Network Virtualization.

Ac ce p

5.8. Pricing models 565

Understanding how traffic matrices—and methods for their estimation—will

evolve with SDN-supported NVF-based networks is also important because traffic volumes are one of the factors influencing transport pricing. In turn, pricing levels determine whether or not new connections and peering agreements are establishment or not, shaping the network topology. An extensive literature dis-

570

cusses pricing models and tactics and their impact on network topology. Faratin et al. [4] introduced the classical distinction of ISPs into Eyeball ISPs–those providing last-mile access–and Content ISPs, specialized in providing connectivity (and hosting) to content providers. Ma et al. [48] devised a game-theoretic approach and extended the pricing model to encompass Transit ISPs. In an

575

effort to explain the evolution of the Internet topology as the result of a revenue

22

Page 23 of 31

optimization process on part of the ASes, Corbo et al. [49] suggested a diffusion process based on an utility function that incorporates benefits for traffic

ip t

routed, congestion costs, and payment transfers. In [50], the economics of ex-

change points were analyzed with reference to volumes and QoS requirements. An investigation of the profits yielded by tiered pricing in the wholesale tran-

cr

580

sit market, and of whether better bundling strategies might exist has been the

us

subject of [51]. A noteworthy finding is that the practice of pricing traffic in contractual tiers based only on the cost of carrying the traffic (e.g., offering a discount for local traffic) is suboptimal. ISPs can achieve near-optimal profits by structuring tiered contracts with a few tiers according to both traffic cost

an

585

and demand. The raising demand for end-to-end connections with guaranteed quality stimulated also a line of research in terms of contract types. Karaoglu

contracts with bailout options. 590

M

et al. [52], for example, suggested a contracting mechanism based on forward

When evaluating the pricing policies in the business ecosystem enabled by

d

NFV, results established by analyzing the relationship between pricing and de-

te

mand in cloud services can point to interesting direction for research, since many of these results are likely to hold also in the NFV business model. Optimally

Ac ce p

pricing cloud resources to improve revenue while not discouraging demand is a 595

challenging task, especially in consideration of the need to dynamically adjust prices to meet sudden changes in demand and of the perishable nature of resources [53]. For an IaaS provider, the optimal allocation of available resources to pricing plans has been recently examined in [54]

6. Conclusions

600

Decisions that network operators have to make regarding the evolution of

their infrastructure depend on detailed and accurate predictions of the traffic that will be carried. An increase in volume and complexity of traffic demand can have substantial effects on the architecture of the network that will service it. Estimation of traffic quantities is a complicated task that has required the

23

Page 24 of 31

605

development of advanced traffic models to reduce complexity to a manageable level. The information available within a traffic matrix is extremely valuable

ip t

for many traffic engineering tasks such as routing and load balancing, network

provisioning and dimensioning, as well as restoration/failover strategies. A clear

610

pacity planning network growth and fault diagnosis.

cr

knowledge on both the size and locality of traffic flows is fundamental for ca-

us

With the advent of NFV and SDN, traffic flows may change dramatically, as entirely new actors enter the business ecosystem and traditional players may exchange traffic between them in unprecedented ways. Offloading of functions

615

an

to a cloud, proactive caching, dynamic network reconfiguration, a boost in the ability to control traffic (to various purposes, including expenditure cuts), and the stringent need to protect assets and guarantee isolation are among the fac-

M

tors that will affect traffic matrices. The assumptions of at least some of the estimation models may need to be revisited in the light of these changes. In addition, to validate the evolved models, the scientific community will need new datasets, which may be difficult to obtain given the unwillingness of operators

d

620

te

to reveal traffic details. We listed the principal transformations induced by NFV and SDN and discussed their potential effect for TME, looking at the problem

Ac ce p

from several points of view, including engineering and econometric perspectives.

References

625

[1] P. Tune, M. Roughan, Internet traffic matrices: A primer, Vol. 1 of ACM SIGCOMM, Springer, 2013.

[2] D. Alderson, H. Chang, M. Roughan, S. Uhlig, W. Willinger, The many facets of Internet topology and traffic, Networks and Heterogeneous Media 1 (4) (2006) 569–600.

630

[3] V. Erramill, M. Crovella, N. Taft, An independent-connection model for traffic matrices, in: Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, ACM, 2006, pp. 251–256.

24

Page 25 of 31

[4] P. Faratin, D. Clark, P. Gilmore, S. Bauer, A. Berger, W. Lehr, Complexity of Internet interconnections: Technology, incentives and implications for policy, in: in Proc. 35th Res. Conf. Commun., Inf. Internet Policy (TPRC,

ip t

635

2007.

cr

[5] R. Guerzoni, et al., Network Functions Virtualisation: an introduction,

benefits, enablers, challenges and call for action – introductory white paper,

640

us

in: SDN and OpenFlow World Congress, 2012.

[6] A. Lodhi, N. Laoutaris, A. Dhamdhere, C. Dovrolis, Complexities in Inter-

an

net peering: Understanding the “black” in the “black art”, in: 2015 IEEE Conference on Computer Communications (INFOCOM), IEEE, 2015, pp. 1778–1786.

645

M

[7] S. Yi, C. Li, Q. Li, A survey of fog computing: concepts, applications and issues, in: Proceedings of the 2015 Workshop on Mobile Big Data, ACM,

d

2015, pp. 37–42.

[8] O. Goldschmidt, Isp backbone traffic inference methods to support traffic

te

engineering, in: Internet Statistics and Metrics Analysis (ISMA) Workshop,

Ac ce p

2000, pp. 1063–1075. 650

[9] C. Tebaldi, M. West, Bayesian inference on network traffic using link count data, Journal of the American Statistical Association 93 (442) (1998) 557– 573.

[10] J. Cao, D. Davis, S. Vander Wiel, B. Yu, Time-varying network tomography: router link data, Journal of the American statistical association

655

95 (452) (2000) 1063–1075.

[11] M. Roughan, Y. Zhang, W. Willinger, L. Qiu, Spatio-temporal compressive sensing and Internet traffic matrices (extended version), IEEE/ACM Transactions on Networking 20 (3) (2012) 662–676. [12] Y. Zhang, M. Roughan, C. Lund, D. L. Donoho, Estimating point-to-

660

point and point-to-multipoint traffic matrices: an information-theoretic 25

Page 26 of 31

approach, IEEE/ACM Transactions on Networking (TON) 13 (5) (2005)

ip t

947–960. [13] Z. Hu, Y. Qiao, J. Luo, Coarse-grained traffic matrix estimation for data

665

cr

center networks, Computer Communications 56 (2015) 25–34.

[14] E. Barbierato, G.-L. D. Rossi, M. Gribaudo, M. Iacono, A. Marin, Exploiting product forms solution techniques in multiformalism modeling,

us

Electronic Notes in Theoretical Computer Science 296 (0) (2013) 61 – 77. doi:http://dx.doi.org/10.1016/j.entcs.2013.07.005.

670

an

[15] H. Chang, S. Jamin, Z. M. Mao, W. Willinger, An empirical approach to modeling inter-AS traffic matrices, in: Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, USENIX Association, 2005,

M

pp. 12–12.

[16] A. Thyagaturu, A. Mercian, M. P. McGarry, M. Reisslein, W. Kellerer,

IEEE Communications Surveys Tutorials.

te

675

d

Software Defined Optical Networks (SDONs): A comprehensive survey,

[17] E. M. Airoldi, A. W. Blocker, Estimating latent processes on a network

Ac ce p

from indirect measurements, Journal of the American Statistical Association 108 (501) (2013) 149–164.

[18] A. Hyv¨ arinen, E. Oja, Independent component analysis: algorithms and

680

applications, Neural networks 13 (4) (2000) 411–430.

[19] A. Medina, N. Taft, K. Salamatian, S. Bhattacharyya, C. Diot, Traffic matrix estimation: Existing techniques and new directions, ACM SIGCOMM Computer Communication Review 32 (4) (2002) 161–174.

[20] W. B. Norton, Internet service providers and peering. 685

URL http://drpeering.net [21] P. Gill, M. Schapira, S. Goldberg, A survey of interdomain routing policies, ACM SIGCOMM Computer Communication Review 44 (1) (2013) 28–34. 26

Page 27 of 31

[22] R. Johari, J. N. Tsitsiklis, Routing and peering in a competitive Internet,

690

ip t

LIDS 2570, MIT (2003). [23] Y. Vardi, Network tomography: Estimating source-destination traffic in-

91 (433) (1996) 365–377.

cr

tensities from link data, Journal of the American statistical association

[24] M. L. Hazelton, et al., Network tomography for integer-valued traffic, The

695

us

Annals of Applied Statistics 9 (1) (2015) 474–506.

[25] P. Qin, B. Dai, B. Huang, G. Xu, K. Wu, A survey on network tomography

an

with network coding, IEEE Communications Surveys & Tutorials 16 (4) (2014) 1981–1995.

M

[26] M. Rabbat, R. Nowak, M. Coates, Multiple source, multiple destination network tomography, in: INFOCOM 2004. Twenty-third AnnualJoint Con700

ference of the IEEE Computer and Communications Societies, Vol. 3, IEEE,

d

2004, pp. 1628–1639.

te

[27] E. Zhao, L. Tan, A PCA based optimization approach for IP traffic matrix estimation, Journal of Network and Computer Applications 57 (2015) 12–

Ac ce p

20. 705

[28] F. Palmieri, An MPLS-based architecture for scalable QoS and traffic engineering in converged multiservice mobile IP networks, Computer Networks 47 (2) (2005) 257–269.

[29] T. Taleb, Toward carrier cloud: Potential, challenges, and solutions, IEEE Wireless Communications 21 (3) (2014) 80–91.

710

[30] H. Hawilo, A. Shami, M. Mirahmadi, R. Asal, NFV: state of the art, challenges, and implementation in next generation mobile networks (vEPC), IEEE Network 28 (6) (2014) 18–26. [31] M. Liyanage, I. Ahmed, M. Ylianttila, J. L. Santos, R. Kantola, O. L. Perez, M. U. Itzazelaia, E. M. de Oca, A. Valtierra, C. Jimenez, Security 27

Page 28 of 31

715

for future software defined mobile networks, in: Next Generation Mobile Applications, Services and Technologies, 2015 9th International Conference

ip t

on, IEEE, 2015, pp. 256–264.

[32] A. Basta, W. Kellerer, M. Hoffmann, H. J. Morper, K. Hoffmann, Applying

720

cr

NFV and SDN to LTE mobile core gateways, the functions placement prob-

lem, in: Proceedings of the 4th workshop on All things cellular: operations,

us

applications, & challenges, ACM, 2014, pp. 33–38.

[33] A. Blenk, A. Basta, M. Reisslein, W. Kellerer, Survey on network virtual-

an

ization hypervisors for software defined networking, IEEE Communications Surveys & Tutorials 18 (1) (2016) 655–685. 725

[34] R. Mijumbi, J. Serrat, J.-L. Gorricho, N. Bouten, F. De Turck, R. Boutaba,

M

Network function virtualization: State-of-the-art and research challenges, IEEE Communications Surveys & Tutorials 18 (1) (2015) 236–262.

d

[35] A. Basta, W. Kellerer, M. Hoffmann, K. Hoffmann, E.-D. Schmidt, A virtual SDN-enabled LTE EPC architecture: a case study for S-/P-Gateways functions, in: Future Networks and Services (SDN4FNS), 2013 IEEE SDN

te

730

Ac ce p

for, IEEE, 2013, pp. 1–7.

[36] S. Gebert, D. Hock, T. Zinner, P. Tran-Gia, M. Hoffmann, M. Jarschel, E.D. Schmidt, R.-P. Braun, C. Banse, A. K¨opsel, Demonstrating the optimal placement of virtualized cellular network functions in case of large crowd

735

events, in: ACM SIGCOMM Computer Communication Review, Vol. 44, ACM, 2014, pp. 359–360.

[37] M. Peng, Y. Sun, X. Li, Z. Mao, C. Wang, Recent advances in cloud radio access networks: System architectures, key techniques, and open issues, IEEE Communications Surveys Tutorials PP (99).

740

[38] P. Iovanna, F. Ubaldi, T. Pepe, L. M. Contreras, V. Lopez, J. P. F.-P. Gimenez, Main challenges on WAN due to NFV and SDN: Multi-layer and multi-domain network virtualization and routing, in: Optical Network 28

Page 29 of 31

Design and Modeling (ONDM), 2015 International Conference on, IEEE,

745

ip t

2015, pp. 74–79. [39] A. Fischer, J. F. Botero, M. T. Beck, H. De Meer, X. Hesselbach, Virtual network embedding: A survey, IEEE Communications Surveys & Tutorials

cr

15 (4) (2013) 1888–1906.

[40] L. Cui, F. R. Yu, Q. Yan, When big data meets software-defined network-

750

us

ing: SDN for big data and big data for SDN, IEEE Network 30 (1) (2016) 58–65.

an

[41] L. Kekely, J. Kuˇcera, V. Puˇs, J. Koˇrenek, A. V. Vasilakos, Software defined monitoring of application protocols, IEEE Transactions on Comput-

M

ers 65 (2) (2016) 615–626.

[42] M. Roughan, W. Willinger, O. Maennel, D. Perouli, R. Bush, 10 lessons 755

from 10 years of measuring and modeling the Internet’s autonomous sys-

te

1810–1821.

d

tems, IEEE Journal on Selected Areas in Communications 29 (9) (2011)

[43] A. Lutu, M. Bagnulo, C. Pelsser, K. Cho, R. Stanojevic, An analysis of the

Ac ce p

economic impact of strategic deaggregation, Computer Networks 81 (2015) 760

147–163.

[44] I. Castro, J. C. Cardona, S. Gorinsky, P. Francois, Remote peering: More peering without Internet flattening, in: Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies, ACM, 2014, pp. 185–198.

765

[45] B. Donnet, M. Luckie, P. M´erindol, J.-J. Pansiot, Revealing mpls tunnels obscured from traceroute, ACM SIGCOMM Computer Communication Review 42 (2) (2012) 87–93. [46] V. Bharti, P. Kankar, L. Setia, G. G¨ ursun, A. Lakhina, M. Crovella, Inferring invisible traffic, in: Proceedings of the 6th International Conference

29

Page 30 of 31

770

on emerging Networking EXperiments and Technologies (CoNEXT), ACM,

ip t

2010, p. 22. [47] G. G¨ ursun, M. Crovella, On traffic matrix completion in the Internet, in:

Proceedings of the 2012 ACM conference on Internet measurement confer-

775

cr

ence, ACM, 2012, pp. 399–412.

[48] R. T. Ma, D. M. Chiu, J. C. Lui, V. Misra, D. Rubenstein, On cooper-

us

ative settlement between Content, Transit, and Eyeball Internet service providers, IEEE/ACM Transactions on networking 19 (3) (2011) 802–815.

an

[49] J. Corbo, S. Jain, M. Mitzenmacher, D. C. Parkes, An economicallyprincipled generative model of AS graph connectivity, in: INFOCOM 2009, IEEE, IEEE, 2009, pp. 2941–2945.

M

780

[50] S. Shakkottai, R. Srikant, Economics of network pricing with multiple ISPs, IEEE/ACM Transactions on Networking (TON) 14 (6) (2006) 1233–1245.

d

[51] V. Valancius, C. Lumezanu, N. Feamster, R. Johari, V. V. Vazirani, How

785

te

many tiers? Pricing in the Internet transit market, in: ACM SIGCOMM Computer Communication Review, Vol. 41, ACM, 2011, pp. 194–205.

Ac ce p

[52] H. T. Karaoglu, A. Gupta, M. Yuksel, W. Liu, K. Kar, Bailout forward contracts for edge-to-edge Internet services, Computer Communications 36 (17) (2013) 1708–1725.

[53] H. Xu, B. Li, et al., Dynamic cloud pricing for revenue maximization, in:

790

IEEE Trans. Cloud Comput, Citeseer, 2013.

[54] A. N. Toosi, K. Vanmechelen, K. Ramamohanarao, R. Buyya, Revenue maximization with optimal capacity control in infrastructure as a service cloud markets, IEEE Transactions on Cloud Computing 3 (3) (2015) 261– 274.

30

Page 31 of 31