The service overlay network design problem for interactive internet applications

Computers & Operations Research 57 (2015) 73–82

Contents lists available at ScienceDirect

Computers & Operations Research journal homepage: www.elsevier.com/locate/caor

The service overlay network design problem for interactive internet applications Yuh-Rong Chen a, Sridhar Radhakrishnan a, Sudarshan Dhall a, Suleyman Karabuk b,n a b

School of Computer Science, The University of Oklahoma, Norman, OK, USA School of Industrial & Systems Engineering, The University of Oklahoma, Norman, OK, USA

art ic l e i nf o

a b s t r a c t

Available online 24 November 2014

Consider a private network of geographically dispersed computers with fast and high capacity connections, and an Internet application session, such as a massive multiplayer online game, with a server and a set of clients. We refer to the former as a service overlay network (SON), and assume that it could be connected to the Internet. The problem is to decide how to configure and utilize the SON in support of this application, such that the clients' speed of communication with the server is within given communication performance requirements. We provide an Integer Programming formulation of this problem, and prove that it is NP
Hard. In an attempt to solve the problem within strict computational time requirements of actual applications, we develop a solution framework based on partitioning and enumerating the solution space into smaller subproblems, one or more of which contains an optimal solution. In this framework, we develop and test an optimal seeking exact, and a fast polynomial time heuristic algorithm with success. The exact algorithm sets optimally solvable sizes of the subject problem, whereas the heuristic algorithm sets the size of solvable instances in a real application. Published by Elsevier Ltd.

Keywords: Network design Large scale optimization Interactive network application Service overlay network Communication network

1. Introduction Network applications that involve a large number of users who participate in a communication session have become more popular today with the availability of affordable high-speed Internet. Some of these applications are interactive, meaning that two-way communication takes place between users, such as distributed multimedia collaboration [13], networked audio/video conferences [11,18], multiplayer online games [2,3,23,27] (including online virtual worlds [7]), and online auctions [29]. Whereas some are non-interactive such as Internet radio, and movie streaming. This research is concerned with interactive network applications, where participation takes place over the Internet. A multiplayer online game is arguably the most widely utilized application type thanks to the pervasive availability of game consoles with Internet capability such as Xbox or PlayStation and their corresponding online multiplayer gaming services, which have enabled a wide-range of individuals to participate in a game. It is also the largest application type with a number of participants reported to be as many as 4000 [21], although a typical session

n

Corresponding author. E-mail address: [email protected] (S. Karabuk).

http://dx.doi.org/10.1016/j.cor.2014.11.003 0305-0548/Published by Elsevier Ltd.

within a game is usually limited to several hundreds. For ease of exposition, throughout the rest of the manuscript we will utilize a multiplayer online game (MOG) application as context when needed. Consider p users involved in a MOG session. In order to keep a consistent view of the common virtual world, every user (or client)1 sends its updates such as locations or actions, in the format of network packets, to other users and receives updates from others. As illustrated in Fig. 1, there are two major network models to facilitate this interaction: centralized (or client–server), where all communications from users are routed through a server, and decentralized (or Peer-to-Peer), where users communicate directly without any central coordinator. In the centralized model (Fig. 1a), a global state of the common virtual world is maintained by the central server S. The users send their updates to S, which in turn restores the order of these updates, computes the new global state, and sends global state updates to affected users. On the other hand, a replication of the global state is maintained by each user in the decentralized model (Fig. 1b). The users exchange their updates directly and utilize

1 To be precise, the term client is the computing device involved in the application and user is the entity that controls the client. We will use these two terms interchangeably in the paper but readers should be able to distinguish them from the context.

74

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

P4

P1

P1 P2

P5

S3

P5

P2

P5

P3

S2 P6

S S6

P2

P4

P3

P4

S4

P3 S5

Fig. 1. Two communication models of interactive network applications. (a) Centralized and (b) decentralized.

P1

S1 P7 P8

more complex algorithms to restore the order of the events before computing the new global state. The decentralized model is known to be less scalable due to the lack of a centralized control [16]. The focus of this research is on the design of centralized networks. During a MOG session a player performs an action using input devices. This action must take effect within a short period of time, otherwise users may stop playing and leave the game [10]. The time difference between when an action is performed and when it takes effect is called the perceived latency, most of which is attributable to the network (i.e. the Internet) latency. An update needs to travel to the central server and the corresponding global state update comes back to the users over the Internet. The perceived latency can be roughly estimated as twice of the network latency from a user to the central server. Therefore, the duration of perceived latency, which must be set to an acceptable value and be satisfied for all players, is one critical performance measure of the application [12]. Henceforth, we refer to this consideration as the delay. Consider a MOG session with the central server S and two users A and B, such that the network delay from A to S is significantly shorter than that from B to S. At some point of the game session, A and B find some desirable object (e.g. some powerful weapon) about the same time, and they both attempt to grab it simultaneously. To do this, both players need to send a state update to S. Since the delay from A to S is shorter, S will receive A's packet first and allow A to pick up the object. Later, when S receives the packet from B, it will deny B's state update request because the object is no longer available. Furthermore, as long as B's delay is significantly longer than that of A's, B will be at a permanent disadvantage in the competition against A. Another example can be observed in an online auction, where many users place (automated) bids for an item in a short amount of time, for example during the last few seconds of the auction. A user who receives updates of the currently maximum bid disproportionately faster than the others would have an information advantage that can be exploited to win the auction. Therefore, the magnitude of variation in delay among users is another critical performance metric, specifically in applications where competition is a major element. Henceforth, we refer to this consideration as the delay variation. We adapt the definition of delay variation as the difference between the maximum delay and the minimum delay between all pairs of users, as was first mentioned in [25]. Approaches for reducing the delay variation, such as finding alternative paths in the computer network or packet buffering in the server, are presented in [5,25]. The maximum acceptable levels of the delay and the delay variation vary from application to application, even under different scenarios within the same application. For example, “raiding” in a

Application Server User Internet Node SON Server

Fig. 2. Service overlay network.

massive MOG (wherein a large amount of users assemble at a location to perform fast-paced game actions) has more stringent delay requirements in comparison to actions involving one player simply roaming and exploring in the game world. In a typical application, the delay can vary greatly depending on where and when a user connects to the Internet, the routing policy of the user's Internet Service Provider, and the location of the server on the Internet. None of these factors that effect delay can be controlled or mitigated easily, which leads to the utilization of so called service overlay networks that we discuss next.

1.1. The service overlay network approach It is a well established approach to utilize a private computer network of servers that are interconnected by low latency, high speed, and high capacity links in support of interactive Internet applications [1,14,26,28]. The main idea is that both the users of the network application, and the server connect to this private network via the Internet, after which all communication takes place through the fast private network. This private network is named as a service overlay network (SON). As depicted in Fig. 2, in the proposed system, a user's delay consists of three parts: (1) the delay from the user to the SON server that the user connects to (the contact server), which takes place over the Internet, (2) the delay from the contact server to the SON server that the application server connects to (the root server), which takes place over the SON; (3) delay from the root server to the application server, which takes place on the Internet. Therefore, bypassing the slow Internet via the SON, the performance of an interactive Internet application can be controlled and improved significantly. The first delay component can be minimized by selecting contact servers, and an assignment of users to the selected contact servers; the third delay, similarly, can be minimized by selecting a root server close to the application server. The second delay component can be affected by selecting a configuration of the SON including number and location of servers and the links in between. In this decision problem the constraints are to satisfy the delay, and the delay variation requirements, with the objective of minimizing the size of the SON, hence the total of set up and operating costs. Another notable benefit of a SON is that the servers can be configured to conserve bandwidth via application layer multicasting [4,19,24], much more effectively than possible compared to

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

utilizing the Internet only. In this research, we will ignore the network traffic and link capacities for the sake of simplicity. 1.2. Similarities to logistics network design problems Another related area of literature that deserves special attention is on logistics hub-network design. The general problem is defined as follows. Consider a transportation network consisting of locations (nodes) and routes (links) connecting them, where each location has a shipment to every other location. Each location could be set up as a hub that would collect shipments from other locations that are assigned to it, consolidate them, and send them either to another hub or to their final destinations. Due to economies of scale, the links between hub locations are assumed to operate much more efficiently than direct links between locations. Consequently, the transportation speed and/or costs per unit flow per unit distance on hub to hub links are substantially better. The decision problem is to select the hub locations, and assign locations to hubs, so that the total cost of all shipments is minimized. There are many variations of this general problem [9,15] (for example, the p-hub model requires that all flow go through one or two hubs, and that exactly p hub locations must be selected.) The “latest arrival hub location problem” described in [20], considers the maximum transportation time for any shipment as a hard constraint, and minimizes the total number of hubs. The parallels between the SON design problem with delay constraints in a decentralized (Peer-to-Peer) architecture and the latest arrival hub location problem are clear: the SON servers correspond to hub locations, the users of the Internet application correspond to nonhub locations, and between-user communication delays correspond to the transportation time between a shipping and a receiving location. In [20], the problem is formulated as an Integer Programming model (IP), and relatively small instances are solved with an off the shelf IP solver. A more specialized hub network design problem involves locating a cross docking facility, where all shipments from origins, after going through one or more hubs, are received, sorted (but not stored), and shipped out to final destinations [6]. At the cross docking facility, many smaller shipments to a single destination are placed into one or more of the arriving vehicles, therefore coordination and synchronization of incoming arrivals is a critical performance criterion. This problem can be represented by the SON design problem with delay and delay constraints in a centralized (client–server) architecture, where the application server corresponds to the cross docking facility, and the delay and delay variation constraints correspond to requirement of synchronization of arrivals at the facility.

75

compared to the decentralized architecture (or its variations), and it more easily enforces licensing requirements of the application; (3) we consider delay variation as an explicit constraint, because it is a critical performance requirement when competition between users takes place. Therefore, our main contribution is to solve a larger variety of interactive Internet applications. The rest of this paper is organized as follows. In Section 2, we formulate an integer linear programming model that describes the problem we study. In Section 3, we analyze the computational properties of the model and develop heuristic solution algorithms, followed by an extensive computational study in Section 4. Lastly, we summarize and conclude in Section 5.

2. Problem formulation 2.1. Notation Consider a SON described as a graph G ¼ ðV; EÞ, where V is the set of n service nodes,2 and E is the set of m links connecting the service nodes. Let dii0 denote the delay between service nodes i; i0 A V . When there is a direct link between (i; i0 ), dii0 is the delay on this link, otherwise, dii0 is the sum of the delays of the links along the shortest path between them. (We assume that the potential delay at the service nodes is negligible.) In relation to G, consider an Internet application session (s,C) with a server s and a set of users C (of size p), each of which can access any service node i through the public Internet with a given access delay. Let D ¼ ðG; s; CÞ denote such an application session. Let A ¼ fas ; a1 ; ⋯; ap g denote an assignment, where as is the service node that s is assigned to (root node), aj is the service node user j is assigned to (contact node). Let jAj denote the size (i.e. number of nodes) of the subset of G that are needed to facilitate the application session (s,C). As described earlier, the communication delay for a user j is the round trip time between j and s, therefore we define the delay for a session D under assignment A as λD;A ðjÞ ¼ 2ðdjaj þ daj as þ das s Þ. To keep the notation simple, we only consider the one way trip time and drop the multiplier 2. Consequently, the longest delay in a session D under assignment A is ΛD ðAÞ ¼ maxð λD;A ðjÞ j 8 j A CÞ. Similarly to [25], we define delay variation in session D under assignment A as ΔD ðAÞ ¼ maxðabsðλD;A ðjÞ
λD;A ðj0 ÞÞ; j 8 j; j0 A C; ja j0 Þ, where absð:Þ is the absolute value function. In summary, we define the SON design problem as follows. Given an application session D ¼ ðG; s; CÞ, a delay bound μ, and a delay variation bound υ, the problem is to find a minimal assignment A such that the delay constraints are satisfied. arg min jAj A

1.3. Contributions In [22], in a specialized decentralized network architecture, a SON network design problem is defined with a delay constraint among all pairs of clients, and an objective of minimizing the number of contact servers selected. A distributed heuristic algorithm called Zoom-In Zoom-Out is developed as a fast solution method. Our research extends the work presented in [22] in several ways: (1) we consider the selection of not only contact servers in the SON, but also the in-between servers that do not have any Internet connection; this is an important consideration, because a fully connected SON is expensive to install and operate, therefore a path from a contact server to the root server typically goes through several SON nodes, where data packets are checked and forwarded; (2) we consider a centralized application network architecture, which is more commonly utilized as it scales better

s:t: ΛD ðAÞ r μ ΔD ðAÞ r υ

2.2. Mathematical programming model Due to the assumption that the communication between a service node i and the root node k always takes place on the shortest path, in a preprocessing step we compute the shortest ! path between all pairs of service nodes and define a ki as a column vector of dimension n that represents the path. An entry i0 A V is equal to 1 if node i0 is on the path from i to k, and 0 otherwise (i.e. akii0 A f0; 1g 8 i0 A V). 2 To avoid confusion with the application server, henceforth we adopt the term service node as in [14].

76

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

We define binary decision variables Yjik that take the value 1 if user j is assigned to contact node i when k is the root node, 0 otherwise. Variable Yjik is associated with the path starting from a user j to its contact node i, and continuing with the shortest path to root node k. It is at this step that we impose the delay constraint by defining Yjik only if the delay on the path from j toi to k (djik) is less than the maximum delay μ. Similarly, binary decision variables Xi are defined to take the value 1 if service node i is selected in the solution (either as a contact node, or only on the shortest path from a contact node to the root node), and 0 otherwise. Binary decision variable Zk is used for selecting the root node. The complete formulation for the SON design problem with delay and delay variation is presented below. Model SON-DDV. minimize

∑ Xi:

ð1Þ

iAV

subject to :

∑ Zk ¼ 1

ð2Þ

kAV

∑ Y jik ¼ Z k

iAV

8 j A C; k A V

∑ ∑ ∑ akii0 Y jik r nX i0

kAV jAC iAV

ð3Þ

8 i0 A V

ðdjik
dj0 i0 k ÞðY jik þ Y j0 i0 k
1Þ r υ

8 kAV

8 j; j0 A C; j o j0 8 i; i0 A V Y jik ¼ 0 8 j A C; i A V ; k A V djik 4 μ X i A f0; 1g

8 iA V

ð4Þ ð5Þ

(5) are a major source of complexity due to the large number of them needed in an instance. For example, in a relatively small instance with 50 users and 10 service nodes, in worst case there will 10 be a total ð50 2 Þ 2ð 2 Þ10 ¼ 1; 102; 500 such constraints. In an actual application environment a solution is needed in almost real time. As soon as the users of a session are determined, the session is expected to start within a duration of less than a minute, which includes computation of a solution, actual configuration of the SON, and notification of users of their contact nodes. We define and study model SON-D as the service overlay network design problem with delay constraints only, which is basically model SON-DDV without the delay variation constraints (i.e. (1)–(4), (6), (7)), because it represents a broader problem that we would like to solve first before tackling with the more complex model SON-DDV. In addition, we would like to perform a theoretical computational complexity analysis to which SON-D better lends itself. We prove that SON-D is NP
hard (see Appendix), and so is SON-DDV by way of deduction. Therefore, we know that as the problem size increases to large realistic dimensions, it becomes practically impossible to solve either model with an optimal seeking algorithm such as branch and bound. In fact, a brief computational experiment with Gurobi software confirmed our impression that heuristic algorithms are needed to solve both SON-D and SON-DDV, especially at the level of computational speed that a typical interactive Internet application requires. 3.1. Solving SON-D

ð6Þ

ð7Þ

Z k A f0; 1g 8 k A V Y jik A f0; 1g 8 j A C; i A V; k A V In model SON-DDV, the objective function (1) measures the total number of service nodes that we have to utilize. Constraint (2) ensures that exactly one root node is selected. Subsequently, constraints (3) ensure that a user is assigned to exactly one contact node, and that the assignment of a user to a contact node is only with respect to the selected root node. Constraints (4) relate the assignment decisions to the node selection decisions. Any node that is on the path from a selected contact node to the root node has to be selected once, regardless of how many times it appears on such paths. The delay variation constraints are captured by (5). Suppose k is the root node; if clients j and j0 are assigned to contact nodes i and i0 respectively, then due to the fact that Y jik ¼ Y j0 i0 k ¼ 1, the left hand side (LHS) measures the differences in delay. If however, either j or j0 (or both) are assigned to a different contact node, then the LHS becomes either 0, or negative making the constraint redundant. On the other hand, the maximum delay constraints that we impose during generation of input data are captured by equations (6). Finally, constraints (7) describe binary nature of all the decision variables.

3. Solution approach Model SON-DDV is a large scale integer programming (IP) model, and as such it can be solved by an off the shelf optimization solver such as Gurobi [17]. However, the delay variation constraints

First, we exploit the fact that the number of service nodes n is relatively low and devise a straightforward divide-and-conquer approach, whereby we define SON-D(r) such that Z r ¼ 1 and Z k ¼ 0 8 k a r, and set SON
D ¼ minfSON
DðrÞjr A Vg. Then we simply solve SON-D(r) n times and identify the root node r (and the associated assignment solution) as optimal. Although this approach reduces the total computational requirements of Gurobi in solving SON-D considerably, the savings are still not sufficient to remove the need for a fast heuristic solution algorithm. Nevertheless, we will continue to use this base strategy in all algorithms we develop in this section. We develop Algorithm D–H, which is a construction type greedy heuristic of complexity Oðn3 pÞ, that solves problem SOND (see Algorithm 1 listing below). In the proposed algorithm, starting with a given root node r, we progressively add nodes to the SON, one at a time, from the pool of available nodes that have a direct link to any one of the nodes in the partial SON. To each added node, all users who satisfy the delay constraint are assigned permanently. At each iteration, the candidate node to which the most number of users can be assigned is added to the SON. With this process, we are assured to find a feasible solution with r as root node, if it exists. By comparing all the solutions with different root nodes, we select the one with fewest number of service nodes as our final solution. In the listing of Algorithm 1 below, lines 1–10 initialize a partial SON with root node r, and assign all users who satisfy the delay constraint to r. Line 14 creates the list of candidate nodes to add to the partial SON. The following loop in lines 15–22 identify the most desirable candidate (i.e. one that is assigned the most number of users). The selected node is then added to the partial SON together with its assigned users in lines 23–28. Algorithm 1. Algorithm D–H. Input: s, C, V, μ Output: assignment An

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

A’∅ foreach r A V do Ar ’fas ¼ r; aj ¼ Null 8 jA Cg 0 C ’∅ 0 V ’frg foreach j A C do if djrs rμ then A ’A ⋃fa ¼ rg r j r 0 C ’C 0 ⋃fjg end if jC 0 j ¼ m then jreturn An ¼ A r while jC 0 j o m and jV 0 jo n do 0 0 0 0 0 V ¼ fijði; i Þ or ði ; iÞ A E 8 i A V n V ; i A V g foreach i A V do cnt i ’0 foreach j A C do if djis r μ then j cnt i ’cnt i þ 1 end end n i ’arg maxi A V ðcnt i Þ foreach j A C do if djin s r μ then A ’A ⋃fa ¼ in g r r j 0 C ’C 0 ⋃fjg end 0 V ’V 0 ⋃fin g end if jC 0 j ¼ m then j A’A⋃Ar

32 end n 33 return A ¼ arg minAr A A jAr j

3.2. Solving SON-DDV Inspired by the work described in [5], we propose the following algorithm for partitioning and enumerating the solution space of model SON-DDV into subproblems, in which all included solutions of user to contact node assignments are known to be feasible with respect to delay and delay variation constraints. We call it the chain algorithm and utilize it as part of a complete solution framework. Consider r as the root node. We construct an empty list L and add items to L as follows. First, for all combinations of users jA C and contact nodes i A V, we compute djir, which is the delay from user j to root node r via contact node i. If djir is within the delay bound μ, we add the tuple ðj; i; djir Þ to L. When completed, L contains all possible delay-feasible assignments for each user. Next, we sort L by djir in non-decreasing order. For each element h A L, we scan toward the end of L until all users are covered, if successful we call the consecutive sublist as a chain [5]. (Failure occurs for h that are towards the end of L, when all the entries in L from h to the end do not cover all users.) We refer to the last item in the chain as t. Clearly, we can easily construct a delay-feasible solution from this chain by selecting one element (i.e. assignment) for each of the users, and the delay variation of this solution is bounded by the delay difference between users in the tuples h and t. We only need to consider the chains with delay variation less than or equal to the bound υ.

77

Numerical Example 1.a: Let the set of service nodes in G be V ¼ fv1 …v10 g and the set of users be C ¼ fc1 …c4 g. We skip the listing of delay data, specification of the root node r, and the detailed network topology for the sake of brevity, and show below the sorted list L created by the chain algorithm. For example, notice that all possible assignments of c2 are in indexes 1, 3, 8, 10 for assignment to contact nodes v2 ; v3 ; v4 ; v1 respectively. On the other hand, there are only two entries for c1 in index 6 and 7 for assignment to nodes v3 and v1 respectively. The possible assignments of c1 to all other nodes are infeasible due to delay that is longer than μ, therefore they do not appear in L. Also note that nodes v5-v10 are not directly accessible by users (due to long delay), however they might be on the shortest path from the accessible nodes to the root node. L index 1 User c2 Contact Node v2 Delay 25

2 c3 v2 32

3 c2 v3 32

4 c3 v4 37

5 c4 v4 39

6 c1 v3 39

7 c1 v1 41

8 c2 v4 43

9 c3 v1 51

10 c2 v1 52

11 c4 v1 52

We pick up the first element of L as head, and scan toward the end of L until all clients are covered. This results in the first chain that we evaluate from index 1 to 6, with delay variation of 39
25¼ 14. Based on this chain, one possible delay-feasible assignment is c2 -v3 ; c3 -v4 ; c4 -v4 ; and c1 -v3 . (We skip the computation of the total number of selected server nodes – including intermediate nodes – for the sake of simplicity.) Continuing with the second element of L, the second chain is identified as entries of L from 2 to 6, with delay variation of 39
32¼ 7. The third chain starting at index 3 also concludes at index 6, resulting in the same amount of delay variation of 7. The following chain goes from index 4 to 8, with a delay variation of 43
37 ¼6. Skipping several chains, it is easy to see that the last one is from index 7 to 11, with 52
41 ¼ 11 units of delay variation. A chain l constitutes a subproblem of SON-DDV such that the root node decision is set to r, and the contact node assignments are limited to those that are in l. We develop the following formulation that describes a subproblem associated with a chain. Let Yji be a binary decision variable that is set to 1 if client j is assigned to contact node i in l, and 0 otherwise. Similarly, as in Section 2.2, let Xi and aki be the binary service node selection decision variables, and the path description parameters respectively. Model SON-DDV (r, l) minimize

ð8Þ

∑ Xi

iAV

subject to :

∑ Y ji ¼ 1

ðj;iÞ A l

∑ arii0 Y ji r nX i0

ðj;iÞ A l

8 jAC

8 i0 A V

X i A f0; 1g

8 iA V

Y ji A f0; 1g

8 ðj; iÞ A l

ð9Þ ð10Þ ð11Þ

Model SON-DDV(r, l), for a given root node r and a chain l, minimizes the total number of service nodes selected (8). Constraints (9) ensure that each user is assigned to exactly one contact node, and (10) relate assignment decisions to node selection decisions by ensuring that all contact nodes plus all non-contact service nodes that are on the shortest path from a contact node to the root node are selected. Finally, constraints (11) describe the binary nature of the decision variables.

78

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

Finally, we describe a complete solution algorithm DDV-IP that enumerates all chains and solves subproblems defined by model SON-DDV(r,l) for each chain. It runs for every possible root node selection, and concludes with the optimal solution to problem SON-DDV (see Algorithm 2 listing below.) Algorithm 2. Algorithm DDV-IP.

1

Input: s, C, G, delay bound μ, delay variation bound υ Output: Assignment A Create an empty assignments A

10

foreach r A V as the root node s do Construct and sort L ¼ fðj; i; djir Þjdjir rμg foreach valid chain l A L do 0 Create an assignment A from l by optimally solving SON
DDVðr; lÞ if A is empty or jA0 jr jAj then j Assignment A ¼ A0 end end end

11

return A

2 3 4 5 6 7 8 9

Essentially, algorithm DDV-IP decomposes one very large problem into smaller subproblems that are relatively easy to solve. As such, it has a significant advantage over solving model SON-DDV directly. Recall the example of an instance with 10 nodes and 50 users that would generate more than 1 million constraints in SONDDV. For a given r, the list L would have 500 (np) entries. In worst case, there would be 450 chains (each containing 50 entries), and as many instances of model SON-DDV(r,l) to solve. Repeating this for 10 alternative root nodes brings the total to 4500 subproblems to be solved. This is computationally much more manageable compared to solving SON-DDV directly, as was confirmed by pilot runs. Furthermore, the shorter the chains, the easier it is to solve the subproblems. For instance, a chain with 50 entries from the above example, would include only one assignment per user yielding a trivial subproblem. On the other hand, if the chains are long with many alternative assignments per user, then there would be a few, but more difficult to solve, subproblems. Still, algorithm DDV-IP is an exact algorithm and is not suitable for solving large instances. Therefore, we develop a heuristic algorithm in the next section. 3.2.1. A heuristic method In the framework of algorithm DDV-IP, the solution of subproblems is an obvious area in which we can drastically reduce computation time with heuristic methods. In fact, by varying the extent of search in the solution of a subproblem, we can capture the trade off between computational efficiency and solution quality in a controlled manner. Our research objective is to solve large scale problems very quickly. Therefore, with mostly computational efficiency in mind, in solving the subproblems we simply assign a user to a contact node with the shortest delay value. We refer to the resulting heuristic algorithm DDV-H. (The description of DDV-H is the same as that of DDV-IP except for the solution method specified for SON-DDV(r,l) on line 5.) Numerical example 1.b: Consider example 1.a and the first chain listed (indexes 1–6). Algorithm DDV-H computes the following assignment: c2 -v2 (it is the shorter delay compared to c2 -v3 ); c3 -v2 (shorter delay compared to c3 -v4 ); c4 -v4 ; and c1 -v3 . The myopic nature of DDV-H can easily be observed in this particular instance. For example, it could be that the shortest path from v2 to root node r goes through all not-directly accessible nodes v5
v10 , whereas nodes v3 and v4 could have direct connections to r. In this

case, the heuristic would clearly generate a suboptimal solution with respect to minimizing the total number of selected nodes. As a second example, consider the second chain (indexes 2–6). The heuristic-computed assignment is c3 -v2 (shorter compared to c3 -v4 ); c2 -v3 ; c4 -v4 ; and c1 -v3 . As before, we skip the computation of the total number of selected server nodes – including intermediate nodes – for the sake of simplicity. We conduct a complexity analysis for algorithm DDV-H as follows. Creating L is O(np) and sorting it takes Oðnp log npÞ time. Finding a chain starting with a particular element as its head takes at most O(np), and there are at most O(np) chains. The resulting complexity is Oðn2 p2 Þ since the size of L is at most np. Next, creating an assignment from a chain takes at most O(np) time. The operations above are executed n times (one for each possible root node), hence the overall complexity is OðnÞ  ðOðnpÞ þ Oðnp log npÞ þ Oðn2 p2 Þ þ OðnpÞÞ ¼ Oðn4 p2 Þ. The two DDV-* algorithms represent the extremes in the trade off between computational efficiency and solution quality: a slow and exact algorithm, and a fast and relatively inaccurate heuristic.

4. Computational study 4.1. Experimental Setup We start with randomly generating SON instances utilizing the Tiers [8] software, which is designed to generate graphs that have properties of the Internet such as locality and hierarchy. The Tiers software places the nodes of the SON on a two dimensional plane and measures the delay (in Euclidean distance metric) between nodes where there is a direct link. During our use of Tiers, we kept the parameter settings at default values. In order to simulate the speed advantages of a SON network (compared to the public Internet), we apply a discount factor α ¼ 0:7 to the delay values of the SON links. On the same plane of the SON, we randomly place a given number p of users, and create links from all users to all service nodes with delay values computed by the Euclidean distance. The location of the application server is set in a similar fashion. In Fig. 3, two network instances are shown, where large light-colored nodes are SON nodes, small ones are user locations, and the large dark-colored node is the application server. The setting of the maximum delay μ and the delay variation v require care to make sure that we do not create infeasible instances. First, we compute the minimum possible delay τ with the following algorithm: for a given root node k, build a shortest path spanning tree covering all users, and compute the maximum delay τk; repeat for all candidate root nodes and conclude with the minimum delay τ ¼ mink A V τk . (The observed τk values varied between 10.2 and 13.8.) Next, set μ ¼ 1:1τ, and v ¼ μ=3, which are reasonable values in a real application, and they consistently produced feasible instances in pilot runs. In the experiment, we consider SON sizes of n¼(10, 20, 30, 40, 50), and user-set sizes of p¼ (50, 80, 100, 200, 300, 400, 500), creating a total of 35 problem sizes (n, p). Furthermore, for each size, we generate a total of 30 random instances, which makes the total number of instances we solve 1050. The algorithms and the IP Models are implemented in C/Cþ þ with Gurobi optimizer library version 4.6.1 [17] when needed, and compiled with the GNU Cþ þ compiler. The experiments are conducted on a 16-core Intel Xeon (E5520 CPU at 2.27 GHz) machine with 12 GB of RAM running Linux 3.0 kernel. The Gurobi solver is configured to take advantage of multiple cores automatically. 4.2. SON-D Fig. 4 displays the optimal solution values (i.e. number of SON nodes selected) as computed by solving model SON-D with Gurobi,

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

79

Fig. 3. Example network instances generated with Tiers. (a) n¼ 30, p ¼ 200 and (b) n¼50, p¼ 500.

Solution Quality of Heuristic D−H

50 80 100 200 300 400 500

50 80 100 200 300 400 500

Optimal Heuristic

n=10

n=20

n=30

n=40

n=50

Number of SON nodes selected

12

10

8

6

50 80 100 200 300 400 500

50 80 100 200 300 400 500

50 80 100 200 300 400 500

4

Number of Clients (p)

Fig. 4. Solving SON-D: performance.

and the feasible solution values computed by Algorithm D–H for all problem sizes. Each point on the chart is the average of 30 instances solved for the corresponding problem size (n, p). Overall, the heuristic appears to perform very well. The performance gap between the optimal and the heuristic solutions, measured as the number of additional SON nodes, is an average of (0.06, 0.78, 0.22, 0.03, 1.04) for n¼ 10, 20, 30, 40, 50 sizes respectively. In problems with n ¼10, 30 and 40, heuristic solution values are only marginally higher than that of the optimal values, reflecting the fact that in many of the 30 instances solved per size, the heuristic matched the optimal solution value. In the other cases, it is mostly within 1.0 of the optimal solution meaning that we would have to utilize only one extra service node (i.e. server computer) to support the subject application, when a solution with one fewer node exists. Clearly, this one extra node becomes less significant as the optimal number of nodes selected increases with an increase in the SON size n. One significant observation about the nature of solutions (optimal or heuristic) is that, the number of users p does not seem to have a consistent impact on the solution value. Partly, this is an artifact of the

way we generate the instances. For the same SON size n, at different values of p, the user location data is generated randomly using different random number seeds. For example, the only commonality between (n¼50, p¼200) instances and (n¼ 50, p¼500) is the SON network data. Therefore, while we observe that at n¼50, we need a lesser number of SON nodes when we have 500 versus 200 users, we also observe the opposite outcome at n¼ 30 for the same p values. Nevertheless, this observation should not be totally surprising. Consider a SON configured to accommodate a relatively large number of users already. As new users are added to this SON, they are likely to be located closely to some of the existing users, and be easily accommodated without having to bring another SON node online. Still, even in these situations, a reconfiguration of the SON (i.e. alternative optimal solution) may be triggered. The only time when we would utilize a heuristic solution method is when we cannot afford the computational requirements of an optimal one. As we stated previously, in a typical production environment a solution is needed within 1 min (60 s) at most. Therefore, we set 60 s as the feasibility of computational requirement of any solution method we apply. Fig. 5 shows the run time of the optimal and the heuristic methods, side by side rather than overlayed due to mismatches in scale, with the 60 s line drawn on both plots to serve as a common reference. In all but the three smallest sizes (n¼10, p¼50, 80, 100) the computational requirements of the optimal solutions are infeasible (i.e. longer than 60 s). On the other hand, the heuristic is computationally feasible in all but the two largest problem instances (n¼50, p¼400, p¼500) with requirements of 73 and 87 s respectively. Therefore, the optimal solutions are of referential value only to assess the quality of the heuristic solutions, as they could not be computed fast enough. Considering that requirements of most of the problem sizes have 1 or 2 orders of magnitude higher than our limit, the optimal seeking methods are likely to remain infeasible even with the fastest modern hardware. The charts in Fig. 5 also show the impact of problem size (n, p) on the computational requirements. The main driver appears to be n, and p at high n values. 4.3. SON-DDV The model SON-DDV is not practical to solve with an off the shelf commercial solver. As observed previously, even for the smallest problem size of n¼10, p¼50, the number of delay variation constraints could exceed 1 million. A few pilot runs confirmed our belief. Therefore, in our analysis we study only Algorithm DDV-IP

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

n=10

n=20

n=30

50 80 100 200 300 400 500

Computational Performance of Algorithm D−H

50 80 100 200 300 400 500

50 80 100 200 300 400 500

Computational Performance of IP Model

n=40

n=50

n=10

n=30

n=40

n=50

80

Computing time (seconds)

5000

Computing time (seconds)

n=20

50 80 100 200 300 400 500

80

4000

3000

2000

60

60 seconds limit

40

20 1000

60 seconds limit

50 80 100 200 300 400 500

50 80 100 200 300 400 500

50 80 100 200 300 400 500

50 80 100 200 300 400 500

50 80 100 200 300 400 500

Number of Clients (p)

50 80 100 200 300 400 500

0

0

Number of Clients (p)

Fig. 5. Solving SON-D: computational requirements. (a) Optimal and (b) heuristic.

Solution Quality of Heuristic DDV−H

n=10

n=20

n=30

500

400

300

200

100

80

50

400

500

300

200

100

80

50

Optimal Heuristic

n=40

Number of SON nodes selected

14

12

10

8

6

500

400

300

200

100

80

50

500

400

300

200

100

80

50

4

Number of Clients (p)

Fig. 6. Solving SON-DDV: performance.

(optimal) and Algorithm DDV-H (heuristic). Compared to problem SON-D, it is substantially more challenging to solve SON-DDV even with the novel partitioning approach we developed. Consequently, we were limited in the problem sizes we could solve. Specifically, the largest problem size solved was (n¼40, p¼400), which took a few weeks and 1 day to solve with DDV-IP and DDV-H respectively. Fig. 6 shows the solution values computed by both algorithms. As before each point is the average of 30 instances with the exception of (n ¼40, p ¼400), of which we could collect only 9 data due to computations taking too long. With all instances, Algorithm DDV-IP ran until it exhausted the solution space, hence concluding optimality. This time, the performance gap between the optimal and the heuristic solutions is larger at an average of 1.7, 3.7, 2.2, 1.8 for n ¼10, 20, 30, 40 sizes respectively. One favorable observation is that the performance gap appears to have no significant correlation with problem size, therefore making the relative performance of the heuristic better with larger instances.

In interpretation of the performance results, we also need to keep in mind that Algorithm DDV-H is developed with only computational speed in mind, and as such represents the lower bound (worst possible) in performance within the application of the solution framework we developed. Solving subproblems with a computationally more expensive heuristic, for example, a tailored version of Algorithm D–H, could easily improve the performance gap at the expense of increasing computational requirements. (Algorithm D–H could simply be adapted to solve SON-DDV(r,l) by limiting the assignment of users to nodes – in lines 8 and 25 of listing Algorithm 1 – to those that are in l only.) A significant advantage of our solution framework is that each subproblem can be solved independently in parallel. In addition, an algorithm repeats the same steps for each candidate root node location, providing even more opportunities for parallelization. Therefore, it is straightforward to modify DDV-H to take full advantage of all the 16 cores in our computing hardware, and achieve an almost linear speedup. Please note that the Gurobi software when solving the subproblems in DDV-IP, does take advantage of multiple cores in exploring a branch and bound search tree, therefore DDV-IP is already parallelized to a certain extend. We have the same feasibility criterion of having to solve problem SON-DDV within 60 s. However, with parallelization of the heuristic in mind, we set the acceptable computational time of the heuristic, as we ran it without parallelization, to 1000 s (1000 s/16 cores - assuming full linear speedup – equals 62.5 s). Fig. 7 shows the computational requirements of both the optimal and the heuristic with an overlayed 50,000 s line providing a common reference to put the relative scales in perspective. An immediate observation is that the number of users p has a drastic impact on the computation time on SON sizes of n Z 30. This is partly due to the fact that the values of p are one order of magnitude larger than that of n (100 s versus 10 s). Even with the heuristic algorithm, we enumerate and solve all possible subproblems, the number of which increase very rapidly in n and p. It is clearly apparent in Fig. 7 that we cannot solve all problem sizes within the required computational time limit. In Fig. 8, we show the size and the computing time of the problems that we could solve (with both algorithms) within the required time limit of 1000 s. The optimal method could solve the smallest 3 sizes (n¼10, p¼50, 80, 100), within 294, 616, and 953 s respectively. Because we cannot further parallelize the optimal

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

n=10

n=20

n=30

n=40

n=10

n=30

500

200

400

300

80

50

n=20

100

500

400

200

100

300

80

50

500

400

300

Computational Performance of Algorithm DDV−H 200

100

80

50

500

400

300

200

80

100

50

Computational Performance of Algorithm DDV−IP

81

n=40

3e+06

Computing time (seconds)

Computing time (seconds)

80000

2e+06

1e+06

60000 50,000 seconds

40000

20000

50,000 seconds

400

500

300

200

80

100

50

400

300

500

200

80

50

400

300

500

200

100

80

50

500

400

300

80

200

50

100

Number of Clients (p)

100

0

0e+00

Number of Clients (p)

Fig. 7. Solving SON-DDV: computational requirements. (a) Optimal and (b) heuristic.

Computational Performance − solvable within 1,000 seconds

n=10

n=20

n=30

500

400

300

200

100

80

50

500

400

300

200

100

80

50

Optimal Heuristic

n=40

Computing time (seconds)

800

600

400

500

400

300

200

100

80

50

500

400

300

80

200

100

50

200

Number of Clients (p)

Fig. 8. Solving SON-DDV: solvability.

algorithm easily, it is not feasible to utilize it even for these problem sizes. The heuristic is usable in solving 15 out of 28 problem sizes. As n increases, the number of solvable p levels decrease. This section reveals the computational complexity of the SONDDV problem. Even with a novel method of enumerating the solution space and using substantial amount of computing resources, the combinatorial aspect of the problem overwhelms. Still, we were able to compute optimal solutions to assess the performance of any heuristic method with. The proposed heuristic sets the limits of solvability of this problem in an actual application. Evidently, more work is needed to improve the computational efficiency.

5. Summary and conclusion In this paper, we study the service overlay network (SON) design problem to support interactive Internet applications with

delay and delay variation constraints. The decision problem is to find the minimum number and location of SON nodes (i.e. server computers), as well as the assignment of participating users and the server (of the application) to the SON network. Feasibility is characterized by quality of service measured by a limit on maximum delay and the maximum variation of delays in all two way communications between users. We consider two versions of the problem: SON-D, which captures only the maximum delay constraint and is applicable to a broader variety of applications; and SON-DDV, which includes both the delay and the delay variation constraints and is specifically suited for applications that involve competition between users. Both are difficult to solve combinatorial problems that we proved to be NP
hard. We developed IP models to describe both problems, and to compute optimal solutions for a set of benchmark problem instances. In addition, we developed heuristic solution algorithms that proved to be fast enough to be useable in a real application environment, and produced measurably good quality solutions to the benchmark instances. The SON-D problem that is defined on a decentralized network architecture is already described and solved in the literature. However, this is the first and only reported research that studies it in a centralized network model. On the other hand, the SON-DDV problem is first described and solved in our research. Therefore, our contributions are to solve two difficult problems that were not solved before. Problem SON-D was relatively easier to solve. We solved instances as large as having 50 SON nodes and 500 users optimally, and all but the largest two heuristically within 60 s of computing time. One possible improvement is to parallelize the heuristic to take advantage of the multiple computing cores that are available in all recent CPUs. For example our computing hardware has 16 computing cores. In order to solve the very challenging problem SON-DDV, we developed a novel method to partition and enumerate the feasible solution space into small subproblems. Picking a method to solve these subproblems involves evaluating a trade off between solution quality and computational requirements. We considered two extremely opposite algorithms: optimal, which solves an IP model for each subproblem; and a fastest possible heuristic that computes a quick and dirty solution without much consideration to the objective function. The optimal algorithm sets the size of optimally solvable problem sizes, where the initial IP model failed. The heuristic, on the other hand, sets

82

Y.-R. Chen et al. / Computers & Operations Research 57 (2015) 73–82

the size of problems that can be solved within the computational requirements of an actual application, under the assumption that it can be parallelized and fully utilize 16 computing cores. Therefore, parallelization of any heuristic that solves SON-DDV is critically important to cope with the huge size of the solution space. Another approach to manage the size of the search space is to apply heuristics to select only a subset of the subproblems that are subsequently solved. An important advantage of the proposed solution framework is that additional side constraints such as limited bandwidth capacity on SON links can be incorporated with little additional computational effort. Our future work involves incorporating additional constraints to the SON-DDV problem, developing better performing heuristics, and tailoring these algorithms to solve the logistics network design problem with cross docking. Appendix Theorem 1. Problem SON-D is NP
hard. Proof. We use a similar method as shown in Lee et al. [22] to prove that SON-D is NP
hard by reduction from Set Covering Problem (SetCover). Given a family of n finite sets fV j g ¼ fV 1 ; V 2 ; ⋯V n g. The SetCover problem is to find a subfamily fT h g D fV j g such that ⋃ T h ¼ ⋃ V j and the cardinality of fT h g is minimum. The set covering problem is known to be NP
hard. We show that SetCover is polynomial-time reducible to a special case of SON-D. Let the universe of SetCover be ⋃V j ¼ fC 1 ; C 2 ; ⋯C p g. First, we construct a network with a dummy node r as the root node, and let the rest of the service nodes be v1 ; v2 ; ⋯vn . This network is a star graph and the set of service nodes is fr; v1 ; v2 ; …vn g. Now, we set the delay values on all the edges from v1 ; v2 ; …vn to r as 1. Let vj correspond to Vj, 1 r j rn. Next, we add p user nodes to the network (c1 ; c2 ; …cp ). Let ck corresponds to Ck, 1 rk rp. There are edges between all pairs of (ck, vj) (recall that the service nodes and user nodes form a complete bipartite graph.) Set the delays on these edges as follows. First, we set the delay of ðck ; rÞ to 3 for all ck. If C k A V j , the delay between ck and vj is 1. Otherwise, it is set to 3. Finally, we add the server s to the network with delay to r set to 0, and set to 5 to other vj's. Note that we can either assume that there are no edges between ck and r; s or there are edges but with the delay of a large value such as 5. Now, an instance of SON-D is constructed with the set of service nodes fr; v1 ; v2 ; ⋯vn g, the set of user nodes fc1 ; c2 ; ⋯cp g and edges as described earlier. The delay bound is set to 2, which is the maximum allowed from a user node k to node r in the solution. The construction of this problem instance can be done in Oðn þ npÞ. In this instance, r is selected as the root node in the solution. The set of service nodes (vj's) chosen in an optimal solution of the SON-D problem also provides the solution to the SetCover problem instance by simply mapping vj to Vj. Hence SON-D is NP
hard. □ References [1] Adami Davide, Callegari Christian, Giordano Stefano, Pagano Michele, Pepe Teresa. Optimal design of service overlay networks with economic and performance constraints. Int J Commun Syst 2010;23(March (3)):369–89. [2] Armitage Grenville. An experimental estimation of latency sensitivity in multiplayer Quake 3. In: The 11th IEEE international conference on networks, ICON2003; 2003.

[3] Grenville Armitage, Mark Claypool, Philip Branch. Networking and online games: understanding and engineering multiplayer internet games. West Sussex, England: John Wiley & Sons; 2006. [4] Banerjee Suman, Kommareddy Christopher, Kar Koushik, Bhattacharjee Bobby, Khuller Samir. Construction of an efficient overlay multicast infrastructure for real-time applications. In: Proceedings of IEEE INFOCOM; 2003. p. 1521–31. [5] Banik Shankar M, Radhakrishnan Sridhar, Sekharan Chandra N. Multicast routing with delay and delay variation constraints for collaborative applications on overlay networks. IEEE Trans Parallel Distrib Syst 2007;18(3):421–31. [6] Boysen Nils, Fliedner Malte. Cross dock scheduling: classification, literature review and research agenda. Omega-Int J Manage Sci 2010;38(December (6)):413–22. [7] Bray David A, Konsynski Benn R. Virtual worlds: multi-disciplinary research opportunities. SIGMIS Database 2007;38(4):17–25. [8] Calvert KI, Doar MB, Zegura EW. Modeling Internet topology. IEEE Commun Mag 1997;35(June (6)):160–3. [9] Campbell James F, O'Kelly Morton E. Twenty-five years of hub location research. Transp Sci May 2012;46(2):153–69. [10] Chen Kuan-Ta, Huang Polly, Lei Chin-Laung. Effect of network quality on player departure behavior in online games. IEEE Trans Parallel Distrib Syst 2009;20 (5):593–606. [11] Cisco Systems, Inc. Quality of Service Design Overview—QoS requirements of VoIP. [Online; accessed 2 September 2012]. [12] Claypool Mark, Claypool Kajal. On latency and player actions in online games. Commun ACM 2006;49(11):40–5. [13] Dommel Hans-Peter, Garcia-Luna-Aceves Jose Joaquin. Efficacy of floor control protocols in distributed multimedia collaboration. Clust Comput 1999;2(1): 17–33. [14] Duan Zhenhai, Zhang Zhi-Li, Thomas Hou Yiwei. Service overlay networks: SLAs, QoS, and bandwidth provisioning. IEEE/ACM Trans Netw 2003;11 (December (6)):870–83. [15] Zanjirani Farahani Reza, Hekmatfar Masoud, Boloori Arabani Alireza, Nikbakhsh Ehsan. Hub location problems: a review of models, classification, solution techniques, and applications. Comput Ind Eng 2013;64:1096–109. [16] Ferretti Stefano. A synchronization protocol for supporting peer-to-peer multiplayer online games in overlay networks. In: DEBS '08: proceedings of the second international conference on distributed event-based systems. New York, NY, USA: ACM; 2008. p. 83–94. [17] Gurobi Optimization, Inc. Gurobi Optimizer. [Online; accessed 27, August-2012]. [18] Hua Chu Yang, Rao Sanjay G, Seshan Srinivasan, Zhang Hui. Enabling conferencing applications on the internet using an overlay multicast architecture. In: Proceedings of ACM SIGCOMM; 2001. p. 55–67. [19] Hua Chu Yang, Rao Sanjay G, Seshan Srinivasan, Zhang Hui. A case for end system multicast. IEEE J Sel Areas Commun 2002;20(8):1456–71. [20] Kara Bahar Y, Tansel Barbaros Ç. The latest arrival hub location problem. Manage Sci 2001;47(October (10)):1408–20. [21] Knutsson Bjorn, Lu Honghui, Xu Wei, Hopkins Bryan. Peer-to-peer support for massively multiplayer games. In: INFOCOM 2004. Twenty-third annual joint conference of the IEEE computer and communications societies, vol. 1; March 2004. p. 96–107. [22] Lee Kang-Won, Ko Bong-Jun, Calo Seraphin. Adaptive server selection for large scale interactive online games. In: NOSSDAV '04: Proceedings of the 14th international workshop on network and operating systems support for digital audio and video. New York, NY, USA: ACM; 2004. p. 152–7. [23] Nichols James, Claypool Mark. The effects of latency on online Madden NFL football. In: Proceedings of the 14th international workshop on network and operating systems support for digital audio and video, NOSSDAV '04. New York, NY, USA: ACM; 2004. p. 146–51. [24] Oliveira Carlos AS, Pardalos Panos M, Resende Mauricio GC. Optimization problems in multicast tree construction. In: Resende Mauricio GC, Pardalos Panos M, editors. Handbook of optimization in telecommunications. US: Springer; 2006. p. 701–31. [25] Rouskas George N, Baldine Ilia. Multicast routing with end-to-end delay and delay variation constraints. IEEE J Sel Areas Commun 1997;15(April (3)):346–56. [26] Shi Sherlia, Turner Jonathan. Placing servers in overlay networks. In: Symposium on performance evaluation of computer and telecommunication systems (SPETS); 2002. [27] Svoboda Philipp, Karner Wolfgang, Rupp Markus. Traffic analysis and modeling for world of Warcraft. In: IEEE International conference on communications; 2007. p. 1612–7. [28] Tuan Tran Hung, Bessler Ford. A framework toward optimal operation of service overlay network assuring inter-domain QoS. In: Proceedings of the international network optimization conference. INOC; 2005. p. 1–67. [29] Zhang Meng, Sun Lifeng, Fang Yechang, Yang Shiqiang. iGridMedia: the system to provide low delay peer-to-peer live streaming service over internet. Peerto-Peer Netw Appl 2010;3(3):175–85.