- Email: [email protected]
Models for calculating end-to-end delay in packet networks
ITC 18 / J. Charzinski, R. Lehnert and P. Tran-Gia (Editors) 9 2003 Elsevier Science B.V. All rights reserved.
1231
M o d e l s for Calculating E n d - t o - E n d D e l a y in Packet N e t w o r k s Olav Osterbo Telenor R&D, Snaroyveien 30, N-1331 Fornebu, Norway E-mail: [email protected]
Abstract. In this paper we consider an analytical model to calculate end-to-end delays in packet networks. To be more specific we consider a particular path in the network and we approximate the end-to-end delay by assuming that the waiting times in each node are independent and that both the input and output streams at each node are Poisson processes. With these assumptions (approximations) the end-to-end delay yields the convolution of the waiting times in each queue. If the service times in each queue are identically distributed the corresponding convolution may be heavily simplified, and closed form expressions are derived in terms of derivative with respect to the load parameter. We put special emphasis on the case with constant service times since this is an important case for applications. Numerical examples show that end-to-end delays in chains up to 20 nodes may be analysed without numerical difficulties. We also give the corresponding results for a chain containing two groups of links with different capacity in each group. This is a particularly interesting case and makes it possible to model a path in an IP network that includes both access and core links. Numerical examples are given and discussed for the various models.
1
Introduction
There is reason to believe that real time services will be a significant part of the traffic offered in future multi service networks. Real time services require a quite regular bit stream delivered at the receiver's site to maintain the necessary quality for various applications. It is well known that networks based on statistical multiplexing (like IP-networks) will introduce certain disturbance (jitter) in the bit stream mainly due to queuing in routers (or switches). These disturbances will add on along the path from the sender to the receiver, explaining the necessity of some kind of de-jitter buffer at the receiver site to compensate for these variations. The end-to-end delay is therefore an important parameter not only for dimensioning the de-jitter buffers, but also for providing some upper bounds of the total network delay for particular services and it is among the most important QoS (Quality of Service) parameter in networks deploying statistical multiplexing.
1.1
QoS parameters in the SLA (Service Level Agreement)
To realise many types of services the traffic flow will have to cross one or more administrative domains with their own SLAs. Such domains could for instance be 9 access networks with rather low capacity based on different wireless or DSL (Digital Subscriber Line) technologies 9 core networks with high capacity links but with large numbers of routers The end-to-end QoS is therefore realised through the contributions from different domains, and QoS guarantees end-to-end will be realised through different SLAs between the customer and the access network and/or between different core network domains. For each domain it will be important to estimate the contribution to the QoS parameters since each administrative
1232 domain will be responsible for their own contribution through the SLA. The most important parameters will be delay, jitter and information losses due to buffer overflow. It will be important for a network operator to be able to estimate the QoS parameters in his own domain to set the appropriate parameters in the SLAs. In addition it will be of vital importance to implement the necessary control structures that makes it possible to maintain the guarantees especially in his own domain end- and thereby preventing degradation in QoS which is often seen in best effort networks of today [1 ], [2]. The reference configuration for the modelling of the queuing behaviour consists of an upstream access network part, a multi-hop core network and a downstream access network part as illustrated in figure 1. Source application
Destination application Access part (upstream)
Access part (downstream)
Figure 1. Overview of the reference configuration." upstream access network parts, core network part (multi-hop) and downstream access part between the source and destination application.
1.2
Main contribution
The main contribution of this paper is to provide some approximations to calculate the distribution of end-to-end delay in packet based networks, and thereby gain better insight into the important question of how to design the control structure to maintain necessary QoS for delay-critical services. The main focus will be on real time traffic where we assume that packets of constant lengths are generated and shall traverse a given number of nodes (routers) to reach the receiving destination. The end-to-end delay calculations are based on the approach using convolutions. This means that the delay components in the different router queues are taken to be independent. Unless for some specific cases this assumption will not be valid and therefore the resulting model will be approximative. It is, however, reason to believe that if the actual load from each stream is small compared with the total load on the network elements considered, this assumption will be quite reasonable and provides the corresponding end-to-end delay distribution quite accurate.
2
Convolution of a given numbers of M/G/I queues waiting times having identical service time distributions
In the following we consider a model to calculate the end-to-end delays in a large scale IPnetwork. The aim is to calculate the distribution of the end-to-end delay for a particular path consisting of a series of routers. We assume that all the nodes in the end-to-end path are statistically independent; this is a key assumption to obtain the end-to-end delay by convolution. For queueing networks with FCFS (First-Come First-Served) queueing discipline this property only yields for acyclic form of Jackson Networks (where a packet visits a node at most once), see [3]. In [4], however, it is argued that if the load from a particular flow only is a small fraction of the total amount of traffic in a node and the input processes to the network is 'smoother than Poisson' (i.e. with less variability) then the independent assumption will be quite reasonable and will represent a worst case scenario. We shall therefore use the M/G/1 queue as the model to obtain the waiting time distribution in each node and then apply the convolution to obtain the end-to-end waiting time distribution. We are interested in the sum of waiting times in a series of K queues and we denote the sum W = W~ +... + WK . If all the waiting times may be taken to be independent the PDF
1233 (Probability Density Function) of the sum yields the convolution of the waiting times in each queue. The LST (Laplace-Stieltjes Transform) of the convolution (of waiting times for all the K queues) yields the product: K
I'V(s, A ,.-., PK ) = 1-'I I~(S, Pk )
(1)
k=l
where
l'V(s, Pk) =
1 - Pk
1- pk/~k (S)
is the LST of the waiting time for the k th queue (known as
the Pollaczek-Khinchin formula [5]), and where Pk is the load, /~k(s) is the LST of the remaining service time (given by/~k (s) = (1 -s B)k
where B k (s) is the LST of the service time
and b k = E [ B k ]is the mean service time), k = 1..... K . Generally it is possible to obtain the end-to-end distributions above by inverting the LST (1) numerically. Abate & Whitt have given a method to calculate the inversion integral based on Poisson's summation formula. The result yields an alternating series that might be difficult to use to determine the tail of the distribution and thereby obtaining the desired quantiles, (see [6]). Other methods such as the Saddle Point method (Large Deviation) and even more effective, the Uniform Asymptotic Approximation (UAA) [7] are also available to find approximations of the inversion integral of the LST (1), (see [8] for a survey of different methods). In [4] a heuristic method is proposed to find the quantiles of the convolution above if all the queues are identical which turn out to be very effective in terms of computation effort. 2.1
Convolution of waiting times in M/G/1 queues all having identical service time distributions In the following we shall assume that all the nodes have identically service time distributions, implying /}k (s) = /}(s) ; k = 1..... K . It turns out that in this case it is possible to obtain substantial simplification of the convolution (1). Theorem 1. If all the loads of the different M/G/1 queues all are distinct, that is Pi :r Pj for all i , j = 1..... K , then the LST of the convolution can be written as a weighted sum of the individual LST for each queue as follows: ,-,., K K l_pl (2) W(s, p, ,..., PK ) = ~'~ CkW'(S, Pk ) where c, = 1--I 1 p , / p k k=l
l=l,l~k
only depend on the loads in the different queues. Further the DF (Distribution Function) of the convolution has similar form K
W(x, p, ,..., p~ ) = Z ~ W(x, p~ )
(3)
k=l
where W(x,Pk ) is the DF for the k' th queue.
_Proof By taking partial fractions expansion of the LST (I) and taking B(s) as the free variable, n Often we are interested in the case where the loads on the different queues are equal. This result is possible to obtain relatively easily from Theorem 1 by letting pk ~ p for all k = 1..... K , (or by direct differentiation). We therefore get the following results when convoluting the waiting time for K identical M/G/1 queues:
1234
Theorem 2. We have =
Opt-' L1- p
_
'r
=
-
[1- p
O p K-'
W(x, p)
}
W(s, p) and
(4)
}
(5)
We can now state the more general theorem where we consider the general case where only some of the queues may be equally loaded. Theorem 3. Suppose we have total N groups of size n/ of queues that are equal loaded and p / r p / for all i , j = 1,...,N and K =n~ + .... + n
N.
Then the LST of the convolution may be
written:
/ W(s,- p, ,..., PN,", ,'","N) = ]--[U
1-- p~
*)=~
1 - p/B(s)
/i
= ~ ~ %. /--, i--,
(6) 1 - p/i~(s)
where the coefficients % only depend on the loads in the different queues and are given by:
d "~-'
(-1)"J -i l - p /
% = (n/-i)
p~
l-p,
(7)
dx "~-i ,=,,,./ 1 - p , x )
x=p;'
for i - 1.... ,n/ and j - 1.... N and the DF of the convolution may be obtained by inverting (6): N
nj
w(~, p, ...., p~,", ,...,"N) = ~ Z ~ow ; (x, pj)
(8)
j=l i=1
where W" (x, p) is given by (5) respectively.
Proof." The result follows by taking partial fractions expansion of the LST (6) (by taking/~(s) as the free variable) and then applying Theorem 2 to get (8).~ Corollary 3. For the case with only two groups of queues, N = 2 (with equal load in each group), the coefficients given by (7) may be found explicitly since the differentiation may be carried out. We get the following expansions: nI
n2
(9)
W(x, p,, p~ , n, , n~) = ~ c,, W' (x, p, ) + y ' ci2W' (x, P2 ) i=1
i=1
where W i (x, p) are given by (5) and Cil =(_l)n._ i n, + n z - i - 1
nz - 1
(1-pz)p ~ Pl P2 -
-
(1-p,)p z Pl
-
-
ci2 = (_ 1).~_,. n~ + n 2 - i - 1 (1 - ,o~),o 2 "' (1 - ,o 2 ) p , n 1- 1
2.2
P2
- Pl
P2
for i=l,...,n,
and
(10)
P2
- Pl
i
for i = 1.... ,n 2
(11)
Convolution of waiting times in M/G/1 queues having different service time
distributions The use of Theorem 3 and Corollary 3 requires that all the service times are identically distributed. For instance, a path that also includes rather slow access links, could not be modelled well by applying these results. An alternative could be to find the delay distributions
1235 for each part; the access and core, and then perform numerical convolution to find the total end-to-end delay. It is, however, possible to extend some of the results to cover convolutions between equal loaded groups (with different service time distributions in each group) if the convolution obtained by taking one single waiting time in each group is found. For simplicity we consider the case with two groups (and use the same notation as in Theorem 3 and
Corollary 3). Theorem 4. Let w(x, p~ , /92) = w(x, pl ) 9 w(x, /92 ) be the convolution of PDFs of two waiting times distributions for M/G/1 queues, one from each group, and let W(x, pl,/92) denote the corresponding DF. Then we may get the DF of the convolution of n~ waiting times from group 1 and n2 waiting times from group 2 as:
W(x'p~'p2'n~ n2)=(1-P')" (n~ -1) (n2-1) cOp,"-'Op2 { p'''-' l-p2
(12)
Proof" The proof follows directly from Theorem 2 and the fact that w(x, pl,p2) is the convolution.ta To apply the last theorem we first need to find the convolution w(x, p~, t92) which might be difficult to find unless for specific models. Below we shall show that for M/D/1 queues this convolution is possible to obtain in closed forms, and therefore we can apply the theorem to find the convolution for a series consisting of two groups having constant but different service times. 2.3
Convolution of the waiting time distribution for a given number of M/D/1 queues all with equal service times In the following we shall consider the case with constant service times, since this will be of specific interest for applications. Without losing any generality we scale the service time to unity. Then it is well known that the DF for the waiting time of the M/D/1 queue is given by [9]:
q(x,p) = ( 1 - , o )
''/~4L~ - x)]k e -p(*-x' k!
(13)
k=O
Below we shall apply the Theorem 2 to find explicit an expression for the convolution of the waiting time distributions for given numbers of equal loaded M/D/1 queues. Theorem 5. The DF of the convolution of the waiting times of K identical M/D/1 queues are given as:
LxJr-' (-1)t (K + k-1)( X))k§ q K ( x ' P ) = ( 1 - - p ) K Z Z l!k! k+l p(ke -ptk-x)
(14)
k=O 1=0
Proof" By applying Theorem 2.a As for q(x, p) we have found an effective numerical algorithm to calculate the convolution qK (x,p) for large values of x. The algorithm is found by applying the corresponding algorithm for q(x, p) as reported in [9] and then apply Theorem 2. 2.4
Convolution of waiting times in M/D/1 queues having different service time distributions For M/D1 model it is possible to find the convolution of two DF of waiting times with different service times. Then by applying Theorem 4 more general convolutions may be obtained by differentiations with respect to the different loads in the different groups.
1236
Theorem 6. We consider two M/D/1 queues with load Pi and service time b i ,i = 1,2, and we denote W(t, p i , b i ) = q(t/b i , p i )
the DF of the waiting time, i = 1,2. Then the DF of the
convolution is given by the following sums: W(t,p,,p2
b, b 2 ) = ( 1 - p , ) '
+(l_P2 )
'
~-~[~] [ t - kb, - jb2 ~-](P~.j(f, rl) q(~,P2)-q( ,=o j=o b2
t - kb, - jb2 ] -1,P2) b2
~.rlp,.+((,rl ) q ( t - k b , - j b 2 t - k b , - j b 2 -1,p,) b, ' P')-q( b------------~-
(15)
,. . . . o
(j 1
where p~.,((,rl) = k + j r/,( ~ and ( =
y
y-x
x
and 1 / = ~ ,
x-y
where we have defined x = ~ and
y = ~- ; (and where we have ( + r/= 1 ).
Proof" By direct calculating the convolution using expression (13).~ We now move to the interesting case with two groups of queues with different service times as described by Theorem 4. Theorem 7. We consider two groups of M/D/1 queues of size n; each with load Pi and service time bi,i = 1,2, and we denote W(t, p i , b i ) = q ( t / b i ,Pi) the DF of the waiting time in each queue, i = 1,2. Then the DF of the convolution is given by the following sums: W(t, Pl, P2 ,b~ ,b 2, nl, n 2 ) = (1- p, )"' ~-] ~ ~-](1-p_.) ...........
(16)
.~_. . . . . . (t - kb, - jb._ ~p,,,_...._...... ((,r/) q - - , p _ . ) b,
q,+, (t - kb, - jb._ b_,
~-4(l_pt).,_.,_,rlp.,_~_,.,,~_,.k.,((,rl ) q.,+,(t-kb,-jb 2
(l_p2),, ~
........
b,
1, p._ ) +
~+'(t-kb'-jb2 -1,p,) ' p' ) - q
b,
where we have = + (K,+K 2 k j j (,,rl,= ,.....X(_I),_ , PK,.K=.~.,((, r/) ~. K= to.,-K,l
and where we define ( =
y
y -x
x x -y
and r / = ~ ;
K,+, / j]()[,k+j_lj(rl
_
<17)
with x = ~- and y = ~-; ( and where we also have
(+r/=1).
Proof" By applying Theorem 4 on equation (15).ta With Theorem 8 we have a tool to analyse end-to-end delay for realistic scenarios in largescale IP networks, also allowing to include the access part of the network. A realistic scenario would be to include two (or more) low capacity queues (links) together with a number of rather high capacity queues (links) representing the core. By the results above it is possible to get realistic estimates of this important QoS parameter for large scale networks. The basic building blocks in the end-to-end delay distribution is given in terms of convolutions qk (x,/9) given by Theorem 5. 3 Some numerical examples End-to-end delay is one of the most important QoS parameters for real time services like voice and video. In an all IP-network the end-to-end delay for a particular stream will be the sum of the delay obtained in a cascade of routers (from the sender to the receiver). The total end-toend delay will then consist of the waiting times in each node plus service times (transmission times onto the links). In the examples below we shall consider a particular chain of routers in a
1237 packet network and we assume that the routers have output buffers with no extra intemal delay due to processing of the packets. The network model is shown in figure 2.
Figure 2. The queueing model of a particular packet-stream traversing n-nodes.
We have made the following assumptions: The queueing discipline is FIFO for all the queues. The background traffic enters (and leaves) node i according to a Poisson process with rate 2 i and the streaming traffic enters node 1 according to a Poisson process with rate 2 s and leaves at node n. All packets (both background and streaming traffic) have constant (and equal) packet lengths of PL and all the links have capacity of C,.. The corresponding parameters used by the convolution approximation are the service times per packets, which are constant and equal to b i = p L / C i , and the load on the different nodes given by Pi = (2s + 2 i)b i . 3.1
Cases with identical nodes that are equally loaded Based on the simplicity of the way the convolutions are performed by Theorem 1 and Theorem 2 makes it feasible to calculate end-to-end convolutions for rather large high numbers of queues. The numerical algorithms derived for the cases with constant service times make it possible to calculate the corresponding CDF (Complementary Distribution Function) for quite large paths containing up to 20 queues in series and will therefore cover paths that are of"real size" in typical IP networks.
0
. . . . . . . . . . . . . . . . .
Or... < ~ ' .............................................. " .
-1
-li
~
n=1,2,3,4,5,7,10,15,20 from rieht to left
!
A
~ ' ~ . !
-\\\\~.~...
n=1,2,3,4,5,7,10,15,20 from rieht to left
i ~
(b): ~
-3
p=o.8 !
,\\\\\\
. 2
4
6
scaled time t
8
I0
\
~
!
17.5
20
~' 2.5
5
7.5
I0
12.5
15
scaled time t
Figure 3. Logarithmic plots of the CDF (Complementary Distribution Function)for end-to-end queueing delay for a series of equally loaded queues with load equal 0.8 (a) and 0.9 (b), scaled by the total service times (end-to-end)for different size of the series.
1238 2.5 f
....
~. . . . . . . . . . . . . . . . . . .
guarantee level ,8=0.01 2i
.~ . f~--i
!< 0.8:
0.6
0.4 ~
J~
, 0.5
n=1,2,3,4,5,7,10,15,20
0.2 .
from below
0.55
t,f/nb
t,,#/nb
=5
=3
t,f/n b = l O
~
0.6
0.65
0.7
'~ ...... J ............... 0.75 0.8 0.85
, 0.9
........
load/7
.L -4
-3
n=S for lower curves
i
n = 10 for upper curves
,
-2
-1
log.Z?
Figure 4. Logarithmic plot of the 1-0.01 quantile of the end-to-end waiting time for various chains of queues as function of the load. The quantile is scaled to one packet transmission time.
Figure 5. The maximal possible load for chains of equally loaded queues as function of the guarantee level (in logarithmic scale), where the corresponding percentile for the end-to-end queueinig delay, scaled by the total service times (end-to-end) is given. In figure 3 we demonstrate how the CDF of the end-to-end queueing delay "converges" when waiting time is scaled by the total service times (end-to-end). Typically for a series of up to 15 queues the distinction is pronounced, but for larger chains the difference between the curves seems to be small. Another important observation is that all the distributions seem to have a common intersection point approximately around 0.1. Form that point the distributions are bounded "from above" by the curves for smaller chains. In practice this means that it is sufficient to calculate the distributions for chains up to say 15 queues and then use the scaled result (for 15 queues) as an approximation for larger chains. In network engineering it is important to make some statement of the guarantee of the end-toend delay. This guarantee is often given in terms of probabilities, for instance that the delay shall not exceed a particular target value by some small probability. So we would like to find the a = 1 - / 3 quantile for small values o f / 3 . We therefore have to find the value t = tff (p) that solves the equation q" (t, p ) = l - fl
(18)
where q" (t, p) is the DF of the convolution. In the figure 4 we have given logarithmic plots of the quantiles as function of the load for different values of size of the chains ranging from 120 queues and fl = 0.01. As an example, suppose that the end-to-end QoS requirement says that only 1% of the packets shall have end-to-end delay that are longer than 50 packet transmission times. (For 2 Mbit/s links and constant packet lengths of 200 bytes this corresponds to end-to-end delay of 40 ms.) Then by figure 4 we find the following load limits (as function on the size of the chains): 9 Pmax ~ 0.74 if the traffic traverses n = 20 nodes 9
Pmax ~ 0.79 if the traffic traverses n = 15 nodes
9
Pmax ~ 0.85 if the traffic traverses n = 10 nodes and
9
Pmax ~ 0.89 if the traffic traverses n = 5 nodes
1239 This simple example shows that the load limits imposed on the routers in a network should to some extent depend on the actual size of the network. A large network containing long paths should operate at slightly lower load than a corresponding network with shorter paths. These load limits could also be found from equation (18) by solving for the load while keeping the quantile fixed. In figure 5 we have plotted this toad limit as a function of the guarantee level /3 given in logarithmic scale for two chain sizes 5 and 10, and for four valued of quantile of the end-to-end delay. (In this figure the quantiles are scaled by the total service time end-to-end.) 3.2
Numerical examples including one or more low capacity access links
An end-to-end path in an IP-network will include one ore more low capacity access links that are well below the capacity deployed in the core networks. On the other hand the core part of a path will typically consist of a rather large number of hops and the core part could therefore contribute to the end-to-end delay. Since the users observe their QoS on end-to-end basis it is important to have models that include both low capacity access part as well as the high capacity core networks that may have large diameter in terms of hops. In Theorem 7 we have given the end-to-end queueing delay for the convolution of two groups of M/D/1 queues where we may have different loads in each group, and more important, also allowing for having different capacity in each groups. As a final example we consider a typical example described by figure 1 where we have a path consisting of an upstream access part a core network with multiple hops and eventually a downstream access part. In the example below we have taken the following parameters: 9 The access part consists of one ore two low capacity links (with the same capacity) 9 The core part consists of five or ten links all with the same capacity 9 The access link capacity is 1/10 of the corresponding core link capacity, giving for instance the access capacity of approximately 15 Mbit/s if the core links are STM-1 link at approximately 150 Mbit/s This example could for instance represent the case of a typical DSL access line that is connected to a core network with minimum STM-1 links (or higher). The CDF of the end-toend waiting times are plotted for some typical load levels in figure 6 scaled by the packet transmission time for the low capacity link.
n2=0 A
n2=5
-0.5
n2=10 %),
-1.5 t
% ~
i
-2.5
2.5
5
7.5
10
12.5
15
17.5
20
scaled time t
Figure 6. Logarithmic plot of the CDF (Complementary Distribution Function)for end-to-end queueing delay for a series of two groups of size nl=O,1,2 and n2=0,5,10 that are equally loaded. The capacity in the first group is 1/10 of that in the second group. The load in both groups is 0.8 and the time unit is scaled to one packet transmission time for the low capacity group.
1240 One of the questions in mind for the given scenario is the following: What would be the proper guarantee for the end-to-end queueing delay (not including other delay components which must be added) for such a scenario? If we assume that the packet lengths is limited to 1500 bytes the corresponding transmission times (service times in queueing terminology) is approximately 0.8 ms for the access links and 0.08 ms for the high capacity links. We find for instance the 0.999 quantile (fl = 0.001 ) for a network loaded at 0.8 to be approximately 18.8 ms for the case with two access links and ten core links. The corresponding result with the slight "looser" 0.99-quantile fl = 0.01 ) is 13.4 ms. An upper bound is possible to obtain by taking the sum of the quantiles from the access part and the core part of the network, and thereby applying the much simpler formulas for delay distribution in each part. By this approach we find the 0.999 quantile to be bounded by 20.8 ms and the 0.99 quantile to be bounded by 15.2 ms.
4
Conclusion
The end-to-end delay is an important QoS parameter for real time services. In IP networks deploying statistical multiplexing this parameter will depend on several parameters like traffic pattern, background traffic, number of hops, the network load, etc. The method proposed gives an effective way of calculating the end-to-end delay distribution. It is shown that load limit will depend on the size of the network indicating that a larger network should be slightly less loaded than a small network provided that the links have the same capacity. The first method proposed in this paper applies only for links with equal capacity, for instance the core part of an IP network. We also give the corresponding results for a chain containing two groups of links with different capacity in each group. This is a particular interesting case and make it possible to model a path in an IP network that includes both access and core links. The latter model is however far more complex and require more computing effort to obtain the desired results.
References [1] [2] [3] [4]
[5] [6] [7] [8] [9]
RFC 2212: Specification of Guaranteed Quality of Service, IETF, September 1997. RFC 2598: An Expedited Forwarding PHB, IETF, June 1999. Hayes J. F., Modeling and Analysis of Computer Communications Networks 1984, Plenum Press, New York. De Vleeschauwer D.,Petit G.H., Steyaert B., Wittevrongel S., Bruneel H., Calculation of end-to-end delay quantile in network of M/G/1 queues, Electronics Letter, Vol. 37, 2001, pp 535-536. Kleinrock L. 1976, Queueing Systems, Volume II: Computer Applications, John Wiley & Sons, New York. Abate, J.,Whitt W., The Fourire-series method for inverting transform of probability distributions, Queueing System, Vol. 10, 1992, pp 5-88. Mitra D., Morrison J. A., Erlang capacity of a shared resource, In proceedings of ITC 14, Vol. lb, 1994, pp 875-885. Wong R., 1989, Asymptotic Approximations of Integrals, Academic Press, Inc. San Diego, CA. Roberts J., Mocci U., Virtamo J. 1996, Broadband Network Teletraffic. Final Report of Action COST 242, Berlin, Springer.
1231
M o d e l s for Calculating E n d - t o - E n d D e l a y in Packet N e t w o r k s Olav Osterbo Telenor R&D, Snaroyveien 30, N-1331 Fornebu, Norway E-mail: [email protected]
Abstract. In this paper we consider an analytical model to calculate end-to-end delays in packet networks. To be more specific we consider a particular path in the network and we approximate the end-to-end delay by assuming that the waiting times in each node are independent and that both the input and output streams at each node are Poisson processes. With these assumptions (approximations) the end-to-end delay yields the convolution of the waiting times in each queue. If the service times in each queue are identically distributed the corresponding convolution may be heavily simplified, and closed form expressions are derived in terms of derivative with respect to the load parameter. We put special emphasis on the case with constant service times since this is an important case for applications. Numerical examples show that end-to-end delays in chains up to 20 nodes may be analysed without numerical difficulties. We also give the corresponding results for a chain containing two groups of links with different capacity in each group. This is a particularly interesting case and makes it possible to model a path in an IP network that includes both access and core links. Numerical examples are given and discussed for the various models.
1
Introduction
There is reason to believe that real time services will be a significant part of the traffic offered in future multi service networks. Real time services require a quite regular bit stream delivered at the receiver's site to maintain the necessary quality for various applications. It is well known that networks based on statistical multiplexing (like IP-networks) will introduce certain disturbance (jitter) in the bit stream mainly due to queuing in routers (or switches). These disturbances will add on along the path from the sender to the receiver, explaining the necessity of some kind of de-jitter buffer at the receiver site to compensate for these variations. The end-to-end delay is therefore an important parameter not only for dimensioning the de-jitter buffers, but also for providing some upper bounds of the total network delay for particular services and it is among the most important QoS (Quality of Service) parameter in networks deploying statistical multiplexing.
1.1
QoS parameters in the SLA (Service Level Agreement)
To realise many types of services the traffic flow will have to cross one or more administrative domains with their own SLAs. Such domains could for instance be 9 access networks with rather low capacity based on different wireless or DSL (Digital Subscriber Line) technologies 9 core networks with high capacity links but with large numbers of routers The end-to-end QoS is therefore realised through the contributions from different domains, and QoS guarantees end-to-end will be realised through different SLAs between the customer and the access network and/or between different core network domains. For each domain it will be important to estimate the contribution to the QoS parameters since each administrative
1232 domain will be responsible for their own contribution through the SLA. The most important parameters will be delay, jitter and information losses due to buffer overflow. It will be important for a network operator to be able to estimate the QoS parameters in his own domain to set the appropriate parameters in the SLAs. In addition it will be of vital importance to implement the necessary control structures that makes it possible to maintain the guarantees especially in his own domain end- and thereby preventing degradation in QoS which is often seen in best effort networks of today [1 ], [2]. The reference configuration for the modelling of the queuing behaviour consists of an upstream access network part, a multi-hop core network and a downstream access network part as illustrated in figure 1. Source application
Destination application Access part (upstream)
Access part (downstream)
Figure 1. Overview of the reference configuration." upstream access network parts, core network part (multi-hop) and downstream access part between the source and destination application.
1.2
Main contribution
The main contribution of this paper is to provide some approximations to calculate the distribution of end-to-end delay in packet based networks, and thereby gain better insight into the important question of how to design the control structure to maintain necessary QoS for delay-critical services. The main focus will be on real time traffic where we assume that packets of constant lengths are generated and shall traverse a given number of nodes (routers) to reach the receiving destination. The end-to-end delay calculations are based on the approach using convolutions. This means that the delay components in the different router queues are taken to be independent. Unless for some specific cases this assumption will not be valid and therefore the resulting model will be approximative. It is, however, reason to believe that if the actual load from each stream is small compared with the total load on the network elements considered, this assumption will be quite reasonable and provides the corresponding end-to-end delay distribution quite accurate.
2
Convolution of a given numbers of M/G/I queues waiting times having identical service time distributions
In the following we consider a model to calculate the end-to-end delays in a large scale IPnetwork. The aim is to calculate the distribution of the end-to-end delay for a particular path consisting of a series of routers. We assume that all the nodes in the end-to-end path are statistically independent; this is a key assumption to obtain the end-to-end delay by convolution. For queueing networks with FCFS (First-Come First-Served) queueing discipline this property only yields for acyclic form of Jackson Networks (where a packet visits a node at most once), see [3]. In [4], however, it is argued that if the load from a particular flow only is a small fraction of the total amount of traffic in a node and the input processes to the network is 'smoother than Poisson' (i.e. with less variability) then the independent assumption will be quite reasonable and will represent a worst case scenario. We shall therefore use the M/G/1 queue as the model to obtain the waiting time distribution in each node and then apply the convolution to obtain the end-to-end waiting time distribution. We are interested in the sum of waiting times in a series of K queues and we denote the sum W = W~ +... + WK . If all the waiting times may be taken to be independent the PDF
1233 (Probability Density Function) of the sum yields the convolution of the waiting times in each queue. The LST (Laplace-Stieltjes Transform) of the convolution (of waiting times for all the K queues) yields the product: K
I'V(s, A ,.-., PK ) = 1-'I I~(S, Pk )
(1)
k=l
where
l'V(s, Pk) =
1 - Pk
1- pk/~k (S)
is the LST of the waiting time for the k th queue (known as
the Pollaczek-Khinchin formula [5]), and where Pk is the load, /~k(s) is the LST of the remaining service time (given by/~k (s) = (1 -s B)k
where B k (s) is the LST of the service time
and b k = E [ B k ]is the mean service time), k = 1..... K . Generally it is possible to obtain the end-to-end distributions above by inverting the LST (1) numerically. Abate & Whitt have given a method to calculate the inversion integral based on Poisson's summation formula. The result yields an alternating series that might be difficult to use to determine the tail of the distribution and thereby obtaining the desired quantiles, (see [6]). Other methods such as the Saddle Point method (Large Deviation) and even more effective, the Uniform Asymptotic Approximation (UAA) [7] are also available to find approximations of the inversion integral of the LST (1), (see [8] for a survey of different methods). In [4] a heuristic method is proposed to find the quantiles of the convolution above if all the queues are identical which turn out to be very effective in terms of computation effort. 2.1
Convolution of waiting times in M/G/1 queues all having identical service time distributions In the following we shall assume that all the nodes have identically service time distributions, implying /}k (s) = /}(s) ; k = 1..... K . It turns out that in this case it is possible to obtain substantial simplification of the convolution (1). Theorem 1. If all the loads of the different M/G/1 queues all are distinct, that is Pi :r Pj for all i , j = 1..... K , then the LST of the convolution can be written as a weighted sum of the individual LST for each queue as follows: ,-,., K K l_pl (2) W(s, p, ,..., PK ) = ~'~ CkW'(S, Pk ) where c, = 1--I 1 p , / p k k=l
l=l,l~k
only depend on the loads in the different queues. Further the DF (Distribution Function) of the convolution has similar form K
W(x, p, ,..., p~ ) = Z ~ W(x, p~ )
(3)
k=l
where W(x,Pk ) is the DF for the k' th queue.
_Proof By taking partial fractions expansion of the LST (I) and taking B(s) as the free variable, n Often we are interested in the case where the loads on the different queues are equal. This result is possible to obtain relatively easily from Theorem 1 by letting pk ~ p for all k = 1..... K , (or by direct differentiation). We therefore get the following results when convoluting the waiting time for K identical M/G/1 queues:
1234
Theorem 2. We have =
Opt-' L1- p
_
'r
=
-
[1- p
O p K-'
W(x, p)
}
W(s, p) and
(4)
}
(5)
We can now state the more general theorem where we consider the general case where only some of the queues may be equally loaded. Theorem 3. Suppose we have total N groups of size n/ of queues that are equal loaded and p / r p / for all i , j = 1,...,N and K =n~ + .... + n
N.
Then the LST of the convolution may be
written:
/ W(s,- p, ,..., PN,", ,'","N) = ]--[U
1-- p~
*)=~
1 - p/B(s)
/i
= ~ ~ %. /--, i--,
(6) 1 - p/i~(s)
where the coefficients % only depend on the loads in the different queues and are given by:
d "~-'
(-1)"J -i l - p /
% = (n/-i)
p~
l-p,
(7)
dx "~-i ,=,,,./ 1 - p , x )
x=p;'
for i - 1.... ,n/ and j - 1.... N and the DF of the convolution may be obtained by inverting (6): N
nj
w(~, p, ...., p~,", ,...,"N) = ~ Z ~ow ; (x, pj)
(8)
j=l i=1
where W" (x, p) is given by (5) respectively.
Proof." The result follows by taking partial fractions expansion of the LST (6) (by taking/~(s) as the free variable) and then applying Theorem 2 to get (8).~ Corollary 3. For the case with only two groups of queues, N = 2 (with equal load in each group), the coefficients given by (7) may be found explicitly since the differentiation may be carried out. We get the following expansions: nI
n2
(9)
W(x, p,, p~ , n, , n~) = ~ c,, W' (x, p, ) + y ' ci2W' (x, P2 ) i=1
i=1
where W i (x, p) are given by (5) and Cil =(_l)n._ i n, + n z - i - 1
nz - 1
(1-pz)p ~ Pl P2 -
-
(1-p,)p z Pl
-
-
ci2 = (_ 1).~_,. n~ + n 2 - i - 1 (1 - ,o~),o 2 "' (1 - ,o 2 ) p , n 1- 1
2.2
P2
- Pl
P2
for i=l,...,n,
and
(10)
P2
- Pl
i
for i = 1.... ,n 2
(11)
Convolution of waiting times in M/G/1 queues having different service time
distributions The use of Theorem 3 and Corollary 3 requires that all the service times are identically distributed. For instance, a path that also includes rather slow access links, could not be modelled well by applying these results. An alternative could be to find the delay distributions
1235 for each part; the access and core, and then perform numerical convolution to find the total end-to-end delay. It is, however, possible to extend some of the results to cover convolutions between equal loaded groups (with different service time distributions in each group) if the convolution obtained by taking one single waiting time in each group is found. For simplicity we consider the case with two groups (and use the same notation as in Theorem 3 and
Corollary 3). Theorem 4. Let w(x, p~ , /92) = w(x, pl ) 9 w(x, /92 ) be the convolution of PDFs of two waiting times distributions for M/G/1 queues, one from each group, and let W(x, pl,/92) denote the corresponding DF. Then we may get the DF of the convolution of n~ waiting times from group 1 and n2 waiting times from group 2 as:
W(x'p~'p2'n~ n2)=(1-P')" (n~ -1) (n2-1) cOp,"-'Op2 { p'''-' l-p2
(12)
Proof" The proof follows directly from Theorem 2 and the fact that w(x, pl,p2) is the convolution.ta To apply the last theorem we first need to find the convolution w(x, p~, t92) which might be difficult to find unless for specific models. Below we shall show that for M/D/1 queues this convolution is possible to obtain in closed forms, and therefore we can apply the theorem to find the convolution for a series consisting of two groups having constant but different service times. 2.3
Convolution of the waiting time distribution for a given number of M/D/1 queues all with equal service times In the following we shall consider the case with constant service times, since this will be of specific interest for applications. Without losing any generality we scale the service time to unity. Then it is well known that the DF for the waiting time of the M/D/1 queue is given by [9]:
q(x,p) = ( 1 - , o )
''/~4L~ - x)]k e -p(*-x' k!
(13)
k=O
Below we shall apply the Theorem 2 to find explicit an expression for the convolution of the waiting time distributions for given numbers of equal loaded M/D/1 queues. Theorem 5. The DF of the convolution of the waiting times of K identical M/D/1 queues are given as:
LxJr-' (-1)t (K + k-1)( X))k§ q K ( x ' P ) = ( 1 - - p ) K Z Z l!k! k+l p(ke -ptk-x)
(14)
k=O 1=0
Proof" By applying Theorem 2.a As for q(x, p) we have found an effective numerical algorithm to calculate the convolution qK (x,p) for large values of x. The algorithm is found by applying the corresponding algorithm for q(x, p) as reported in [9] and then apply Theorem 2. 2.4
Convolution of waiting times in M/D/1 queues having different service time distributions For M/D1 model it is possible to find the convolution of two DF of waiting times with different service times. Then by applying Theorem 4 more general convolutions may be obtained by differentiations with respect to the different loads in the different groups.
1236
Theorem 6. We consider two M/D/1 queues with load Pi and service time b i ,i = 1,2, and we denote W(t, p i , b i ) = q(t/b i , p i )
the DF of the waiting time, i = 1,2. Then the DF of the
convolution is given by the following sums: W(t,p,,p2
b, b 2 ) = ( 1 - p , ) '
+(l_P2 )
'
~-~[~] [ t - kb, - jb2 ~-](P~.j(f, rl) q(~,P2)-q( ,=o j=o b2
t - kb, - jb2 ] -1,P2) b2
~.rlp,.+((,rl ) q ( t - k b , - j b 2 t - k b , - j b 2 -1,p,) b, ' P')-q( b------------~-
(15)
,. . . . o
(j 1
where p~.,((,rl) = k + j r/,( ~ and ( =
y
y-x
x
and 1 / = ~ ,
x-y
where we have defined x = ~ and
y = ~- ; (and where we have ( + r/= 1 ).
Proof" By direct calculating the convolution using expression (13).~ We now move to the interesting case with two groups of queues with different service times as described by Theorem 4. Theorem 7. We consider two groups of M/D/1 queues of size n; each with load Pi and service time bi,i = 1,2, and we denote W(t, p i , b i ) = q ( t / b i ,Pi) the DF of the waiting time in each queue, i = 1,2. Then the DF of the convolution is given by the following sums: W(t, Pl, P2 ,b~ ,b 2, nl, n 2 ) = (1- p, )"' ~-] ~ ~-](1-p_.) ...........
(16)
.~_. . . . . . (t - kb, - jb._ ~p,,,_...._...... ((,r/) q - - , p _ . ) b,
q,+, (t - kb, - jb._ b_,
~-4(l_pt).,_.,_,rlp.,_~_,.,,~_,.k.,((,rl ) q.,+,(t-kb,-jb 2
(l_p2),, ~
........
b,
1, p._ ) +
~+'(t-kb'-jb2 -1,p,) ' p' ) - q
b,
where we have = + (K,+K 2 k j j (,,rl,= ,.....X(_I),_ , PK,.K=.~.,((, r/) ~. K= to.,-K,l
and where we define ( =
y
y -x
x x -y
and r / = ~ ;
K,+, / j]()[,k+j_lj(rl
_
<17)
with x = ~- and y = ~-; ( and where we also have
(+r/=1).
Proof" By applying Theorem 4 on equation (15).ta With Theorem 8 we have a tool to analyse end-to-end delay for realistic scenarios in largescale IP networks, also allowing to include the access part of the network. A realistic scenario would be to include two (or more) low capacity queues (links) together with a number of rather high capacity queues (links) representing the core. By the results above it is possible to get realistic estimates of this important QoS parameter for large scale networks. The basic building blocks in the end-to-end delay distribution is given in terms of convolutions qk (x,/9) given by Theorem 5. 3 Some numerical examples End-to-end delay is one of the most important QoS parameters for real time services like voice and video. In an all IP-network the end-to-end delay for a particular stream will be the sum of the delay obtained in a cascade of routers (from the sender to the receiver). The total end-toend delay will then consist of the waiting times in each node plus service times (transmission times onto the links). In the examples below we shall consider a particular chain of routers in a
1237 packet network and we assume that the routers have output buffers with no extra intemal delay due to processing of the packets. The network model is shown in figure 2.
Figure 2. The queueing model of a particular packet-stream traversing n-nodes.
We have made the following assumptions: The queueing discipline is FIFO for all the queues. The background traffic enters (and leaves) node i according to a Poisson process with rate 2 i and the streaming traffic enters node 1 according to a Poisson process with rate 2 s and leaves at node n. All packets (both background and streaming traffic) have constant (and equal) packet lengths of PL and all the links have capacity of C,.. The corresponding parameters used by the convolution approximation are the service times per packets, which are constant and equal to b i = p L / C i , and the load on the different nodes given by Pi = (2s + 2 i)b i . 3.1
Cases with identical nodes that are equally loaded Based on the simplicity of the way the convolutions are performed by Theorem 1 and Theorem 2 makes it feasible to calculate end-to-end convolutions for rather large high numbers of queues. The numerical algorithms derived for the cases with constant service times make it possible to calculate the corresponding CDF (Complementary Distribution Function) for quite large paths containing up to 20 queues in series and will therefore cover paths that are of"real size" in typical IP networks.
0
. . . . . . . . . . . . . . . . .
Or... < ~ ' .............................................. " .
-1
-li
~
n=1,2,3,4,5,7,10,15,20 from rieht to left
!
A
~ ' ~ . !
-\\\\~.~...
n=1,2,3,4,5,7,10,15,20 from rieht to left
i ~
(b): ~
-3
p=o.8 !
,\\\\\\
. 2
4
6
scaled time t
8
I0
\
~
!
17.5
20
~' 2.5
5
7.5
I0
12.5
15
scaled time t
Figure 3. Logarithmic plots of the CDF (Complementary Distribution Function)for end-to-end queueing delay for a series of equally loaded queues with load equal 0.8 (a) and 0.9 (b), scaled by the total service times (end-to-end)for different size of the series.
1238 2.5 f
....
~. . . . . . . . . . . . . . . . . . .
guarantee level ,8=0.01 2i
.~ . f~--i
!< 0.8:
0.6
0.4 ~
J~
, 0.5
n=1,2,3,4,5,7,10,15,20
0.2 .
from below
0.55
t,f/nb
t,,#/nb
=5
=3
t,f/n b = l O
~
0.6
0.65
0.7
'~ ...... J ............... 0.75 0.8 0.85
, 0.9
........
load/7
.L -4
-3
n=S for lower curves
i
n = 10 for upper curves
,
-2
-1
log.Z?
Figure 4. Logarithmic plot of the 1-0.01 quantile of the end-to-end waiting time for various chains of queues as function of the load. The quantile is scaled to one packet transmission time.
Figure 5. The maximal possible load for chains of equally loaded queues as function of the guarantee level (in logarithmic scale), where the corresponding percentile for the end-to-end queueinig delay, scaled by the total service times (end-to-end) is given. In figure 3 we demonstrate how the CDF of the end-to-end queueing delay "converges" when waiting time is scaled by the total service times (end-to-end). Typically for a series of up to 15 queues the distinction is pronounced, but for larger chains the difference between the curves seems to be small. Another important observation is that all the distributions seem to have a common intersection point approximately around 0.1. Form that point the distributions are bounded "from above" by the curves for smaller chains. In practice this means that it is sufficient to calculate the distributions for chains up to say 15 queues and then use the scaled result (for 15 queues) as an approximation for larger chains. In network engineering it is important to make some statement of the guarantee of the end-toend delay. This guarantee is often given in terms of probabilities, for instance that the delay shall not exceed a particular target value by some small probability. So we would like to find the a = 1 - / 3 quantile for small values o f / 3 . We therefore have to find the value t = tff (p) that solves the equation q" (t, p ) = l - fl
(18)
where q" (t, p) is the DF of the convolution. In the figure 4 we have given logarithmic plots of the quantiles as function of the load for different values of size of the chains ranging from 120 queues and fl = 0.01. As an example, suppose that the end-to-end QoS requirement says that only 1% of the packets shall have end-to-end delay that are longer than 50 packet transmission times. (For 2 Mbit/s links and constant packet lengths of 200 bytes this corresponds to end-to-end delay of 40 ms.) Then by figure 4 we find the following load limits (as function on the size of the chains): 9 Pmax ~ 0.74 if the traffic traverses n = 20 nodes 9
Pmax ~ 0.79 if the traffic traverses n = 15 nodes
9
Pmax ~ 0.85 if the traffic traverses n = 10 nodes and
9
Pmax ~ 0.89 if the traffic traverses n = 5 nodes
1239 This simple example shows that the load limits imposed on the routers in a network should to some extent depend on the actual size of the network. A large network containing long paths should operate at slightly lower load than a corresponding network with shorter paths. These load limits could also be found from equation (18) by solving for the load while keeping the quantile fixed. In figure 5 we have plotted this toad limit as a function of the guarantee level /3 given in logarithmic scale for two chain sizes 5 and 10, and for four valued of quantile of the end-to-end delay. (In this figure the quantiles are scaled by the total service time end-to-end.) 3.2
Numerical examples including one or more low capacity access links
An end-to-end path in an IP-network will include one ore more low capacity access links that are well below the capacity deployed in the core networks. On the other hand the core part of a path will typically consist of a rather large number of hops and the core part could therefore contribute to the end-to-end delay. Since the users observe their QoS on end-to-end basis it is important to have models that include both low capacity access part as well as the high capacity core networks that may have large diameter in terms of hops. In Theorem 7 we have given the end-to-end queueing delay for the convolution of two groups of M/D/1 queues where we may have different loads in each group, and more important, also allowing for having different capacity in each groups. As a final example we consider a typical example described by figure 1 where we have a path consisting of an upstream access part a core network with multiple hops and eventually a downstream access part. In the example below we have taken the following parameters: 9 The access part consists of one ore two low capacity links (with the same capacity) 9 The core part consists of five or ten links all with the same capacity 9 The access link capacity is 1/10 of the corresponding core link capacity, giving for instance the access capacity of approximately 15 Mbit/s if the core links are STM-1 link at approximately 150 Mbit/s This example could for instance represent the case of a typical DSL access line that is connected to a core network with minimum STM-1 links (or higher). The CDF of the end-toend waiting times are plotted for some typical load levels in figure 6 scaled by the packet transmission time for the low capacity link.
n2=0 A
n2=5
-0.5
n2=10 %),
-1.5 t
% ~
i
-2.5
2.5
5
7.5
10
12.5
15
17.5
20
scaled time t
Figure 6. Logarithmic plot of the CDF (Complementary Distribution Function)for end-to-end queueing delay for a series of two groups of size nl=O,1,2 and n2=0,5,10 that are equally loaded. The capacity in the first group is 1/10 of that in the second group. The load in both groups is 0.8 and the time unit is scaled to one packet transmission time for the low capacity group.
1240 One of the questions in mind for the given scenario is the following: What would be the proper guarantee for the end-to-end queueing delay (not including other delay components which must be added) for such a scenario? If we assume that the packet lengths is limited to 1500 bytes the corresponding transmission times (service times in queueing terminology) is approximately 0.8 ms for the access links and 0.08 ms for the high capacity links. We find for instance the 0.999 quantile (fl = 0.001 ) for a network loaded at 0.8 to be approximately 18.8 ms for the case with two access links and ten core links. The corresponding result with the slight "looser" 0.99-quantile fl = 0.01 ) is 13.4 ms. An upper bound is possible to obtain by taking the sum of the quantiles from the access part and the core part of the network, and thereby applying the much simpler formulas for delay distribution in each part. By this approach we find the 0.999 quantile to be bounded by 20.8 ms and the 0.99 quantile to be bounded by 15.2 ms.
4
Conclusion
The end-to-end delay is an important QoS parameter for real time services. In IP networks deploying statistical multiplexing this parameter will depend on several parameters like traffic pattern, background traffic, number of hops, the network load, etc. The method proposed gives an effective way of calculating the end-to-end delay distribution. It is shown that load limit will depend on the size of the network indicating that a larger network should be slightly less loaded than a small network provided that the links have the same capacity. The first method proposed in this paper applies only for links with equal capacity, for instance the core part of an IP network. We also give the corresponding results for a chain containing two groups of links with different capacity in each group. This is a particular interesting case and make it possible to model a path in an IP network that includes both access and core links. The latter model is however far more complex and require more computing effort to obtain the desired results.
References [1] [2] [3] [4]
[5] [6] [7] [8] [9]
RFC 2212: Specification of Guaranteed Quality of Service, IETF, September 1997. RFC 2598: An Expedited Forwarding PHB, IETF, June 1999. Hayes J. F., Modeling and Analysis of Computer Communications Networks 1984, Plenum Press, New York. De Vleeschauwer D.,Petit G.H., Steyaert B., Wittevrongel S., Bruneel H., Calculation of end-to-end delay quantile in network of M/G/1 queues, Electronics Letter, Vol. 37, 2001, pp 535-536. Kleinrock L. 1976, Queueing Systems, Volume II: Computer Applications, John Wiley & Sons, New York. Abate, J.,Whitt W., The Fourire-series method for inverting transform of probability distributions, Queueing System, Vol. 10, 1992, pp 5-88. Mitra D., Morrison J. A., Erlang capacity of a shared resource, In proceedings of ITC 14, Vol. lb, 1994, pp 875-885. Wong R., 1989, Asymptotic Approximations of Integrals, Academic Press, Inc. San Diego, CA. Roberts J., Mocci U., Virtamo J. 1996, Broadband Network Teletraffic. Final Report of Action COST 242, Berlin, Springer.