- Email: [email protected]
Near optimal admission control for multiserver loss queues in series
European Journal of Operational Research 144 (2003) 166–178 www.elsevier.com/locate/dsw
Stochastics and Statistics
Near optimal admission control for multiserver loss queues in series Cheng-Yuan Ku a
a,*
, Scott Jordan
b
Department of Information Management, National Chung Cheng University, 160 San-Hsing, Chia-Yi County, Taiwan, ROC b Department of Electrical and Computer Engineering, University of California, Irvine, CA 92697-2625, USA Received 26 March 2000; accepted 18 October 2001
Abstract This paper considers access control policies in multiserver loss queues in series such as might arise in the context of computer and telecommunication networks. Each queue is presented with both served upstream customers and Poisson arrivals from outside the network, and it may route serviced customers out of the network or to the downstream queue. Service times of each customer are i.i.d. and exponentially distributed. Revenue is earned by each station when it serves a customer, but the amount of revenue depends on whether the customer entered the network at this station or was routed from an upstream station. We propose a simple recursive method to solve the problem using dynamic programming on a set of reduced state spaces. This approach includes a rate estimation technique for upstream stations, and a revenue estimation technique for downstream stations. Numerical results demonstrate the performance of these near-optimal policies under light, moderate, and heavy traffic. Ó 2002 Elsevier Science B.V. All rights reserved. Keywords: Queueing; Dynamic programming; Connection admission control
1. Introduction This paper considers access control policies in multiserver loss queues in series. Each queue is presented with arrivals of customers of two types: a stream of ‘‘internal customers’’ from the previous queue, and a Poisson stream of ‘‘new customers’’ from outside the network. Each queue has an arbitrary number of servers, but no queue.
*
Corresponding author. Tel.: +886-5-2720411x34603. E-mail addresses: [email protected] (C.-Y. Ku), [email protected] (S. Jordan).
Service times of each customer are i.i.d., but the revenue generated is differentiated, with internal customers typically paying more than new customers. Departures from the queue are Bernoulli routed either to the next queue or out of the network, independent of their type. We are interested in the access control policy, which will make a decision to accept or deny upon the arrival of any customer at each queue. The objective here is to maximize the total discounted revenue over an infinite horizon. This access control problem arises in the context of computer and telecommunication networks, e.g., Internet, cellular phone networks, and
0377-2217/03/$ - see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 1 ) 0 0 3 8 7 - 3
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
asynchronous transfer mode (ATM) [1,3,9,17]. Computer networks were originally designed for ‘‘best-effort traffic’’, and typically serve traffic on a first-come first-served basis in a series of queues. Telecommunication networks, on the other hand, were originally designed for real-time traffic using a ‘‘circuit-switched architecture’’ – each station has essentially no queue and reserves one server (defined as the transmission capacity required for one call) for each call passing through it. Both types of networks are currently evolving toward multi-service architectures, in which transmission of traffic at each station is differentiated based on the service type. In particular, we are interested here in real-time traffic. The new network architectures strictly limit queueing time for real-time traffic, because of their inherent low-delay requirements. In addition, these architectures implement some type of reservation or priority mechanisms to give real-time traffic preference over best-effort traffic [10–13]. These reservation or priority mechanisms typically result in a connection admission control (CAC) policy on real-time calls, based on the capacity of each station. In this paper, we model this type of network as a series of multiserver loss queues. Only a single series of queues is considered here; extensions to a general network topology will be considered at a later time. We assume that real-time traffic is given preference over best-effort traffic; hence only realtime traffic is modeled. Delay is minimized by not allowing real-time traffic to queue at any station. We only model the network traffic on an intermediate timescale. A ‘‘customer’’ corresponds to a burst of real-time traffic, i.e., a randomly sized batch of packets; we do not explicitly model longterm timescale phenomena, e.g., calls. The CAC policy is modeled by a server policy at each station that can reserve any number of servers for internal customers. In order to allow the policy to achieve the desired balance of blocking of internal and new customers, we assume that each customer pays revenue to each station it passes through. The amount of the revenue can depend on the station and on its type. In general, internal customers would be assumed to pay more, in order to bias the network in favor of completing transmission of
167
traffic that has already received service at a previous station. In the long and rich literature on stochastic dynamic programming, there have been many authors that have chosen to use expected discounted measures and many authors that have chosen to use average per unit time measures. With respect to the particular problem addressed in this paper, we believe that expected discounted revenue is a better choice for two reasons. First, it is possible that some type of pricing may be used in the future in computer networks such as the Internet. In such systems, discounting naturally reflects the presence of an underlying interest rate, and maximizing expected discounted revenue more closely corresponds to the likely desired optimization metric. Second, results concerning the structure of policies that maximize expected discounted revenue can often be used to prove similar structural results for policies that maximize average revenue per unit time. However, the converse is not true (see e.g., [14] for a general presentation of such correspondences and [15,16] for a discussion of why expected discounted cost measures have often been chosen in papers on admission control in queueing networks). We therefore choose to present these structural results using expected discounted revenue as the optimization metric. A good survey of research on access control policies in queueing networks can be found in [15,16]. Most of the research on this topic, however, focuses on infinite queue models and uses holding costs as the performance metric. In contrast, here we assume a loss network, and use loss as the performance metric. The typical approach to determine the optimal decision policy at each queue in a network, as a function of the number of customers at each station, is to use dynamic programming [2,4–6,15]. However, the size of the Markov chain for the entire network grows exponentially with the number of stations, so this approach quickly becomes numerically infeasible. Our goal in this paper is to reduce the amount of information contained in the state, and correspondingly the size of the state space. We borrow an idea often used in the stochastic modeling
168
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
literature, where a common approach models only the bottleneck queue(s) in a network, by recursively estimating internal flow rates. We are unaware of such an approach being used in optimization of loss networks. We propose determining a near-optimal policy for station i by creating a reduced model whose state contains only the number of customers at station i and the number of customers at its immediate upstream station i 1. This procedure ignores the states of stations upstream from i 1, so it requires estimation of the flow into that station. We also need to eliminate stations downstream from i. This reduction, however, contributes significant error to the decision as to whether to accept new customers. Therefore we propose a method to estimate the downstream revenue and to take this information into account at station i. The control policy at one station affects upstream stations by altering their expected downstream revenue, and it affects downstream stations by altering their expected flow of internal customers. We therefore propose to apply these two techniques, rate estimation and revenue estimation, using simple recursions. Numerical results are presented to demonstrate the success of this approach. The multiserver-loss-queues-in-series model and problem formulation is presented in Section 2. In Section 3, the state reduction method, along with rate and revenue estimation techniques, is proposed. In Section 4, the numerical results are presented.
2. Model and problem formulation Denote by n the number of stations in the tandem network. Station i is a non-preemptive loss queue with mi servers. Each station is presented with a Poisson stream of new customers, with arrival rate ki . Service times are i.i.d. and exponentially distributed with rate li . Departures from station i are routed out of the network with probability Pi or to station i þ 1 otherwise, independent where the customer entered the network, the customer’s service time, and of other customers. Revenue is paid by each customer at the start of service. In telecommunication networks, blocking
of customers who have already been accepted into the network (herein referred to as ‘‘internal customers’’) is known as ‘‘forced termination’’ and is considered to be an order of magnitude worse than blocking of new customers. Such networks often target very low forced termination rates. One model of this metric would be to directly impose a cost Ci on each blocked internal customer. We instead adopt an alternate model in which revenue is earned for every customer accepted at every station, but the revenue for internal customers is much higher than for new customers. This alternate model, although not in general equivalent to the cost-based model, still biases the network in favor of internal customers, and more easily lends itself to analysis. Specifically, we assume that a new customer who enters the network at station i pays an amount ri at station i, and that an internal customer pays an amount Ri at station i. We assume Ri > ri ð2 6 i 6 nÞ, so that internal customers are preferable to new customers at station i, and that 1 > Pi P 0 ð1 6 i 6 n 1Þ. We consider connection admission control policies that are capable of accepting or blocking arrivals of each customer type (internal or new) at each station. Blocked arrivals are lost. Our objective is to maximize the total discounted revenue over an infinite horizon. It has been shown that the optimal policy does not block customers at station 1 and does not block internal customers at any station [6,7]. Admission policies therefore reduce to determining whether to block or accept new arrivals at station i; 2 6 i 6 n. For a two-stationin-tandem network, the optimal policy can be shown to consist of a switching curve in the twodimensional state space [6,7]. On one side of the curve, new arrivals are accepted; on the other side, new arrivals are blocked. Unfortunately, a similar characterization for the optimal policy in a series of more than two stations is unknown. Our objective here is to numerically determine a nearoptimal policy.
3. State space reduction Dynamic programming can be used to determine the optimal policy, which states whether to
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
accept or block each new arrival at each station as a function of the number of current customers at each station in the network. However, the numerical complexity grows exponentially with the dimension of the state space, n. This renders computation of the optimal policy difficult for all but very small n. In this section, we propose a technique to reduce the dynamic programming problem to a collection of smaller problems, each of which consists of determining a switching curve for a single station. Consider a particular station i. We propose determining a near-optimal policy for station i by reducing the state to a vector containing only the number of customers at station i and the number of customers at its immediate upstream station i 1. This procedure requires estimation of two quantities: (1) the flow into station i 1 and (2) the expected discounted reward paid at downstream stations. We will denote our estimate of the flow into station i by Ci , our estimate of the flow out of station i by di , and our estimate of the downstream discounted net revenue for a customer leaving station i by Hi . The estimation of rates is discussed in the first subsection, and the estimation of rewards is discussed in the second subsection. 3.1. Estimation of upstream rates Reduction of the state space will involve elimination of both upstream and downstream stations, reducing the state space for a single station to a vector containing the number of customers in that station and the number in its immediate upstream neighbor. This tandem queue can be easily analyzed. In this subsection, we consider elimination of upstream stations. Focus on station i. Elimination of stations 1 through i 2 will reduce the information we have about short-term arrival rate variations of internal customers. The number of servers reserved for internal customers at station i will typically be increasing with the number of current customers present at each station upstream from i. However, we expect that the contribution of an upstream station to the reservation level at station i will be decreasing with the distance between the two sta-
169
tions. A reasonable approach, therefore, is to eliminate all but the immediate upstream neighbor, station i 1. (A more accurate, but computationally more intensive, model is to keep additional upstream stations, e.g., from i 2 to i and estimate the arrival rate into station i 2.) We make two approximations to simplify the model. First, we assume that the arrival process into station i 1 is Poisson. The arrival process into station i 1 is the multiplexing of the departure process from station i 2, thinned by those departing the network, and the new arrivals into station i 1. Since the departure process from station i 2 is not Poisson, the arrival process into station i 1 is not Poisson. Nevertheless, we have found this approximation to yield good results. Second, for purposes of estimating the arrival rate into station i 1, we model the upstream stations 1 through i 1 as uncontrolled loss queues. The rate of the departures from station i 2 depends on the control policy used in station i 2. However, under a low level of blocking, ignoring this control policy is a reasonable approximation. Under higher blocking rates, this second approximation can be avoided by explicitly using information about the control policies used in upstream stations. The resulting method then includes an iteration, which significantly increases the computational complexity [6,7]. Our approach requires estimation of the arrival rate of customers into station i 1, Ci1 . We approximate the flow into station i 1 as a Poisson process, with a rate equal to the flow rate from station i 2 to station i 1 plus the rate of new arrivals into station i 1. Every station in this system is a M=M=m=m loss queue. Therefore, the loss probability is given by the Erlang B formula. If the arrival rate is k and the service rate is l, then Blocking probability m
ðk=lÞ =m! pðk; l; mÞ ¼ Pm : k k¼0 ðk=lÞ =k!
ð1Þ
For station j; 2 6 j 6 i 1, we estimate rates using conservation of flow: Cj dj1 ð1 Pj1 Þ þ kj :
ð2Þ
170
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
As noted above, this approximation method of the arrival rate depends on the departure rate of the previous station. Using the Erlang B formula, we can approximate the departure rate as dj1 Cj1 1 pðCj1 ; lj1 ; mj1 Þ : ð3Þ Denote gi ðkÞ k½1 pðk; li ; mi Þ ð1 Pi Þ. This approximation can then be implemented using a recursion, and the two preceding equations. For station 1, the estimated arrival rate is equal to the arrival rate of new customers: C1 ¼ k1 :
ð4Þ
For station 2, the estimated arrival rate is based on the departure rate of station 1 customers and the arrival of new customers: C2 d1 ð1 P1 Þ þ k2 C1 ½1 pðC1 ; l1 ; m1 Þ ð1 P1 Þ þ k2 ¼ g1 ðC1 Þ þ k2 ¼ g1 ðk1 Þ þ k2 :
ð5Þ
The recursion continues by estimating the arrival rate into station i 1: Ci1 di2 ð1 Pi2 Þ þ ki1 Ci2 ½1 pðCi2 ; li2 ; mi2 Þ ð1 Pi2 Þ þ ki1 ¼ gi2 ðCi2 Þ þ ki1 gi2 ðgi3 ð ðg2 ðg1 ðk1 Þ þ k2 Þ þ k3 Þ Þ þ ki2 Þ þ ki1 :
ð6Þ
3.2. Estimation of downstream revenues The second part of the reduction method of the state space involves elimination of the downstream stations. Again, we focus on station i. Customers departing from station i with a positive probability will enter station i þ 1 (and possibly additional downstream stations). Therefore, these customers will generate further income to the network, which should be taken into account when the system managers are formulating the admission control policy at station i. If we knew the number of customers at each downstream station, we could estimate (using dy-
namic programming) the future net revenue generated downstream. Indeed, the probability that a current customer at station i will be blocked at downstream stations increases with the number of current customers at these downstream stations; therefore the expected revenue this customer at station i will generate downstream is decreasing with the number of downstream customers. Our estimation technique is based on the idea that the total expected discounted net revenue a customer will generate at downstream stations can replace the explicit effect of including all stations in the dynamic programming problem. Net revenue is defined as the revenue generated by this customer minus the lost revenue from blocking of new customers due to this customer. Using this estimate, we base the control decision at station i on the reward a customer pays at station i plus the expected discounted net downstream revenue, denoted as Hi . The downstream revenue is independent of whether the customer at station i is internal or new. Our method therefore assumes a new customer at station i will contribute ri þ Hi , and an internal customer coming from station i 1 will contribute Ri þ Hi . Accompanying this change in revenue, we eliminate all stations downstream from station i, leaving only stations i 1 and i. The resulting model is a tandem queue, for which we can easily calculate the optimal admission policy. This optimal policy depends on the ratio of the revenues generated by internal versus new customers [6,7]. If downstream stations were eliminated, but downstream revenue was not included in the decision at station i, then the admission policy would be based on ri =Ri . By including an estimate of downstream revenue, the decision is based on ðri þ Hi Þ=ðRi þ Hi Þ. Since this latter ratio is higher than the former one, the switching curve when downstream revenue is taken into account will be higher than (or equal to) the switching curve in the two-dimensional state space when the downstream revenue is not taken into account [6,7]. Namely, consideration of future revenue will result in a lower reservation for internal customers at station i. We now proceed to describe the method for estimating discounted net downstream revenue, Hi
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
for 2 6 i 6 n. We start by observing that Hi depends on the expected net revenue earned at station i þ 1 plus Hiþ1 . Furthermore, the expected net revenue earned at station i þ 1 for an internal customer leaving station i is equal to the expected revenue paid at station i þ 1 minus the expected loss from any blocking of new customers due to the acceptance of this internal customer. We first approximate the throughputs of internal and new customers at station i þ 1. We approximate the result of the admission policy at station i þ 1, under low blocking of internal customers, as equivalent to internal customers receiving priority service. Consequently, we approximate the probability that an internal customer arriving at station i þ 1 will obtain service as 1 p di ð1 Pi Þ; liþ1 ; miþ1 ;
ð7Þ
which we denote by piþ1 . Therefore, the throughput for internal customers at station i þ 1 is approximately di ð1 Pi Þ½1 pðdi ð1 Pi Þ; liþ1 ; miþ1 Þ . This gives us a basis for estimating the revenue gained at station i þ 1. From our upstream flow estimation above, we approximate the total flow rate going through station i þ 1, which includes internal customers and new customers, as Ciþ1 ½1 pðCiþ1 ; liþ1 ; miþ1 Þ . Hence, the throughput for new customers arriving from outside the network is approximately given by the difference between the total flow rate and the internal customer throughput, i.e., Ciþ1 ½1 pðCiþ1 ; liþ1 ; miþ1 Þ di ð1 Pi Þ½1 pðdi ð1 Pi Þ; liþ1 ; miþ1 Þ : The difference between the arrival rate of new customers and the throughput of new customers is due to blocking. We attribute this blocking mostly to the presence of internal customers. Therefore, the reduction in new customer at station i þ 1, per internal customer passing through station i þ 1, is approximately given by the new customer arrival rate minus the new customer throughput, divided by the internal customer throughput,
171
kiþ1 Ciþ1 1 pðCiþ1 ; liþ1 ; miþ1 Þ þ di ð1 Pi Þ 1 p di ð1 Pi Þ; liþ1 ; miþ1 di ð1 Pi Þ 1 p di ð1 Pi Þ; liþ1 ; miþ1 ; ð8Þ which we denote by qiþ1 . This gives us a basis to estimate for estimating the loss in revenue due to blocking of new customers at station i þ 1. To complete the estimates, we must also take into account the discounting of these revenues and losses. Denote the uniformization constant as Q, henceforth referred to as the reciprocal of one unit of time [6,8]. Denote the discount factor per unit of time as a. We define bðlÞ ¼ aQ=l which represents the average discounting of future revenue for a customer receiving service at a rate l. We now use this discount factor, the rewards, and the losses obtained above to create an recursion to estimate the net discounted downstream revenue. We start with the last station n: Hn ¼ 0:
ð9Þ
A customer leaving station n 1 will gain service at station n with a probability of approximately ð1 Pn1 Þ½1 pðdn1 ð1 Pn1 Þ; ln ; mn Þ ¼ ð1 Pn1 Þpn : Our estimate for the downstream discounted net revenue for a customer leaving station n 1 is the downstream discounted revenue generated by this customer minus the discounted lost revenue from blocking of new customers due to this customer. The discounted net revenue at station n is equal to the revenue this customer would gain if admitted to station n minus the lost revenue due to potential blocking of a new customer plus the discounted net revenue from stations past nðRn qn rn þ Hn Þ. This quantity must be multiplied by the probability that a customer leaving station n 1 will be admitted to station n and by a discount factor to account for the time delay. Altogether this results in Hn1 ð1 Pn1 Þpn bðln1 ÞðRn qn rn þ Hn Þ ¼ ð1 Pn1 Þpn bðln1 ÞðRn qn rn Þ:
ð10Þ
172
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
The recursion continues with each further upstream station: Hi ð1 Pi Þpiþ1 bðli ÞðRiþ1 qiþ1 riþ1 þ Hiþ1 Þ
work. Numerical results were obtained by the method of successive approximation. 4.1. Three queues in series
ð1 Pi Þpiþ1 bðli ÞðRiþ1 qiþ1 riþ1 Þ The nominal system parameters are set as follows: m1 ¼ 8, m2 ¼ 7, m3 ¼ 8, a ¼ 0:99995, Q ¼ 2000, k1 ¼ 8:2, k2 ¼ 8:18, k3 ¼ 8, l1 ¼ 2:2, l2 ¼ 2:5, l3 ¼ 2:5, P1 ¼ 0:1, P2 ¼ 0:1, r1 ¼ 5, r2 ¼ 5, r3 ¼ 5, R2 ¼ 25 and R3 ¼ 25.
þ ð1 Pi Þð1 Piþ1 Þpiþ1 piþ2 bðli Þbðliþ1 Þ ðRiþ2 qiþ2 riþ2 þ Hiþ2 Þ " ! j n1 Y X ð1 Pk Þpkþ1 bðlk Þ ¼ j¼i
k¼i
#
Rjþ1 qjþ1 rjþ1 :
ð11Þ
3.3. Calculation of near-optimal policies In summary, the proposed procedure of reduction method operates as follows. First, we estimate the upstream rates for each station, resulting in Ci and di , as outlined in Section 3.1. Then, we estimate the downstream revenues for each station, resulting in Hi , as outlined in Section 3.2. Finally, each tandem pair of stations (i 1 and i) is analyzed as a separate system, using Ci1 and Hi . Then the optimal admission policy is calculated using dynamic programming, and is guaranteed to be a switching curve [6,7]. This policy is used, along with the policies for all other stations, as a policy for the network. We believe that this simple recursive approach results in a near-optimal policy.
4.1.1. Policy at station 3 The optimal admission policy imposed on new customers at station 3 is shown in Fig. 1. The policy has a threshold: a new customer will be accepted if and only if the state ði; j; kÞ of this system is below the switching surface. In order to estimate the near-optimal admission control policy for station 3, we eliminate station 1 and compute the expected arrival rate into station 2, i.e., C2 . Following our upstream rate estimation technique in (5): C2 k1 ½1 pðk1 ; l1 ; m1 Þ ð1 P1 Þ þ k2 ¼ 8:2 ð1 0:022549Þð1 0:1Þ þ 8:18 ¼ 15:3936: The optimal admission policy on new arrivals at station 3, for the resulting tandem queue, is shown in Fig. 2. As guaranteed, the policy is a switching curve. Acceptance decisions for new customers are now based solely on the occupancy of stations 2 and 3, ignoring station 1. This switching curve also
4. Numerical results In this section, we will demonstrate the accuracy of the approach. In the first subsection, we consider a network consisting of three stations in series. We compare the optimal policies in 3-D space with the near-optimal policies in 2-D space. We investigate the accuracy of this method under varying levels of traffic and under different line balancing. In the second subsection, we demonstrate the accuracy of the approach for a network consisting of six stations in series, to determine how the accuracy depends on the size of the net-
Fig. 1. The optimal admission policy on new customers at station 3.
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
173
and C3 d2 ð1 P2 Þ þ k3 ¼ 12:3874 0:9 þ 8 ¼ 19:1487: We then apply our downstream revenue estimation techniques (7) and (8): p3 ¼ 1 pðd2 ð1 P2 Þ; l3 ; m3 Þ ¼ 1 pð0:9d2 ; 2:5; 8Þ Fig. 2. The near-optimal admission policy on new customers at station 3.
¼ 1 pð11:1487; 2:5; 8Þ ¼ 0:9533; and
coincides with the cross-section of the optimal switching surface when station 1 has 2–4 customers. 4.1.2. Policy at station 2 The optimal admission control policy imposed on new customers at station 2 is shown in Fig. 3. This policy also has a threshold. In order to estimate the near-optimal admission control policy for station 2, we eliminate station 3 and compute the discounted expected net downstream revenue for a customer leaving station 2, i.e., H2 . We first use the upstream rate estimation recursion (3) and (6) to get d2 C2 ½1 pðC2 ; l2 ; m2 Þ ¼ 15:3936½1 pð15:3936; 2:5; 7Þ ¼ 12:3874;
q3 ¼ fk3 C3 ½1 pðC3 ; l3 ; m3 Þ þ d2 ð1 P2 Þ½1 pðd2 ð1 P2 Þ; l3 ; m3 Þ g =fd2 ð1 P2 Þ½1 pðd2 ð1 P2 Þ; l3 ; m3 Þ g 0:3411; and finally H2 ð1 P2 Þp3 bðl2 ÞðR3 q3 r3 Þ ¼ 0:9 0:9533 0:9608 ð25 0:3411 5Þ ¼ 19:2025: The admission policy at station 2 is then found by solving the dynamic programming problem associated with the tandem queues of stations 1 and 2 using r2 þ H2 24:2025 ; ¼ R2 þ H2 44:2025 and is shown in Fig. 4. As guaranteed, the policy is a switching curve. Acceptance decisions for new
Fig. 3. The optimal admission policy on new customers at station 2.
Fig. 4. The near-optimal admission policy on new customers at station 2.
174
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
customers are now based solely on the occupancy of stations 1 and 2, ignoring station 3. This switching curve also coincides with the crosssection of the optimal switching surface when station 3 has 0–4 customers. 4.1.3. Accuracy To evaluate the accuracy of the proposed techniques, the optimal and near-optimal admission control policies were numerically evaluated on the basis of the total discounted revenue they earn. For the purposes of comparison, an uncontrolled system was also evaluated, since the ‘‘admit all new customers’’ policy is often used in telecommunication networks. A range of initial states were evaluated, since discounted revenue depends on the initial state of the system. A typical result is shown in Table 1, wherein the initial occupancy of stations 2 and 3 are fixed, and the initial occupancy of station 1 is varied along its range. The revenue is increasing with the occupancy of station 1, due to the increased short-term (and thus lightly discounted) revenue that will be generated at stations 2 and 3 from these customers. The uncontrolled system generates approximately 94.7% of the revenue of the optimal policy, indi-
cating that there is a low level of blocking as designed. The near-optimal policy generates at least 99.9% of the revenue of the optimal policy. This increase in revenue is significant for telecommunication networks, since it represents a significant decrease in forced terminations, which are targeted to be very low. A relative performance measure is given the revenue difference between the optimal and near-optimal policies divided by the revenue difference between the optimal and uncontrolled systems, shown in the last row of the table. This performance index remains above 99.3%, indicating that the near-optimal policy gains almost all of the additional revenue that is achievable within the considered class of policies. We conclude that the algorithm works exceedingly well for this set of parameters. In the following sections, we investigate how the accuracy varies with load, line imbalance, and the size of the network. 4.1.4. Comparison with numerical examples of light traffic and heavy traffic In this subsection, we vary the loads on each of the three stations by varying the entire set of arrival rates. Light traffic is demonstrated in Table 2
Table 1 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with nominal parameters Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
4906.24 4904.72 4656.78 99.39
4931.12 4929.60 4679.56 99.40
4954.59 4953.07 4700.88 99.40
4976.60 4975.06 4720.67 99.40
4997.06 4995.50 4738.82 99.40
5015.81 5014.22 4755.18 99.39
5032.40 5030.78 4769.39 99.38
5045.77 5044.11 4780.61 99.37
5053.37 5051.66 4786.67 99.36
Table 2 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with k1 ¼ 4:2, k2 ¼ 4:18, and k3 ¼ 4 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
3161.39 3161.17 3145.22 98.64
3197.15 3196.93 3180.41 98.69
3231.83 3231.61 3214.45 98.73
3265.18 3264.94 3247.09 98.67
3296.94 3296.68 3278.08 98.62
3326.87 3326.56 3307.18 98.43
3354.66 3354.28 3334.05 98.16
3379.42 3378.97 3357.80 97.92
3397.53 3397.04 3374.99 97.83
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
by lowering the arrival rates to (k1 ¼ 4:2, k2 ¼ 4:18, k3 ¼ 4). Heavy traffic is demonstrated in Table 3 by raising the arrival rates to (k1 ¼ 16:2, k2 ¼ 16:18, k3 ¼ 16). We find that the algorithm produces excellent results in all three cases. In light traffic, the uncontrolled system generates approximately 99.3% of the possible revenue, while the near-optimal policy increases this to above 99.9%. The relative performance index decreases, but remains above 97.8%. In heavy traffic, the uncontrolled system generates approximately 83.8% of the possible revenue, while the near-optimal policy increases this to above 99.8%. The relative performance index remains above 99.1%. We conclude that the approximation is fairly insensitive to proportional changes in load at all stations, and works well in a wide range of loads. 4.1.5. Variation with line imbalance In this subsection, we vary the load on an individual station to create line imbalance, by varying the service rate of a single station. In Tables 4 and 5, we demonstrate the effect of increasing the service rate in the first station, l1 , first to 5.5 and then to 8.8. This results in a network in which the
175
first station is relatively fast compared to downstream stations. The near-optimal policy continues to achieve a performance index above 99.3%, increasing with l1 . This is to be expected. A higher service rate at station 1 results in less sensitivity to the occupancy of station 1 in the policy at station 3. Since the near-optimal policy at station 3 is not based on the occupancy of station 1, a higher service rate results in less error. A better test is to vary the service rate of the second station. In Tables 6 and 7, we increase l2 to 6.5 and then to 10. This results in a network in which the second station is relatively fast compared both to the upstream and the downstream station. We may worry that this may result in higher sensitivity to the occupancy of station 1 in the policy at station 3. As a result, the nearoptimal policy will fare worse, since the policy at station 3 is not based on the occupancy of station 1. Our numerical results, however, demonstrate that the performance degradation is small. The near-optimal policy continues to generate at least 99.9% of the achievable revenue with a performance index above 98.8%. We conclude that the algorithm’s performance is fairly insensitive to imbalance in the loads at different stations.
Table 3 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with k1 ¼ 16:2, k2 ¼ 16:18, and k3 ¼ 16 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
6158.20 6149.72 5170.05 99.14
6171.39 6162.91 5180.35 99.14
6183.56 6175.08 5189.60 99.15
6194.63 6186.14 5197.76 99.15
6204.38 6195.88 5204.73 99.15
6212.56 6204.05 5210.36 99.15
6218.84 6210.31 5214.42 99.15
6222.68 6214.13 5216.47 99.15
6223.03 6214.49 5215.73 99.15
Table 4 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l1 ¼ 5:5 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5002.86 5001.27 4729.85 99.42
5028.76 5027.16 4753.36 99.42
5052.59 5050.97 4774.67 99.42
5074.23 5072.58 4793.59 99.41
5093.69 5091.99 4810.06 99.40
5111.06 5109.32 4824.23 99.39
5126.54 5124.75 4836.27 99.38
5139.61 5137.78 4846.22 99.38
5148.53 5146.68 4852.88 99.37
176
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
Table 5 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l1 ¼ 8:8 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5023.54 5022.88 4741.63 99.77
5049.67 5049.01 4765.29 99.77
5073.36 5072.68 4786.37 99.76
5094.32 5093.58 4804.45 99.74
5112.44 5111.62 4819.40 99.72
5127.94 5127.10 4831.51 99.72
5140.44 5139.59 4841.25 99.72
5150.18 5149.32 4849.04 99.71
5156.72 5155.86 4854.40 99.72
Table 6 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l2 ¼ 6:5 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5662.27 5658.77 5325.93 98.96
5694.50 5691.07 5355.08 98.99
5725.94 5722.54 5383.53 99.01
5756.44 5753.03 5411.16 99.01
5785.75 5782.32 5437.79 99.01
5813.47 5810.02 5463.05 99.02
5838.82 5835.33 5486.22 99.01
5860.10 5856.59 5505.73 99.01
5873.33 5869.81 5517.82 99.01
Table 7 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l2 ¼ 10 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5691.82 5688.07 5353.66 98.89
5723.89 5720.23 5382.81 98.93
5755.22 5751.60 5411.33 98.95
5785.68 5782.07 5439.13 98.96
5815.07 5811.45 5466.05 98.96
5842.99 5839.34 5491.74 98.96
5868.66 5864.99 5515.46 98.96
5890.36 5886.68 5535.61 98.96
5903.98 5900.30 5548.24 98.97
4.2. Six queues in series We now consider a network consisting of six tandem queues to investigate the dependence of the proposed method’s accuracy with the size of the network. We expect that the accuracy should degrade as the number of stations increases, since
the method is based on solving for the admission policy in a set of tandem queues. In Table 8, we present results using the following parameters: m1 ¼ 5, m2 ¼ 5, m3 ¼ 5, m4 ¼ 5, m5 ¼ 5, m6 ¼ 5, a ¼ 0:9999, Q ¼ 2000, k1 ¼ 5, k2 ¼ 5, k3 ¼ 5, k4 ¼ 5, k5 ¼ 5, k6 ¼ 5, l1 ¼ 6, l2 ¼ 5:5, l3 ¼ 5, l4 ¼ 4:5, l5 ¼ 4, l6 ¼ 3:5, P1 ¼ 0:2,
Table 8 The expected discounted revenues of optimal, near-optimal and no-control policies for six-queue-in-series system Control Optimal Near-optimal No-control Performance index (%)
State ð2; 3; 0; 1; 4; 2Þ
ð2; 3; 1; 1; 4; 2Þ
ð2; 3; 2; 1; 4; 2Þ
ð2; 3; 3; 1; 4; 2Þ
ð2; 3; 4; 1; 4; 2Þ
ð2; 3; 5; 1; 4; 2Þ
2869.767 2865.421 2772.975 95.51
2884.461 2880.006 2786.745 95.44
2897.215 2892.623 2798.582 95.34
2907.249 2902.483 2807.728 95.21
2913.160 2908.181 2812.832 95.04
2912.321 2907.230 2811.499 94.95
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
P2 ¼ 0:2, P3 ¼ 0:2, P4 ¼ 0:2, P5 ¼ 0:2, r1 ¼ 1, r2 ¼ 1, r3 ¼ 1, r4 ¼ 1, r5 ¼ 1, r6 ¼ 1, R2 ¼ 15, R3 ¼ 15, R4 ¼ 15, R5 ¼ 15 and R6 ¼ 15. We find that the accuracy does degrade with an increasing number of stations, but not much. The nearoptimal policy continues to generate at least 99.8% of the revenue of the optimal policy (compared to 96.6% for the uncontrolled system). The relative difference between the revenue generated by the near-optimal policy and by no control remains at least 94.9% of the difference between the optimal policy and no control. This minor decrease in accuracy, however, is achieved with dramatically less computation. The optimal policy, found using dynamic programming on the six-dimensional state space, requires several days of computation on a typical computer. In contrast, the near-optimal policy, found using the procedure proposed here, requires a few minutes of computation.
5. Conclusion We have considered access control policies in multiserver loss queues in series. Such models arise in computer and telecommunication networks, in which continued downstream service to the internal customers is preferable to admission of new customers. We are particularly motivated by telecommunication networks in which forced termination is considered to be an order of magnitude worse than blocking of new customers, and in which it is often the goal to achieve very low forced termination probabilities. A direct dynamic programming approach to determining the optimal admission policy at each station results in a computational complexity that is geometric in the number of stations. We proposed a simple recursive method to solve the problem using dynamic programming on a set of reduced state spaces. This approach includes a rate estimation technique for upstream stations, and a revenue estimation technique for downstream stations. Numerical results demonstrate that these nearoptimal policies are accurate under a wide range of loads. Increasing line imbalance and (or) in-
177
creasing the number of station in the network resulting in a minor degradation of the performance of the proposed algorithm. However, the decrease in computation can be many orders of magnitude.
References [1] E. Altman, S. Stidham, Optimality of monotonic policies for two-action Markovian decision processes, with applications to control of queues with delayed information, Queueing Systems 21 (1995) 267–291. [2] R.-R. Chen, S. Meyn, Value iteration and optimization of multiclass queueing networks, Queueing Systems 32 (1999) 65–97. [3] P.-J. Courtois, G. Scheys, Minimization of the total loss rate for two finite queues in series, IEEE Transactions on Communications 39 (1991) 1651–1661. [4] H. Ghoneim, S. Stidham, Optimal control of arrivals to two queues in series, European Journal of Operational Research 21 (1985) 399–409. [5] G. Koole, Structural results for the control of queueing systems using event-based dynamic programming, Queueing Systems 30 (1998) 323–339. [6] C.-Y. Ku, Access control for loss network, Ph.D. Dissertation, Department of EECS, Northwestern University, 1995. [7] C.-Y. Ku, S. Jordan, Access control to two multiserver loss queues in series, IEEE Transactions on Automatic Control 42 (1997) 1017–1023. [8] P.R. Kumar, P.P. Varaiya, Stochastic Systems, PrenticeHall, Englewood Cliffs, NJ, 1986. [9] J.M. Hah, P.L. Tien, M.C. Yuang, Neural-network-based call admission control in ATM networks with heterogeneous arrivals, Computer Communications 20 (1997) 732– 740. [10] G. Karlsson, Capacity reservation in ATM networks, Computer Communications 19 (1996) 180–193. [11] T.-H. Lee, K.-C. Lai, S.-T. Duann, Design of a real-time call admission controller for ATM networks, IEEE/ACM Transactions on Networking 4 (1996) 758–765. [12] P. Mohge, I. Rubin, Reserving for future clients in a multipoint application – why and how, IEEE Journal on Selected Areas in Communications 15 (1997) 531– 544. [13] J.S. Park, S.H. Lee, S.C. Kim, J.Y. Lee, S.B. Lee, A conferencing system for real-time, multiparty, multimedia services, IEEE Transactions on Consumer Electronics 44 (1998) 857–865. [14] S.M. Ross, Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983. [15] S. Stidham, R. Weber, A survey of Markov decision models for control of networks of queues, Queueing Systems 13 (1993) 291–314.
178
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
[16] S. Stidham, Optimal control of admission to a queueing system, IEEE Transactions on Automatic Control 30 (1985) 705–713.
[17] R. Zhang, Y.A. Phillis, Fuzzy control of arrivals to tandem queues with two stations, IEEE Transactions on Fuzzy Systems 7 (1999) 361–367.
Stochastics and Statistics
Near optimal admission control for multiserver loss queues in series Cheng-Yuan Ku a
a,*
, Scott Jordan
b
Department of Information Management, National Chung Cheng University, 160 San-Hsing, Chia-Yi County, Taiwan, ROC b Department of Electrical and Computer Engineering, University of California, Irvine, CA 92697-2625, USA Received 26 March 2000; accepted 18 October 2001
Abstract This paper considers access control policies in multiserver loss queues in series such as might arise in the context of computer and telecommunication networks. Each queue is presented with both served upstream customers and Poisson arrivals from outside the network, and it may route serviced customers out of the network or to the downstream queue. Service times of each customer are i.i.d. and exponentially distributed. Revenue is earned by each station when it serves a customer, but the amount of revenue depends on whether the customer entered the network at this station or was routed from an upstream station. We propose a simple recursive method to solve the problem using dynamic programming on a set of reduced state spaces. This approach includes a rate estimation technique for upstream stations, and a revenue estimation technique for downstream stations. Numerical results demonstrate the performance of these near-optimal policies under light, moderate, and heavy traffic. Ó 2002 Elsevier Science B.V. All rights reserved. Keywords: Queueing; Dynamic programming; Connection admission control
1. Introduction This paper considers access control policies in multiserver loss queues in series. Each queue is presented with arrivals of customers of two types: a stream of ‘‘internal customers’’ from the previous queue, and a Poisson stream of ‘‘new customers’’ from outside the network. Each queue has an arbitrary number of servers, but no queue.
*
Corresponding author. Tel.: +886-5-2720411x34603. E-mail addresses: [email protected] (C.-Y. Ku), [email protected] (S. Jordan).
Service times of each customer are i.i.d., but the revenue generated is differentiated, with internal customers typically paying more than new customers. Departures from the queue are Bernoulli routed either to the next queue or out of the network, independent of their type. We are interested in the access control policy, which will make a decision to accept or deny upon the arrival of any customer at each queue. The objective here is to maximize the total discounted revenue over an infinite horizon. This access control problem arises in the context of computer and telecommunication networks, e.g., Internet, cellular phone networks, and
0377-2217/03/$ - see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 1 ) 0 0 3 8 7 - 3
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
asynchronous transfer mode (ATM) [1,3,9,17]. Computer networks were originally designed for ‘‘best-effort traffic’’, and typically serve traffic on a first-come first-served basis in a series of queues. Telecommunication networks, on the other hand, were originally designed for real-time traffic using a ‘‘circuit-switched architecture’’ – each station has essentially no queue and reserves one server (defined as the transmission capacity required for one call) for each call passing through it. Both types of networks are currently evolving toward multi-service architectures, in which transmission of traffic at each station is differentiated based on the service type. In particular, we are interested here in real-time traffic. The new network architectures strictly limit queueing time for real-time traffic, because of their inherent low-delay requirements. In addition, these architectures implement some type of reservation or priority mechanisms to give real-time traffic preference over best-effort traffic [10–13]. These reservation or priority mechanisms typically result in a connection admission control (CAC) policy on real-time calls, based on the capacity of each station. In this paper, we model this type of network as a series of multiserver loss queues. Only a single series of queues is considered here; extensions to a general network topology will be considered at a later time. We assume that real-time traffic is given preference over best-effort traffic; hence only realtime traffic is modeled. Delay is minimized by not allowing real-time traffic to queue at any station. We only model the network traffic on an intermediate timescale. A ‘‘customer’’ corresponds to a burst of real-time traffic, i.e., a randomly sized batch of packets; we do not explicitly model longterm timescale phenomena, e.g., calls. The CAC policy is modeled by a server policy at each station that can reserve any number of servers for internal customers. In order to allow the policy to achieve the desired balance of blocking of internal and new customers, we assume that each customer pays revenue to each station it passes through. The amount of the revenue can depend on the station and on its type. In general, internal customers would be assumed to pay more, in order to bias the network in favor of completing transmission of
167
traffic that has already received service at a previous station. In the long and rich literature on stochastic dynamic programming, there have been many authors that have chosen to use expected discounted measures and many authors that have chosen to use average per unit time measures. With respect to the particular problem addressed in this paper, we believe that expected discounted revenue is a better choice for two reasons. First, it is possible that some type of pricing may be used in the future in computer networks such as the Internet. In such systems, discounting naturally reflects the presence of an underlying interest rate, and maximizing expected discounted revenue more closely corresponds to the likely desired optimization metric. Second, results concerning the structure of policies that maximize expected discounted revenue can often be used to prove similar structural results for policies that maximize average revenue per unit time. However, the converse is not true (see e.g., [14] for a general presentation of such correspondences and [15,16] for a discussion of why expected discounted cost measures have often been chosen in papers on admission control in queueing networks). We therefore choose to present these structural results using expected discounted revenue as the optimization metric. A good survey of research on access control policies in queueing networks can be found in [15,16]. Most of the research on this topic, however, focuses on infinite queue models and uses holding costs as the performance metric. In contrast, here we assume a loss network, and use loss as the performance metric. The typical approach to determine the optimal decision policy at each queue in a network, as a function of the number of customers at each station, is to use dynamic programming [2,4–6,15]. However, the size of the Markov chain for the entire network grows exponentially with the number of stations, so this approach quickly becomes numerically infeasible. Our goal in this paper is to reduce the amount of information contained in the state, and correspondingly the size of the state space. We borrow an idea often used in the stochastic modeling
168
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
literature, where a common approach models only the bottleneck queue(s) in a network, by recursively estimating internal flow rates. We are unaware of such an approach being used in optimization of loss networks. We propose determining a near-optimal policy for station i by creating a reduced model whose state contains only the number of customers at station i and the number of customers at its immediate upstream station i 1. This procedure ignores the states of stations upstream from i 1, so it requires estimation of the flow into that station. We also need to eliminate stations downstream from i. This reduction, however, contributes significant error to the decision as to whether to accept new customers. Therefore we propose a method to estimate the downstream revenue and to take this information into account at station i. The control policy at one station affects upstream stations by altering their expected downstream revenue, and it affects downstream stations by altering their expected flow of internal customers. We therefore propose to apply these two techniques, rate estimation and revenue estimation, using simple recursions. Numerical results are presented to demonstrate the success of this approach. The multiserver-loss-queues-in-series model and problem formulation is presented in Section 2. In Section 3, the state reduction method, along with rate and revenue estimation techniques, is proposed. In Section 4, the numerical results are presented.
2. Model and problem formulation Denote by n the number of stations in the tandem network. Station i is a non-preemptive loss queue with mi servers. Each station is presented with a Poisson stream of new customers, with arrival rate ki . Service times are i.i.d. and exponentially distributed with rate li . Departures from station i are routed out of the network with probability Pi or to station i þ 1 otherwise, independent where the customer entered the network, the customer’s service time, and of other customers. Revenue is paid by each customer at the start of service. In telecommunication networks, blocking
of customers who have already been accepted into the network (herein referred to as ‘‘internal customers’’) is known as ‘‘forced termination’’ and is considered to be an order of magnitude worse than blocking of new customers. Such networks often target very low forced termination rates. One model of this metric would be to directly impose a cost Ci on each blocked internal customer. We instead adopt an alternate model in which revenue is earned for every customer accepted at every station, but the revenue for internal customers is much higher than for new customers. This alternate model, although not in general equivalent to the cost-based model, still biases the network in favor of internal customers, and more easily lends itself to analysis. Specifically, we assume that a new customer who enters the network at station i pays an amount ri at station i, and that an internal customer pays an amount Ri at station i. We assume Ri > ri ð2 6 i 6 nÞ, so that internal customers are preferable to new customers at station i, and that 1 > Pi P 0 ð1 6 i 6 n 1Þ. We consider connection admission control policies that are capable of accepting or blocking arrivals of each customer type (internal or new) at each station. Blocked arrivals are lost. Our objective is to maximize the total discounted revenue over an infinite horizon. It has been shown that the optimal policy does not block customers at station 1 and does not block internal customers at any station [6,7]. Admission policies therefore reduce to determining whether to block or accept new arrivals at station i; 2 6 i 6 n. For a two-stationin-tandem network, the optimal policy can be shown to consist of a switching curve in the twodimensional state space [6,7]. On one side of the curve, new arrivals are accepted; on the other side, new arrivals are blocked. Unfortunately, a similar characterization for the optimal policy in a series of more than two stations is unknown. Our objective here is to numerically determine a nearoptimal policy.
3. State space reduction Dynamic programming can be used to determine the optimal policy, which states whether to
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
accept or block each new arrival at each station as a function of the number of current customers at each station in the network. However, the numerical complexity grows exponentially with the dimension of the state space, n. This renders computation of the optimal policy difficult for all but very small n. In this section, we propose a technique to reduce the dynamic programming problem to a collection of smaller problems, each of which consists of determining a switching curve for a single station. Consider a particular station i. We propose determining a near-optimal policy for station i by reducing the state to a vector containing only the number of customers at station i and the number of customers at its immediate upstream station i 1. This procedure requires estimation of two quantities: (1) the flow into station i 1 and (2) the expected discounted reward paid at downstream stations. We will denote our estimate of the flow into station i by Ci , our estimate of the flow out of station i by di , and our estimate of the downstream discounted net revenue for a customer leaving station i by Hi . The estimation of rates is discussed in the first subsection, and the estimation of rewards is discussed in the second subsection. 3.1. Estimation of upstream rates Reduction of the state space will involve elimination of both upstream and downstream stations, reducing the state space for a single station to a vector containing the number of customers in that station and the number in its immediate upstream neighbor. This tandem queue can be easily analyzed. In this subsection, we consider elimination of upstream stations. Focus on station i. Elimination of stations 1 through i 2 will reduce the information we have about short-term arrival rate variations of internal customers. The number of servers reserved for internal customers at station i will typically be increasing with the number of current customers present at each station upstream from i. However, we expect that the contribution of an upstream station to the reservation level at station i will be decreasing with the distance between the two sta-
169
tions. A reasonable approach, therefore, is to eliminate all but the immediate upstream neighbor, station i 1. (A more accurate, but computationally more intensive, model is to keep additional upstream stations, e.g., from i 2 to i and estimate the arrival rate into station i 2.) We make two approximations to simplify the model. First, we assume that the arrival process into station i 1 is Poisson. The arrival process into station i 1 is the multiplexing of the departure process from station i 2, thinned by those departing the network, and the new arrivals into station i 1. Since the departure process from station i 2 is not Poisson, the arrival process into station i 1 is not Poisson. Nevertheless, we have found this approximation to yield good results. Second, for purposes of estimating the arrival rate into station i 1, we model the upstream stations 1 through i 1 as uncontrolled loss queues. The rate of the departures from station i 2 depends on the control policy used in station i 2. However, under a low level of blocking, ignoring this control policy is a reasonable approximation. Under higher blocking rates, this second approximation can be avoided by explicitly using information about the control policies used in upstream stations. The resulting method then includes an iteration, which significantly increases the computational complexity [6,7]. Our approach requires estimation of the arrival rate of customers into station i 1, Ci1 . We approximate the flow into station i 1 as a Poisson process, with a rate equal to the flow rate from station i 2 to station i 1 plus the rate of new arrivals into station i 1. Every station in this system is a M=M=m=m loss queue. Therefore, the loss probability is given by the Erlang B formula. If the arrival rate is k and the service rate is l, then Blocking probability m
ðk=lÞ =m! pðk; l; mÞ ¼ Pm : k k¼0 ðk=lÞ =k!
ð1Þ
For station j; 2 6 j 6 i 1, we estimate rates using conservation of flow: Cj dj1 ð1 Pj1 Þ þ kj :
ð2Þ
170
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
As noted above, this approximation method of the arrival rate depends on the departure rate of the previous station. Using the Erlang B formula, we can approximate the departure rate as dj1 Cj1 1 pðCj1 ; lj1 ; mj1 Þ : ð3Þ Denote gi ðkÞ k½1 pðk; li ; mi Þ ð1 Pi Þ. This approximation can then be implemented using a recursion, and the two preceding equations. For station 1, the estimated arrival rate is equal to the arrival rate of new customers: C1 ¼ k1 :
ð4Þ
For station 2, the estimated arrival rate is based on the departure rate of station 1 customers and the arrival of new customers: C2 d1 ð1 P1 Þ þ k2 C1 ½1 pðC1 ; l1 ; m1 Þ ð1 P1 Þ þ k2 ¼ g1 ðC1 Þ þ k2 ¼ g1 ðk1 Þ þ k2 :
ð5Þ
The recursion continues by estimating the arrival rate into station i 1: Ci1 di2 ð1 Pi2 Þ þ ki1 Ci2 ½1 pðCi2 ; li2 ; mi2 Þ ð1 Pi2 Þ þ ki1 ¼ gi2 ðCi2 Þ þ ki1 gi2 ðgi3 ð ðg2 ðg1 ðk1 Þ þ k2 Þ þ k3 Þ Þ þ ki2 Þ þ ki1 :
ð6Þ
3.2. Estimation of downstream revenues The second part of the reduction method of the state space involves elimination of the downstream stations. Again, we focus on station i. Customers departing from station i with a positive probability will enter station i þ 1 (and possibly additional downstream stations). Therefore, these customers will generate further income to the network, which should be taken into account when the system managers are formulating the admission control policy at station i. If we knew the number of customers at each downstream station, we could estimate (using dy-
namic programming) the future net revenue generated downstream. Indeed, the probability that a current customer at station i will be blocked at downstream stations increases with the number of current customers at these downstream stations; therefore the expected revenue this customer at station i will generate downstream is decreasing with the number of downstream customers. Our estimation technique is based on the idea that the total expected discounted net revenue a customer will generate at downstream stations can replace the explicit effect of including all stations in the dynamic programming problem. Net revenue is defined as the revenue generated by this customer minus the lost revenue from blocking of new customers due to this customer. Using this estimate, we base the control decision at station i on the reward a customer pays at station i plus the expected discounted net downstream revenue, denoted as Hi . The downstream revenue is independent of whether the customer at station i is internal or new. Our method therefore assumes a new customer at station i will contribute ri þ Hi , and an internal customer coming from station i 1 will contribute Ri þ Hi . Accompanying this change in revenue, we eliminate all stations downstream from station i, leaving only stations i 1 and i. The resulting model is a tandem queue, for which we can easily calculate the optimal admission policy. This optimal policy depends on the ratio of the revenues generated by internal versus new customers [6,7]. If downstream stations were eliminated, but downstream revenue was not included in the decision at station i, then the admission policy would be based on ri =Ri . By including an estimate of downstream revenue, the decision is based on ðri þ Hi Þ=ðRi þ Hi Þ. Since this latter ratio is higher than the former one, the switching curve when downstream revenue is taken into account will be higher than (or equal to) the switching curve in the two-dimensional state space when the downstream revenue is not taken into account [6,7]. Namely, consideration of future revenue will result in a lower reservation for internal customers at station i. We now proceed to describe the method for estimating discounted net downstream revenue, Hi
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
for 2 6 i 6 n. We start by observing that Hi depends on the expected net revenue earned at station i þ 1 plus Hiþ1 . Furthermore, the expected net revenue earned at station i þ 1 for an internal customer leaving station i is equal to the expected revenue paid at station i þ 1 minus the expected loss from any blocking of new customers due to the acceptance of this internal customer. We first approximate the throughputs of internal and new customers at station i þ 1. We approximate the result of the admission policy at station i þ 1, under low blocking of internal customers, as equivalent to internal customers receiving priority service. Consequently, we approximate the probability that an internal customer arriving at station i þ 1 will obtain service as 1 p di ð1 Pi Þ; liþ1 ; miþ1 ;
ð7Þ
which we denote by piþ1 . Therefore, the throughput for internal customers at station i þ 1 is approximately di ð1 Pi Þ½1 pðdi ð1 Pi Þ; liþ1 ; miþ1 Þ . This gives us a basis for estimating the revenue gained at station i þ 1. From our upstream flow estimation above, we approximate the total flow rate going through station i þ 1, which includes internal customers and new customers, as Ciþ1 ½1 pðCiþ1 ; liþ1 ; miþ1 Þ . Hence, the throughput for new customers arriving from outside the network is approximately given by the difference between the total flow rate and the internal customer throughput, i.e., Ciþ1 ½1 pðCiþ1 ; liþ1 ; miþ1 Þ di ð1 Pi Þ½1 pðdi ð1 Pi Þ; liþ1 ; miþ1 Þ : The difference between the arrival rate of new customers and the throughput of new customers is due to blocking. We attribute this blocking mostly to the presence of internal customers. Therefore, the reduction in new customer at station i þ 1, per internal customer passing through station i þ 1, is approximately given by the new customer arrival rate minus the new customer throughput, divided by the internal customer throughput,
171
kiþ1 Ciþ1 1 pðCiþ1 ; liþ1 ; miþ1 Þ þ di ð1 Pi Þ 1 p di ð1 Pi Þ; liþ1 ; miþ1 di ð1 Pi Þ 1 p di ð1 Pi Þ; liþ1 ; miþ1 ; ð8Þ which we denote by qiþ1 . This gives us a basis to estimate for estimating the loss in revenue due to blocking of new customers at station i þ 1. To complete the estimates, we must also take into account the discounting of these revenues and losses. Denote the uniformization constant as Q, henceforth referred to as the reciprocal of one unit of time [6,8]. Denote the discount factor per unit of time as a. We define bðlÞ ¼ aQ=l which represents the average discounting of future revenue for a customer receiving service at a rate l. We now use this discount factor, the rewards, and the losses obtained above to create an recursion to estimate the net discounted downstream revenue. We start with the last station n: Hn ¼ 0:
ð9Þ
A customer leaving station n 1 will gain service at station n with a probability of approximately ð1 Pn1 Þ½1 pðdn1 ð1 Pn1 Þ; ln ; mn Þ ¼ ð1 Pn1 Þpn : Our estimate for the downstream discounted net revenue for a customer leaving station n 1 is the downstream discounted revenue generated by this customer minus the discounted lost revenue from blocking of new customers due to this customer. The discounted net revenue at station n is equal to the revenue this customer would gain if admitted to station n minus the lost revenue due to potential blocking of a new customer plus the discounted net revenue from stations past nðRn qn rn þ Hn Þ. This quantity must be multiplied by the probability that a customer leaving station n 1 will be admitted to station n and by a discount factor to account for the time delay. Altogether this results in Hn1 ð1 Pn1 Þpn bðln1 ÞðRn qn rn þ Hn Þ ¼ ð1 Pn1 Þpn bðln1 ÞðRn qn rn Þ:
ð10Þ
172
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
The recursion continues with each further upstream station: Hi ð1 Pi Þpiþ1 bðli ÞðRiþ1 qiþ1 riþ1 þ Hiþ1 Þ
work. Numerical results were obtained by the method of successive approximation. 4.1. Three queues in series
ð1 Pi Þpiþ1 bðli ÞðRiþ1 qiþ1 riþ1 Þ The nominal system parameters are set as follows: m1 ¼ 8, m2 ¼ 7, m3 ¼ 8, a ¼ 0:99995, Q ¼ 2000, k1 ¼ 8:2, k2 ¼ 8:18, k3 ¼ 8, l1 ¼ 2:2, l2 ¼ 2:5, l3 ¼ 2:5, P1 ¼ 0:1, P2 ¼ 0:1, r1 ¼ 5, r2 ¼ 5, r3 ¼ 5, R2 ¼ 25 and R3 ¼ 25.
þ ð1 Pi Þð1 Piþ1 Þpiþ1 piþ2 bðli Þbðliþ1 Þ ðRiþ2 qiþ2 riþ2 þ Hiþ2 Þ " ! j n1 Y X ð1 Pk Þpkþ1 bðlk Þ ¼ j¼i
k¼i
#
Rjþ1 qjþ1 rjþ1 :
ð11Þ
3.3. Calculation of near-optimal policies In summary, the proposed procedure of reduction method operates as follows. First, we estimate the upstream rates for each station, resulting in Ci and di , as outlined in Section 3.1. Then, we estimate the downstream revenues for each station, resulting in Hi , as outlined in Section 3.2. Finally, each tandem pair of stations (i 1 and i) is analyzed as a separate system, using Ci1 and Hi . Then the optimal admission policy is calculated using dynamic programming, and is guaranteed to be a switching curve [6,7]. This policy is used, along with the policies for all other stations, as a policy for the network. We believe that this simple recursive approach results in a near-optimal policy.
4.1.1. Policy at station 3 The optimal admission policy imposed on new customers at station 3 is shown in Fig. 1. The policy has a threshold: a new customer will be accepted if and only if the state ði; j; kÞ of this system is below the switching surface. In order to estimate the near-optimal admission control policy for station 3, we eliminate station 1 and compute the expected arrival rate into station 2, i.e., C2 . Following our upstream rate estimation technique in (5): C2 k1 ½1 pðk1 ; l1 ; m1 Þ ð1 P1 Þ þ k2 ¼ 8:2 ð1 0:022549Þð1 0:1Þ þ 8:18 ¼ 15:3936: The optimal admission policy on new arrivals at station 3, for the resulting tandem queue, is shown in Fig. 2. As guaranteed, the policy is a switching curve. Acceptance decisions for new customers are now based solely on the occupancy of stations 2 and 3, ignoring station 1. This switching curve also
4. Numerical results In this section, we will demonstrate the accuracy of the approach. In the first subsection, we consider a network consisting of three stations in series. We compare the optimal policies in 3-D space with the near-optimal policies in 2-D space. We investigate the accuracy of this method under varying levels of traffic and under different line balancing. In the second subsection, we demonstrate the accuracy of the approach for a network consisting of six stations in series, to determine how the accuracy depends on the size of the net-
Fig. 1. The optimal admission policy on new customers at station 3.
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
173
and C3 d2 ð1 P2 Þ þ k3 ¼ 12:3874 0:9 þ 8 ¼ 19:1487: We then apply our downstream revenue estimation techniques (7) and (8): p3 ¼ 1 pðd2 ð1 P2 Þ; l3 ; m3 Þ ¼ 1 pð0:9d2 ; 2:5; 8Þ Fig. 2. The near-optimal admission policy on new customers at station 3.
¼ 1 pð11:1487; 2:5; 8Þ ¼ 0:9533; and
coincides with the cross-section of the optimal switching surface when station 1 has 2–4 customers. 4.1.2. Policy at station 2 The optimal admission control policy imposed on new customers at station 2 is shown in Fig. 3. This policy also has a threshold. In order to estimate the near-optimal admission control policy for station 2, we eliminate station 3 and compute the discounted expected net downstream revenue for a customer leaving station 2, i.e., H2 . We first use the upstream rate estimation recursion (3) and (6) to get d2 C2 ½1 pðC2 ; l2 ; m2 Þ ¼ 15:3936½1 pð15:3936; 2:5; 7Þ ¼ 12:3874;
q3 ¼ fk3 C3 ½1 pðC3 ; l3 ; m3 Þ þ d2 ð1 P2 Þ½1 pðd2 ð1 P2 Þ; l3 ; m3 Þ g =fd2 ð1 P2 Þ½1 pðd2 ð1 P2 Þ; l3 ; m3 Þ g 0:3411; and finally H2 ð1 P2 Þp3 bðl2 ÞðR3 q3 r3 Þ ¼ 0:9 0:9533 0:9608 ð25 0:3411 5Þ ¼ 19:2025: The admission policy at station 2 is then found by solving the dynamic programming problem associated with the tandem queues of stations 1 and 2 using r2 þ H2 24:2025 ; ¼ R2 þ H2 44:2025 and is shown in Fig. 4. As guaranteed, the policy is a switching curve. Acceptance decisions for new
Fig. 3. The optimal admission policy on new customers at station 2.
Fig. 4. The near-optimal admission policy on new customers at station 2.
174
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
customers are now based solely on the occupancy of stations 1 and 2, ignoring station 3. This switching curve also coincides with the crosssection of the optimal switching surface when station 3 has 0–4 customers. 4.1.3. Accuracy To evaluate the accuracy of the proposed techniques, the optimal and near-optimal admission control policies were numerically evaluated on the basis of the total discounted revenue they earn. For the purposes of comparison, an uncontrolled system was also evaluated, since the ‘‘admit all new customers’’ policy is often used in telecommunication networks. A range of initial states were evaluated, since discounted revenue depends on the initial state of the system. A typical result is shown in Table 1, wherein the initial occupancy of stations 2 and 3 are fixed, and the initial occupancy of station 1 is varied along its range. The revenue is increasing with the occupancy of station 1, due to the increased short-term (and thus lightly discounted) revenue that will be generated at stations 2 and 3 from these customers. The uncontrolled system generates approximately 94.7% of the revenue of the optimal policy, indi-
cating that there is a low level of blocking as designed. The near-optimal policy generates at least 99.9% of the revenue of the optimal policy. This increase in revenue is significant for telecommunication networks, since it represents a significant decrease in forced terminations, which are targeted to be very low. A relative performance measure is given the revenue difference between the optimal and near-optimal policies divided by the revenue difference between the optimal and uncontrolled systems, shown in the last row of the table. This performance index remains above 99.3%, indicating that the near-optimal policy gains almost all of the additional revenue that is achievable within the considered class of policies. We conclude that the algorithm works exceedingly well for this set of parameters. In the following sections, we investigate how the accuracy varies with load, line imbalance, and the size of the network. 4.1.4. Comparison with numerical examples of light traffic and heavy traffic In this subsection, we vary the loads on each of the three stations by varying the entire set of arrival rates. Light traffic is demonstrated in Table 2
Table 1 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with nominal parameters Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
4906.24 4904.72 4656.78 99.39
4931.12 4929.60 4679.56 99.40
4954.59 4953.07 4700.88 99.40
4976.60 4975.06 4720.67 99.40
4997.06 4995.50 4738.82 99.40
5015.81 5014.22 4755.18 99.39
5032.40 5030.78 4769.39 99.38
5045.77 5044.11 4780.61 99.37
5053.37 5051.66 4786.67 99.36
Table 2 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with k1 ¼ 4:2, k2 ¼ 4:18, and k3 ¼ 4 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
3161.39 3161.17 3145.22 98.64
3197.15 3196.93 3180.41 98.69
3231.83 3231.61 3214.45 98.73
3265.18 3264.94 3247.09 98.67
3296.94 3296.68 3278.08 98.62
3326.87 3326.56 3307.18 98.43
3354.66 3354.28 3334.05 98.16
3379.42 3378.97 3357.80 97.92
3397.53 3397.04 3374.99 97.83
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
by lowering the arrival rates to (k1 ¼ 4:2, k2 ¼ 4:18, k3 ¼ 4). Heavy traffic is demonstrated in Table 3 by raising the arrival rates to (k1 ¼ 16:2, k2 ¼ 16:18, k3 ¼ 16). We find that the algorithm produces excellent results in all three cases. In light traffic, the uncontrolled system generates approximately 99.3% of the possible revenue, while the near-optimal policy increases this to above 99.9%. The relative performance index decreases, but remains above 97.8%. In heavy traffic, the uncontrolled system generates approximately 83.8% of the possible revenue, while the near-optimal policy increases this to above 99.8%. The relative performance index remains above 99.1%. We conclude that the approximation is fairly insensitive to proportional changes in load at all stations, and works well in a wide range of loads. 4.1.5. Variation with line imbalance In this subsection, we vary the load on an individual station to create line imbalance, by varying the service rate of a single station. In Tables 4 and 5, we demonstrate the effect of increasing the service rate in the first station, l1 , first to 5.5 and then to 8.8. This results in a network in which the
175
first station is relatively fast compared to downstream stations. The near-optimal policy continues to achieve a performance index above 99.3%, increasing with l1 . This is to be expected. A higher service rate at station 1 results in less sensitivity to the occupancy of station 1 in the policy at station 3. Since the near-optimal policy at station 3 is not based on the occupancy of station 1, a higher service rate results in less error. A better test is to vary the service rate of the second station. In Tables 6 and 7, we increase l2 to 6.5 and then to 10. This results in a network in which the second station is relatively fast compared both to the upstream and the downstream station. We may worry that this may result in higher sensitivity to the occupancy of station 1 in the policy at station 3. As a result, the nearoptimal policy will fare worse, since the policy at station 3 is not based on the occupancy of station 1. Our numerical results, however, demonstrate that the performance degradation is small. The near-optimal policy continues to generate at least 99.9% of the achievable revenue with a performance index above 98.8%. We conclude that the algorithm’s performance is fairly insensitive to imbalance in the loads at different stations.
Table 3 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with k1 ¼ 16:2, k2 ¼ 16:18, and k3 ¼ 16 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
6158.20 6149.72 5170.05 99.14
6171.39 6162.91 5180.35 99.14
6183.56 6175.08 5189.60 99.15
6194.63 6186.14 5197.76 99.15
6204.38 6195.88 5204.73 99.15
6212.56 6204.05 5210.36 99.15
6218.84 6210.31 5214.42 99.15
6222.68 6214.13 5216.47 99.15
6223.03 6214.49 5215.73 99.15
Table 4 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l1 ¼ 5:5 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5002.86 5001.27 4729.85 99.42
5028.76 5027.16 4753.36 99.42
5052.59 5050.97 4774.67 99.42
5074.23 5072.58 4793.59 99.41
5093.69 5091.99 4810.06 99.40
5111.06 5109.32 4824.23 99.39
5126.54 5124.75 4836.27 99.38
5139.61 5137.78 4846.22 99.38
5148.53 5146.68 4852.88 99.37
176
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
Table 5 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l1 ¼ 8:8 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5023.54 5022.88 4741.63 99.77
5049.67 5049.01 4765.29 99.77
5073.36 5072.68 4786.37 99.76
5094.32 5093.58 4804.45 99.74
5112.44 5111.62 4819.40 99.72
5127.94 5127.10 4831.51 99.72
5140.44 5139.59 4841.25 99.72
5150.18 5149.32 4849.04 99.71
5156.72 5155.86 4854.40 99.72
Table 6 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l2 ¼ 6:5 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5662.27 5658.77 5325.93 98.96
5694.50 5691.07 5355.08 98.99
5725.94 5722.54 5383.53 99.01
5756.44 5753.03 5411.16 99.01
5785.75 5782.32 5437.79 99.01
5813.47 5810.02 5463.05 99.02
5838.82 5835.33 5486.22 99.01
5860.10 5856.59 5505.73 99.01
5873.33 5869.81 5517.82 99.01
Table 7 The expected discounted revenues of optimal, near-optimal and no-control policies for three-queue-in-series system with l2 ¼ 10 Control Optimal Near-optimal No-control Performance index (%)
State ð0; 4; 5Þ
ð1; 4; 5Þ
ð2; 4; 5Þ
ð3; 4; 5Þ
ð4; 4; 5Þ
ð5; 4; 5Þ
ð6; 4; 5Þ
ð7; 4; 5Þ
ð8; 4; 5Þ
5691.82 5688.07 5353.66 98.89
5723.89 5720.23 5382.81 98.93
5755.22 5751.60 5411.33 98.95
5785.68 5782.07 5439.13 98.96
5815.07 5811.45 5466.05 98.96
5842.99 5839.34 5491.74 98.96
5868.66 5864.99 5515.46 98.96
5890.36 5886.68 5535.61 98.96
5903.98 5900.30 5548.24 98.97
4.2. Six queues in series We now consider a network consisting of six tandem queues to investigate the dependence of the proposed method’s accuracy with the size of the network. We expect that the accuracy should degrade as the number of stations increases, since
the method is based on solving for the admission policy in a set of tandem queues. In Table 8, we present results using the following parameters: m1 ¼ 5, m2 ¼ 5, m3 ¼ 5, m4 ¼ 5, m5 ¼ 5, m6 ¼ 5, a ¼ 0:9999, Q ¼ 2000, k1 ¼ 5, k2 ¼ 5, k3 ¼ 5, k4 ¼ 5, k5 ¼ 5, k6 ¼ 5, l1 ¼ 6, l2 ¼ 5:5, l3 ¼ 5, l4 ¼ 4:5, l5 ¼ 4, l6 ¼ 3:5, P1 ¼ 0:2,
Table 8 The expected discounted revenues of optimal, near-optimal and no-control policies for six-queue-in-series system Control Optimal Near-optimal No-control Performance index (%)
State ð2; 3; 0; 1; 4; 2Þ
ð2; 3; 1; 1; 4; 2Þ
ð2; 3; 2; 1; 4; 2Þ
ð2; 3; 3; 1; 4; 2Þ
ð2; 3; 4; 1; 4; 2Þ
ð2; 3; 5; 1; 4; 2Þ
2869.767 2865.421 2772.975 95.51
2884.461 2880.006 2786.745 95.44
2897.215 2892.623 2798.582 95.34
2907.249 2902.483 2807.728 95.21
2913.160 2908.181 2812.832 95.04
2912.321 2907.230 2811.499 94.95
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
P2 ¼ 0:2, P3 ¼ 0:2, P4 ¼ 0:2, P5 ¼ 0:2, r1 ¼ 1, r2 ¼ 1, r3 ¼ 1, r4 ¼ 1, r5 ¼ 1, r6 ¼ 1, R2 ¼ 15, R3 ¼ 15, R4 ¼ 15, R5 ¼ 15 and R6 ¼ 15. We find that the accuracy does degrade with an increasing number of stations, but not much. The nearoptimal policy continues to generate at least 99.8% of the revenue of the optimal policy (compared to 96.6% for the uncontrolled system). The relative difference between the revenue generated by the near-optimal policy and by no control remains at least 94.9% of the difference between the optimal policy and no control. This minor decrease in accuracy, however, is achieved with dramatically less computation. The optimal policy, found using dynamic programming on the six-dimensional state space, requires several days of computation on a typical computer. In contrast, the near-optimal policy, found using the procedure proposed here, requires a few minutes of computation.
5. Conclusion We have considered access control policies in multiserver loss queues in series. Such models arise in computer and telecommunication networks, in which continued downstream service to the internal customers is preferable to admission of new customers. We are particularly motivated by telecommunication networks in which forced termination is considered to be an order of magnitude worse than blocking of new customers, and in which it is often the goal to achieve very low forced termination probabilities. A direct dynamic programming approach to determining the optimal admission policy at each station results in a computational complexity that is geometric in the number of stations. We proposed a simple recursive method to solve the problem using dynamic programming on a set of reduced state spaces. This approach includes a rate estimation technique for upstream stations, and a revenue estimation technique for downstream stations. Numerical results demonstrate that these nearoptimal policies are accurate under a wide range of loads. Increasing line imbalance and (or) in-
177
creasing the number of station in the network resulting in a minor degradation of the performance of the proposed algorithm. However, the decrease in computation can be many orders of magnitude.
References [1] E. Altman, S. Stidham, Optimality of monotonic policies for two-action Markovian decision processes, with applications to control of queues with delayed information, Queueing Systems 21 (1995) 267–291. [2] R.-R. Chen, S. Meyn, Value iteration and optimization of multiclass queueing networks, Queueing Systems 32 (1999) 65–97. [3] P.-J. Courtois, G. Scheys, Minimization of the total loss rate for two finite queues in series, IEEE Transactions on Communications 39 (1991) 1651–1661. [4] H. Ghoneim, S. Stidham, Optimal control of arrivals to two queues in series, European Journal of Operational Research 21 (1985) 399–409. [5] G. Koole, Structural results for the control of queueing systems using event-based dynamic programming, Queueing Systems 30 (1998) 323–339. [6] C.-Y. Ku, Access control for loss network, Ph.D. Dissertation, Department of EECS, Northwestern University, 1995. [7] C.-Y. Ku, S. Jordan, Access control to two multiserver loss queues in series, IEEE Transactions on Automatic Control 42 (1997) 1017–1023. [8] P.R. Kumar, P.P. Varaiya, Stochastic Systems, PrenticeHall, Englewood Cliffs, NJ, 1986. [9] J.M. Hah, P.L. Tien, M.C. Yuang, Neural-network-based call admission control in ATM networks with heterogeneous arrivals, Computer Communications 20 (1997) 732– 740. [10] G. Karlsson, Capacity reservation in ATM networks, Computer Communications 19 (1996) 180–193. [11] T.-H. Lee, K.-C. Lai, S.-T. Duann, Design of a real-time call admission controller for ATM networks, IEEE/ACM Transactions on Networking 4 (1996) 758–765. [12] P. Mohge, I. Rubin, Reserving for future clients in a multipoint application – why and how, IEEE Journal on Selected Areas in Communications 15 (1997) 531– 544. [13] J.S. Park, S.H. Lee, S.C. Kim, J.Y. Lee, S.B. Lee, A conferencing system for real-time, multiparty, multimedia services, IEEE Transactions on Consumer Electronics 44 (1998) 857–865. [14] S.M. Ross, Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983. [15] S. Stidham, R. Weber, A survey of Markov decision models for control of networks of queues, Queueing Systems 13 (1993) 291–314.
178
C.-Y. Ku, S. Jordan / European Journal of Operational Research 144 (2003) 166–178
[16] S. Stidham, Optimal control of admission to a queueing system, IEEE Transactions on Automatic Control 30 (1985) 705–713.
[17] R. Zhang, Y.A. Phillis, Fuzzy control of arrivals to tandem queues with two stations, IEEE Transactions on Fuzzy Systems 7 (1999) 361–367.