- Email: [email protected]
An Approach to the Modelling and Control of Feedback Queuing Systems
COP\ I'IKht Bud ;t pt'~ 1.
©
I F.·\( : ~' lIl '1 rit' 11111 ;d \"01"ld I-Iung.,n. 1 ~ IH4
( :nn).{r
n'
AN APPROACH TO THE MODELLING AND CONTROL OF FEEDBACK QUEUING SYSTEMS B. Bengtsson f)i ,'/ll/JlI 0/ .-\ lItu1I/tlll( COl/frill. /) 1,/)(1/"/"/1' 111 0/ F/f'f tmrl/ 1:" "gilll'l'I"lIlg. i _lIIkii/)/I/,1.! l "m" I'n 't.'" .
.\ . i8/83 1.I1IJUj/)/lIg . .)" ,rn/1'I1
Abstract. Practical queueing problems are very often of feedback type, i.e . c ust ome rs r epeated l y visit th e queue before l eavi ng the system. This feedback r e sults in dynam i cal characterist i cs , whi ch are esse n tial t o acco unt for when desi gnin g a con trol s trat egy . A simple app r oach t o the treatme nt of s uch problems is sugges t ed in this pape r . Ass umin g indep ende nce betwe e n th e individu a l de l ays in th e feedback lo op , a lin ear mode l of the fee db ack (stochast i c or de t ermi n istic) fol l ows f r om the usual statistical l i mit t heo r ems. The resulting model is we ll suit ed for direc t applica t io n of s t a ndard co ntrol -th eo retic i deas. This is i l lustrated wi t h a s uccessf ul appli ca tion t o th e load regulation of a t e l e phon e sta ti on. Keywords . Markov proc e ss ; ope rati ons r esearc h ; optimal control ; queueing t heory; stochas ti c co ntr o l . INTRODUCTION In computer and communication networks, it is important to be able to control certain queueing situations appropriately. Examples of such control situations are the routing problem in computer networks, and the problem of resource allocation in multi-user systems. A further example is the load regulation in telephone stations. The latter problem is treated as a case study in section 5, and it is the basic source of inspiration for the work presented here.
VJ
l [)~/0!l
A common feature to many practical queueing control problems is "customer feedback " , Le. the completion of a task in itself produces more tasks to be processed later on. In a telephone exchange e.g., the acceptance of a call implies that further work has subsequently to be done in analyzing the number that will be dialled, and in making the connection. Such a feedback gives rise to a dynamic behaviour of the system. The problem is hence one of controlling a (stochastic) dynamical system, although this is usually not the approach taken.
Fig.1 The queueing situation. All customers (messages, jobs et c) are identical, and each one is served twice. After his first passage through the queue, a customer reenters the queue after a stochastic delay ~ ; and after the second service completion he leaves the system. The "external" arrival rate is A customers/second, which can be controlled, whereas the distribution of service times (~/second) and delays are fixed. (In a telephone exchange, the variable arrival rate is due to the possibility of refusing service to some calls in an overload situation). All time delays are independent and identically distributed, and so are the service times. The control problem will eventually be to balance the tradeoff between a high arrival rate -which is desirable- and large values of the queue length.
In this paper we attempt to capture the dynamical aspects of the feedback loop, by means of a rather crude diffusion-type model. In section 2, the feedback queueing situation is introduced. In section 3, a linear model is developed from the basic limit theorems of statistics, and aspects on the control is discussed in section 4. The application of these ideas to a practical problem is examplified in section 5, where a comparison is also made with the present control strategy. 2
The matter of interest here will be to find a dynamic model for the feedback loop, i.e. for the "delay box " in Fig. 1. As stated earlier, the delays for different customers are assumed to be independent, identically distributed, random variables. In particular, the distribution does not depend on how many customers are presently in the delay line.
THE QUEUEING SITUATION
We shall be dealing with the queueing situation depicted in Fig. 1. Following Kleinrock (1975, p 159) we call it a feedback queue.
Denote by D(·) the probability distribution function of the delay, i.e.
We introduce the follOwing assumptions, which are an idealization e.g. of the application in section 5. 1223
1224
pI . .
B. Bengtsson (2.1)
2. t} - D(t)
(3.4 )
var wi - di(l-d i ) " Xi •
where ... is the delay as in Fig. 1.
Note in particular that d -1. n
Our primary goal is to describe the combined effect of a large number of customers, each experiencing a delay according to (2.1).
The difference equation for the state is now immediate. Denote by u(t) the number of customers arriving to the delay during the interval (t,t+1], and let vet) be the number leaving the delay during the same interval. Then we have the following equations:
3
A DIFFUSION-TYPE MODEL OF THE FEEDBACK LOOP
Suppose first that the delay is exponentially distrib.ted. Then the feedback loop is Markovian, snd its output v in Fig. 1. is (generalized) Poisson with a parameter which is proportional to the number of customers present in the delay. The state of the feedback loop can further be represented by a single (observable) number, and its output process has simple properties. If the delay is not exponential, however, we must usually include in the state the "age" of each customer, i.e. how long time he has spent in the delay so far. We shall assume later on that the number of customers present in the delay is very large, which means that it becomes unrealistic to keep track of all these ages, and to process that information. In addition, since the customers are indistinguishable, we cannot even in principle determine the state from u and v in Fig. 1: We do not know the age of the customers returning from the delay. The first step in the modelling approach proposed here, is to consider the situation in discrete time. Conceptually, we consider this as a sampling of the continuous-time processes, which gives rise to the natural terms sampling instant and sampling interval. We take the unit of time to be the sampling interval. We also assume that the variations of the delay distribution D(o) are small within a sampling interval, so that all customers arriving to the delay within an interval can be considered as equi va1en t. Now consider a customer of age i. By this we mean that he has so far spent i sampling intervals in the delay. Given that he is still in the delay, the conditional probabilit~ that he will leave it during the interval (i,i+1J, is clearly d
i
= D(i+1~-D(i)
(3.1)
l-D 1)
We further assume that the delay is strictly positive and bounded, i.e. D(O) - 0, D(n+1)
s
1,
(3.2)
where n is the smallest value of k such that D(k+1)-1. Hence no customers pass instantly through the delay. If necessary, we can achieve a bounded delay by truncating the distribution D( ").
The advantage of these assumptions is that we now have a fixed complexity of the state space. The age of a customer in the delay is sn integer k, 12.k2.n, so that the state is now specified by the number of customers of each age. Now introduce the state vector x(t) E Rn. Its i:th component, xi(t), is the number of customers of age i in the delay at sampling instant t. Independently of the others, each one of these customers will leave the delay before time t+1 with probabity d i according to (3.1). Hence, given Xi' the number of customers of age i leaving the delay during the next interval is a random variable w i with a binomial distribution and (3.3)
x(t+1) - A(x(t)-w(t»
+ B u(t)
(3.5)
T vet) - 1 w(t).
(3.6)
Here,
A-
(
0
:) ,- (j)
0
(3.7)
and IT is a row vector with only unit elements. wet) is a vector, whose i:th component is the number of customers of age i leaving the delay during the interval (t,t+1]. It has a binomial distribution given by (3.3), (3.4) as discussed above. The difference (x(t)-w(t» in (3.5) is hence the number of remaining customers of each age, and the matrix A shifts these ages by one step. Eq:s (3.5). (3.6) thus state that customers in the delay line first have age 1. and then gradually grow older until they leave. on which occasion they show up in the output v. Note that the state vector x(t) cannot be observed directly. Next. we shall take into account what is known about the vector wet). By (3.3) we have that. given x(t), E wet) - D
0
x(t).
(3.8)
where D-diag(d 1 • d 2 ••••• d ). Hence we can rewrite (3.5). (3.6) as: n x(t+1) - A(I-D)x(t)+Bu(t)-Av(t)
(3.9a) (3.9b)
where v(t)-w(t)-Ew(t) is a stochastic vector which depends only on the state x(t) at time t. and whose components are independent. Further, its i:th component has zero mean and variance d i (l-d i )"X according to (3.4). i We conclude the modelling of the delay with a
final approximation. Suppose that the "traffic intensity" is very high. so that all components of x have large values. In the limit. as x +=, the i binomially distributed random variable w i is asymptotically Normal by the central li;!t theorem (or. equivalently. since the binomial distribution tends to the Normal one for large samples). Hence. for high traffic intensities. we can regard wet) and thereby vet) - as being Gaussian. The variance of each component is given by (3.4). and it follows that we can equivalently represent vi(t) as (3.10)
where ei(t) - N(O,l) is a Gaussian white-noise sequence. The corresponding representation of the vector vet) can be written as vet) - [D(I_D)]1/2
Q y(x(t»
0
[x(t)]1/2
" e(t).
" e(t) (3.11)
where the square roots are taken by components.
Modellin g and Control of Feedback Queuing Systems Note that x is nonnegative, and that O~di~l for all i. Note also that vi has standard deviation Yi , which is proportional to Ix i • Hence the relative importance of v is small if x is large, and in the limit (3.9) becomes a deterministic model of the delay.
is then as follows: (i)
(3.l2a)
v(t) = lTDx(t) + lTy(x(t»e(t),
(3.l2b)
4
ASPECTS ON THE CONTROL PROBLEM
At this point we have an approximate model for the delay in Fig. 1. The next question is how to control the whole system. As stated in section 2, the goal will be to keep the arrival rate, A, as large as possible without getting unacceptable queue lengths. The optimal control depends, of course, on the exact formulation of the objective, and will in general be very hard to find. A few issues will however be of obvious interest. We discuss these briefly below. First, it would clearly be an advantage if we could observe the state vector x(t) in (3.12) directly. In the Linear-Quadratic Gaussian case, the problem consists of two separate sub-problems, namely filtering and control. Analogously, it appears natural to consider the filtering problem separately in the present case, i.e. to impose a certainty-equivalence assumption; acting as if the separation theorem were true in this case also. In the LQG case, the filtering problem is solved with the Kalman filter, as specified e.g. in Astr~m (1970, Theorem 4.1). Now, our state description (3.9) fits into the framework of the Kalman filter, except that the disturbance v(t) depends on the state x(t). This can be seen as state-dependent covariance matrices for the noise. A natural ad hoc solution in the present situation is thus to run the Kalman filter as usual, but using as noise covariance matrices what is given by (3.11) with x replaced by its current estimate. This idea is very similar to the Extended Kalman Filter (e.g. Jazwinski, 1970, Theorem 8.1 and Ljung, 1979).
As to the control problem, we need a model for the entire system. It is trivial to write down the dynamics of the queue in Fig. 1. The result is a first-order system, which is linear except at queue length O. The delay, in turn, has the model (3.9) or (3.12), which yields a linear input-output relation for the first moments. (Note, however, that this description is in discrete time, and in heavy traffic.) One very simple way of constructing a controller
~,
(4.1)
where z(t) is the Queue length at time t, A(t) and v(t) are the munber of "external " and "delayed" arrivals during (t, t+ll, respectively, and ~ is the server capacity. (il)
where A and B are given by (3.7), D is diagonal with d given by (3.1) and y(x(t» is defined in i (3.11). e(t) is a sequence of independent Gaussian vectors with all components independently N(O,l)distributed. We have arrived at a model whose dynamics depend on the delay distribution {d ) of (3.1). It looks i very much like the model in the LQG case (e.g. Astr~m, 1970, ch. 8). The difference is that the noise in (3.12) is state-dependent according to y(x(t»oe(t). Nevertheless, conditioned upon the present state, we have modelled the stochastic properties as Normal, and we have modelled the state as taking on a continuum of values. This is in the spirit of diffusion approximations, as treated e.g. in Kleinrock (1975, sec. 2.8).
Disregard the nonlinearity of the queue at queue length 0 (under heavy traffic, an empty queue will be a rare event), and model it as a first-order linear system: z(t+l) - z(t) + A(t) + v(t) -
Under heavy-traffic conditions, the delay box of Fig. 1 can thus be approximated by the stochastic system x(t+l)=A(I-D)x(t)+Bu(t)-Ay(x(t»e(t)
1225
Determine a "set-point" for the entire system (i.e. desired values of all the variables including z, A in (4.1) and x, u, v in (3.12», evaluate the noise variances (3.4) at that point, and use Linear-Quadratic Gaussian theory to design a certainty-equivalence controller.
(iii) Perform state estimation according to the Extended Kalman filter as above, and use the state estimates in the control law. The above controller foresees the effects of the dynamics of the delay and determines how many customers should be accepted during the next interval. Because of the heavy-traffic assumption, a large number of customers will however arrive to the queue between sampling instsnts. Hence, the model (3.9) gives little information about the fluctuations of the queue length. Consequently a "fast controller", working in continuous time on the actual queue length, may conceivably be a good way of implementing the acceptance determined by the above controller. This two-time-scale idea will not be pursued here. 5 AN APPLICATION: LOAD REGULATION IN A TELEPHONE STATION In this section we will examplify the ideas of the preceding sections as applied to a practical problem. The results were obtained in an M.S . thesis by Almhem and Claesson (1982); for further details reference is made to this thesis, and to Bengtsson (1982) • 5.1
Description of Problem
The system at hand is LM Ericsson' s "AXE" telephone station, which gives service to approximately 50000 subscribers. The heart of the stati.on is a Central Processor, CP, which initiates, coordinates and supervises all the numerous activities that are necessary for the performance of the station.
As usual in computer applications, the CP works on jobs. These jobs queue up for CP service in the job buffer, which is thus the queue under consideration here. It should be noted that this description is very much si.mplified: In the true system, there are e.g. four different job buffers, corresponding to different priority levels. It is important that jobs do not have to wait too long in the job buffer. Acceptable service delays are part of the specifications of the equipment, and it is clear that a subscriber trying to place a call may hang up if he considers the delay unacceptable (this is particularly true for automatic calls generated by other stations). As shall presently be outlined, each call generates many different jobs, and if the caller hangs up, those jobs already completed will be lost. Hence as delays increase in an overload situation, an increasing proportion of the work performed by the CP is subsequently lost.
B. Bengtss on
1226
In order to maintain the functions of the system, the load thus has to be limited. The only available control variable is the acceptance of arriving calls: Some calls may have to be blocked out, i.e . not given a dialling tone. Once a call has been accepted, it must be given service according to the following : Each call generates in the order of magnitude of 60 jobs in the CP. Very roughly, these can be divided into three groups: (i)
Before the caller hears the dialling tone, some tests and other functions must be performed. This accounts for about 15 jobs.
(ii)
After the caller has dialled the number, the lines have to be tested and the connection established. Before the recipient's telephone sounds, another 30 jobs have to be completed.
(iii) When the conversation is over, it takes about 15 more jobs to restore the lines to their original status, etc. The time it takes to dial a number is about 5-10 seconds. If we accept a call, we thus see from (i), (ii) above that we have to perform some 15 jobs almost immediately, and then another 30 jobs in about 10 seconds. Further away in the future, an additional 15 jobs will have to be done. Suppose now that we have a sudden increase in the arrival rate of new calls. Then if we accept as many as we can, there will be an "explosion" in the demand for processor capacity after 10 seconds, and the performance will be unacceptable.
continue as before. Otherwise step up or down the acceptance rate by unity. The dynamics of the system is thus accounted for by a cautious control: Increases of the acceptance rate are small (the maximum rate is about 70 calls/second), and they are rare enough that the effect of each one can be separately observed. Since the CP is active on several priority levels, its capacity for call handling is not constant. This is one reason why it is not possible to assign a fixed value to the desired acceptance rate. Another reason is that under normal load, short "bursts" of arriving calls will occur. These should of course not be delayed. On the whole, this load regulation works very well. In a normal situation, it accepts approximately as many calls as is possible, which is the objective of the control. Simulations indicate, however, that its transient properties are not optimal, which seems natural with regard to its cautious behaviour. In section 5.5 we shall compare its behaviour to a simple linear controller. 5.3
A Linear Dynamical Model
We shall now use the ideas of section 3 to obtain a linear dynamical model of the system: The demand for processor capacity generated by each call was summarized in (i)-(iii) of section 5.1. This demand can be visualized according to Fig . 2. O~mCl/7d
rar
'proc~~~or
Basically, the problem of control can thus be seen as a prediction problem: If we accept a new call now, then will the situation be acceptable 10 seconds from now? We must clearly take the dynamics of the system into account.
CCl,POCliy
~
r-
hICl61t~;,
5.2
The present load regulation is called LOAS (originally "LOAd Supervision") and is described in Karlstedt- Wildling (1979). It is based on the steady-state behaviour of the job buffer. Suppose first that a fixed number of calls per second are accepted. Clearly the expected queue length in the job buffer - and thereby the waiting times increases as the rate of accepted calls increases. Hence, given what waiting times are acceptable, it is possible to determine what fixed number of calls per second can be accepted. We call this the acceptance rate. Now, a quantity which is closely related to the waiting times, is the CP load, i.e. the proportion of time that the CP works. The CP load is easy to measure, and hence the acceptance rate can easily be checked against the CP load. If the acceptance rate is changed, it takes about 10 seconds before the full effect of this begins to show up in the CP load: cf (i)-(ii) in section 5.1. The philosophy of the present control can be summarized as follows (it should be noted that several other rules are active for security reasons): (1)
(ii)
£)IO///"'j c tJ/7/7~ C lio/7
Present Control Strategy
Keep the number of accepted calls per second constant during 10 seconds, so that the result can be seen in the CP load. After each 10-second-interval, check the CP load against a predetermined band .
(iii) If the CP load is within the band, then
'\.Olo/lm9
\
tone
(tJ/7v~r~oli'o/7
Ii
Oi~cor7/7~ct
ACCI!''pICl/7C~
Fig . 2
Demand for processor capacity generated by one call.
The time it takes to dial and to complete the conversation, respectively, can be viewed as two random variables, whose distributions are known. Hence we can calculate the probability density function for the processor demand of a call. Its general shape is as in Fig. 3. This is thus the demand of an "average" call.
O~mo/7(1
IOr
'prOC~~.5or
CCl,PClCl ly
L -______________________-=__ 11i77~
Fig. 3
Average demand for processor capacity.
Now, the differences between the present set-up and what we had in section 3 are that first of all each customer now returns twice to the queue (Fig. 2), and secondly his "size" changes as he is recycled. (We identify each group of jobs with the
1227
Mod e lli ng and Co n trol of Fe edba ck Que ui ng Sys tems service of a customer). It is clear that the same modelling approach as in section 3 can be applied to the present problem, however, and since the delay distributions are known, we can again obtain a linear stochastic model with state-dependent approximately Gaussian, noise. As to the first moments of this system, the dynamics of the feedback loop are given by the "step response " of Fig. 3. Given the distributions of the delays, it is easy to write down explicitly a discrete-time model of suitable order, and with a suitable sampling interval. Note that under heavytraffic conditions we can interpret the firstmoments (deterministic) model as resulting from the Law of Large Numbers just as the Gaussian disturbances in the model of section 3 stemmed from the Central Limit Theorem.
The model is the following : As before, we denote the input to the feedback loop by u and its output by v. Then we have the following first-moments model for the delay x(t+l) = Ax(t) + Bu(t)
(5.la)
v(t) = Cx(t)
(5.lb)
where A and B are as in (3.7) and C is a row vector, whose elements depend on the delay distributions; see Almhem-Claesson (1982) for numerical values. Note that the interpretation of x is different in (5.1) compared to (3.5): In (3.5), x contains the number of customers of each age, and in (5.1) it consists of old values of u. 5.4
A Simple Linear Controller
Given the model 5.1 of the feedback dynamics, the discussion in section 4 suggests a control strategy according to the following :
(0
(i)
The set-point for the job buffer queue length z is z=lO, and for the acceptance rate" it is X=70 calls/second (the latter parameter is technical, and depends on the desired CP load and on the higher-priority work rate). The sampling interval is 0 . 5 seconds.
(ii)
The penalty function for the LQG formalism is as follows: With notations according to (i) we have the penalty function (5.2) The result will be called the LQ regulator.
Generally speaking, both regulators work well as long as the call rate is constant. In a normal situation, when the system is not overloaded, both regulators accept arriving calls virtually immediately. We remarked in section 5.2 that the call handling capacity will not be exactly constant, so that the probing strategy of LOAS will have an advantage in assuming essentially nothing about the system structure. It turns out, however, that the LQ regulator is robust enough that its performance is not noticeably affected by a varying capacity. The same remark holds for the dynamics of the feedback loop: Different types of calls have different demands for attention by the processor, which results in variations of the feedback dynamics. The situation of interest here is when a step in the arrival rate occurs. In Figures 4 and 5, such a step occurs at time t=50, when the arrival rate increases from 25 to 90 calls/second . Figures 4 and 5
Results after sudden increase in the arrival rate at time t=50.
Estimate the state of the feedback loop by simply simulating (5.1). This is the same as taking v=O in (3.9); cf the remark following (3.11).
(it)
section, with the following specifications.
Solid curves : LQ regulator Dashed curves: LOAS regulator Ito
Approximate the job buffer with a firstorder linear system as in (4.1), i.e. a discrete-time integrator.
(iii) Consider the estimates in (i) as accurate and use them in the linear control law resulting from an application of the LinearQuadratic Gaussian formalism to the total system. (This requires, of course, that a "set-point" for the entire system has been selected i.e. reasonable values of the job buffer queue length, the acceptance rate etc). In point (ii) above, the rate-of-change of the queue length is approximated by the difference between the total input ( ",, +V" in Fig. 1) and the processor capacity. This is a good approximation if the queue is rarely empty, which is the case when a good control is needed. At point (iii), the control law determines the number of new calls to accept during the next sampling interval . We shall consider the simplest way of effectuating this, namely to accept them equally spaced over the sampling interval. 5.5
51!C017ch
Fig. 4
Actual acceptance .
75 (', I I
60
I
I
I
I
I
95
Comparison of Performances
In this section we briefly present a simulated experiment, where the present control strategy (LOAS) was compared to the very simple one of the previous section. The model used in the simulations was a fairly accurate one, as opposed to (5.1), which was used to derive the control strategy. We shall consider the linear regulator of the previous
/5 O+---~---r~~~~~~£L3LT-------.
o Fig. 5
100
15tJ
Queue length in job buffer.
B. Bengtsson
1228
As the step occurs, both regulators initially accept all the arrivals as before. Since the LQ regulator accounts for the dynamics, it rapidly compensates for the predicted peak demand at t-60, and starts to decrease the acceptance rate. After some oscillations, it quickly stabilizes the acceptance rate. With the LOAS regulator, the following happens: The maximal acceptance rate is very high in order to allow fast fluctuations in the arrivals. Hence, all arrivals are accepted during the first 10 seconds. As a consequence, the demand for processor capacity becomes very large as these calls become ready to be connected. Hence the queue length in the job buffer grows far beyond the security limit. This, in turn, implies that virtually no new calls are accepted during the next 10 seconds. Following thiS, finally, the acceptance rate is stepped up from a safe value by one call per second in every 10 seconds. After a couple of minutes the appropriate rate has been found. The gain in the number of accepted customers when using the LQ controller instead of LOAS is about 10% during the first 100 seconds after the step (7000 and 6300 calls, respectively). Note that after the transient the queue lengths are larger with the LQ regulator: This obviously follows from the larger acceptance rate. 6
CONCLUSIONS
We have considered queueing control problems of feedback type; see Fig. 1. A simple approach to the modelling and control of such systems in heavy-traffic conditions is suggested. The crucial assumption is that the delay of each .. customer" in the feedback loop is independent of the others. Applications of the Central Limit Theorem or the Law of Large Numbers then result in linear dynamical models. In the former case, the model has state-dependent Gaussian noise, and in the latter case it is deterministic. The advantage of these models is that they capture the dynamical aspects of the system in a simple way, which is well suited for direct application of ideas from Control Theory . This is illustrated with the problem of load regulation in a telephone exchange: A very simple controller based on the suggested approach is seen to perform very well compared to the control presently in use . REFERENCES Almhem, P. and A. Claesson, (1982). Load Regulation in a Telephone Station. Department of Electrical Engineering, Link~ping University, Link~ping, Sweden. LiTH-ISY-EX-0338. (In Swedish). Astr~m,
K. J. (1970). Introduction to Stochastic Control Theory. Academic Press, New Yo r k.
Bengtsson, B. (1982) . On Some Control Problems for Queues. Link~ping Studies in Science and Technology, Dissertations Department of Electrical Engineering, Link~ping University Link~ping, Sweden. Jazwinski, A. H. (1970). Stochastic Processes and Filtering Theory. Academic Press. Karlstedt, T. and K. Wildling (1979). Call Handling and Load Regulation in an SPC System. Proceedings of the 9:th International Teletraffic Congress, Torremolinos, Spain.
Kleinrock, L. (1975). Queueing Systems. Vol Computer Applications. Wiley-Interscience (New York). Ljung, L. (1979). Asymptotic Behaviour of the Extended Kalman Filter as a Parameter Estimator for Linear Systems. IEEE Trans Automat Contr vol AC-24 , pp 36-50 .
11 :
©
I F.·\( : ~' lIl '1 rit' 11111 ;d \"01"ld I-Iung.,n. 1 ~ IH4
( :nn).{r
n'
AN APPROACH TO THE MODELLING AND CONTROL OF FEEDBACK QUEUING SYSTEMS B. Bengtsson f)i ,'/ll/JlI 0/ .-\ lItu1I/tlll( COl/frill. /) 1,/)(1/"/"/1' 111 0/ F/f'f tmrl/ 1:" "gilll'l'I"lIlg. i _lIIkii/)/I/,1.! l "m" I'n 't.'" .
.\ . i8/83 1.I1IJUj/)/lIg . .)" ,rn/1'I1
Abstract. Practical queueing problems are very often of feedback type, i.e . c ust ome rs r epeated l y visit th e queue before l eavi ng the system. This feedback r e sults in dynam i cal characterist i cs , whi ch are esse n tial t o acco unt for when desi gnin g a con trol s trat egy . A simple app r oach t o the treatme nt of s uch problems is sugges t ed in this pape r . Ass umin g indep ende nce betwe e n th e individu a l de l ays in th e feedback lo op , a lin ear mode l of the fee db ack (stochast i c or de t ermi n istic) fol l ows f r om the usual statistical l i mit t heo r ems. The resulting model is we ll suit ed for direc t applica t io n of s t a ndard co ntrol -th eo retic i deas. This is i l lustrated wi t h a s uccessf ul appli ca tion t o th e load regulation of a t e l e phon e sta ti on. Keywords . Markov proc e ss ; ope rati ons r esearc h ; optimal control ; queueing t heory; stochas ti c co ntr o l . INTRODUCTION In computer and communication networks, it is important to be able to control certain queueing situations appropriately. Examples of such control situations are the routing problem in computer networks, and the problem of resource allocation in multi-user systems. A further example is the load regulation in telephone stations. The latter problem is treated as a case study in section 5, and it is the basic source of inspiration for the work presented here.
VJ
l [)~/0!l
A common feature to many practical queueing control problems is "customer feedback " , Le. the completion of a task in itself produces more tasks to be processed later on. In a telephone exchange e.g., the acceptance of a call implies that further work has subsequently to be done in analyzing the number that will be dialled, and in making the connection. Such a feedback gives rise to a dynamic behaviour of the system. The problem is hence one of controlling a (stochastic) dynamical system, although this is usually not the approach taken.
Fig.1 The queueing situation. All customers (messages, jobs et c) are identical, and each one is served twice. After his first passage through the queue, a customer reenters the queue after a stochastic delay ~ ; and after the second service completion he leaves the system. The "external" arrival rate is A customers/second, which can be controlled, whereas the distribution of service times (~/second) and delays are fixed. (In a telephone exchange, the variable arrival rate is due to the possibility of refusing service to some calls in an overload situation). All time delays are independent and identically distributed, and so are the service times. The control problem will eventually be to balance the tradeoff between a high arrival rate -which is desirable- and large values of the queue length.
In this paper we attempt to capture the dynamical aspects of the feedback loop, by means of a rather crude diffusion-type model. In section 2, the feedback queueing situation is introduced. In section 3, a linear model is developed from the basic limit theorems of statistics, and aspects on the control is discussed in section 4. The application of these ideas to a practical problem is examplified in section 5, where a comparison is also made with the present control strategy. 2
The matter of interest here will be to find a dynamic model for the feedback loop, i.e. for the "delay box " in Fig. 1. As stated earlier, the delays for different customers are assumed to be independent, identically distributed, random variables. In particular, the distribution does not depend on how many customers are presently in the delay line.
THE QUEUEING SITUATION
We shall be dealing with the queueing situation depicted in Fig. 1. Following Kleinrock (1975, p 159) we call it a feedback queue.
Denote by D(·) the probability distribution function of the delay, i.e.
We introduce the follOwing assumptions, which are an idealization e.g. of the application in section 5. 1223
1224
pI . .
B. Bengtsson (2.1)
2. t} - D(t)
(3.4 )
var wi - di(l-d i ) " Xi •
where ... is the delay as in Fig. 1.
Note in particular that d -1. n
Our primary goal is to describe the combined effect of a large number of customers, each experiencing a delay according to (2.1).
The difference equation for the state is now immediate. Denote by u(t) the number of customers arriving to the delay during the interval (t,t+1], and let vet) be the number leaving the delay during the same interval. Then we have the following equations:
3
A DIFFUSION-TYPE MODEL OF THE FEEDBACK LOOP
Suppose first that the delay is exponentially distrib.ted. Then the feedback loop is Markovian, snd its output v in Fig. 1. is (generalized) Poisson with a parameter which is proportional to the number of customers present in the delay. The state of the feedback loop can further be represented by a single (observable) number, and its output process has simple properties. If the delay is not exponential, however, we must usually include in the state the "age" of each customer, i.e. how long time he has spent in the delay so far. We shall assume later on that the number of customers present in the delay is very large, which means that it becomes unrealistic to keep track of all these ages, and to process that information. In addition, since the customers are indistinguishable, we cannot even in principle determine the state from u and v in Fig. 1: We do not know the age of the customers returning from the delay. The first step in the modelling approach proposed here, is to consider the situation in discrete time. Conceptually, we consider this as a sampling of the continuous-time processes, which gives rise to the natural terms sampling instant and sampling interval. We take the unit of time to be the sampling interval. We also assume that the variations of the delay distribution D(o) are small within a sampling interval, so that all customers arriving to the delay within an interval can be considered as equi va1en t. Now consider a customer of age i. By this we mean that he has so far spent i sampling intervals in the delay. Given that he is still in the delay, the conditional probabilit~ that he will leave it during the interval (i,i+1J, is clearly d
i
= D(i+1~-D(i)
(3.1)
l-D 1)
We further assume that the delay is strictly positive and bounded, i.e. D(O) - 0, D(n+1)
s
1,
(3.2)
where n is the smallest value of k such that D(k+1)-1. Hence no customers pass instantly through the delay. If necessary, we can achieve a bounded delay by truncating the distribution D( ").
The advantage of these assumptions is that we now have a fixed complexity of the state space. The age of a customer in the delay is sn integer k, 12.k2.n, so that the state is now specified by the number of customers of each age. Now introduce the state vector x(t) E Rn. Its i:th component, xi(t), is the number of customers of age i in the delay at sampling instant t. Independently of the others, each one of these customers will leave the delay before time t+1 with probabity d i according to (3.1). Hence, given Xi' the number of customers of age i leaving the delay during the next interval is a random variable w i with a binomial distribution and (3.3)
x(t+1) - A(x(t)-w(t»
+ B u(t)
(3.5)
T vet) - 1 w(t).
(3.6)
Here,
A-
(
0
:) ,- (j)
0
(3.7)
and IT is a row vector with only unit elements. wet) is a vector, whose i:th component is the number of customers of age i leaving the delay during the interval (t,t+1]. It has a binomial distribution given by (3.3), (3.4) as discussed above. The difference (x(t)-w(t» in (3.5) is hence the number of remaining customers of each age, and the matrix A shifts these ages by one step. Eq:s (3.5). (3.6) thus state that customers in the delay line first have age 1. and then gradually grow older until they leave. on which occasion they show up in the output v. Note that the state vector x(t) cannot be observed directly. Next. we shall take into account what is known about the vector wet). By (3.3) we have that. given x(t), E wet) - D
0
x(t).
(3.8)
where D-diag(d 1 • d 2 ••••• d ). Hence we can rewrite (3.5). (3.6) as: n x(t+1) - A(I-D)x(t)+Bu(t)-Av(t)
(3.9a) (3.9b)
where v(t)-w(t)-Ew(t) is a stochastic vector which depends only on the state x(t) at time t. and whose components are independent. Further, its i:th component has zero mean and variance d i (l-d i )"X according to (3.4). i We conclude the modelling of the delay with a
final approximation. Suppose that the "traffic intensity" is very high. so that all components of x have large values. In the limit. as x +=, the i binomially distributed random variable w i is asymptotically Normal by the central li;!t theorem (or. equivalently. since the binomial distribution tends to the Normal one for large samples). Hence. for high traffic intensities. we can regard wet) and thereby vet) - as being Gaussian. The variance of each component is given by (3.4). and it follows that we can equivalently represent vi(t) as (3.10)
where ei(t) - N(O,l) is a Gaussian white-noise sequence. The corresponding representation of the vector vet) can be written as vet) - [D(I_D)]1/2
Q y(x(t»
0
[x(t)]1/2
" e(t).
" e(t) (3.11)
where the square roots are taken by components.
Modellin g and Control of Feedback Queuing Systems Note that x is nonnegative, and that O~di~l for all i. Note also that vi has standard deviation Yi , which is proportional to Ix i • Hence the relative importance of v is small if x is large, and in the limit (3.9) becomes a deterministic model of the delay.
is then as follows: (i)
(3.l2a)
v(t) = lTDx(t) + lTy(x(t»e(t),
(3.l2b)
4
ASPECTS ON THE CONTROL PROBLEM
At this point we have an approximate model for the delay in Fig. 1. The next question is how to control the whole system. As stated in section 2, the goal will be to keep the arrival rate, A, as large as possible without getting unacceptable queue lengths. The optimal control depends, of course, on the exact formulation of the objective, and will in general be very hard to find. A few issues will however be of obvious interest. We discuss these briefly below. First, it would clearly be an advantage if we could observe the state vector x(t) in (3.12) directly. In the Linear-Quadratic Gaussian case, the problem consists of two separate sub-problems, namely filtering and control. Analogously, it appears natural to consider the filtering problem separately in the present case, i.e. to impose a certainty-equivalence assumption; acting as if the separation theorem were true in this case also. In the LQG case, the filtering problem is solved with the Kalman filter, as specified e.g. in Astr~m (1970, Theorem 4.1). Now, our state description (3.9) fits into the framework of the Kalman filter, except that the disturbance v(t) depends on the state x(t). This can be seen as state-dependent covariance matrices for the noise. A natural ad hoc solution in the present situation is thus to run the Kalman filter as usual, but using as noise covariance matrices what is given by (3.11) with x replaced by its current estimate. This idea is very similar to the Extended Kalman Filter (e.g. Jazwinski, 1970, Theorem 8.1 and Ljung, 1979).
As to the control problem, we need a model for the entire system. It is trivial to write down the dynamics of the queue in Fig. 1. The result is a first-order system, which is linear except at queue length O. The delay, in turn, has the model (3.9) or (3.12), which yields a linear input-output relation for the first moments. (Note, however, that this description is in discrete time, and in heavy traffic.) One very simple way of constructing a controller
~,
(4.1)
where z(t) is the Queue length at time t, A(t) and v(t) are the munber of "external " and "delayed" arrivals during (t, t+ll, respectively, and ~ is the server capacity. (il)
where A and B are given by (3.7), D is diagonal with d given by (3.1) and y(x(t» is defined in i (3.11). e(t) is a sequence of independent Gaussian vectors with all components independently N(O,l)distributed. We have arrived at a model whose dynamics depend on the delay distribution {d ) of (3.1). It looks i very much like the model in the LQG case (e.g. Astr~m, 1970, ch. 8). The difference is that the noise in (3.12) is state-dependent according to y(x(t»oe(t). Nevertheless, conditioned upon the present state, we have modelled the stochastic properties as Normal, and we have modelled the state as taking on a continuum of values. This is in the spirit of diffusion approximations, as treated e.g. in Kleinrock (1975, sec. 2.8).
Disregard the nonlinearity of the queue at queue length 0 (under heavy traffic, an empty queue will be a rare event), and model it as a first-order linear system: z(t+l) - z(t) + A(t) + v(t) -
Under heavy-traffic conditions, the delay box of Fig. 1 can thus be approximated by the stochastic system x(t+l)=A(I-D)x(t)+Bu(t)-Ay(x(t»e(t)
1225
Determine a "set-point" for the entire system (i.e. desired values of all the variables including z, A in (4.1) and x, u, v in (3.12», evaluate the noise variances (3.4) at that point, and use Linear-Quadratic Gaussian theory to design a certainty-equivalence controller.
(iii) Perform state estimation according to the Extended Kalman filter as above, and use the state estimates in the control law. The above controller foresees the effects of the dynamics of the delay and determines how many customers should be accepted during the next interval. Because of the heavy-traffic assumption, a large number of customers will however arrive to the queue between sampling instsnts. Hence, the model (3.9) gives little information about the fluctuations of the queue length. Consequently a "fast controller", working in continuous time on the actual queue length, may conceivably be a good way of implementing the acceptance determined by the above controller. This two-time-scale idea will not be pursued here. 5 AN APPLICATION: LOAD REGULATION IN A TELEPHONE STATION In this section we will examplify the ideas of the preceding sections as applied to a practical problem. The results were obtained in an M.S . thesis by Almhem and Claesson (1982); for further details reference is made to this thesis, and to Bengtsson (1982) • 5.1
Description of Problem
The system at hand is LM Ericsson' s "AXE" telephone station, which gives service to approximately 50000 subscribers. The heart of the stati.on is a Central Processor, CP, which initiates, coordinates and supervises all the numerous activities that are necessary for the performance of the station.
As usual in computer applications, the CP works on jobs. These jobs queue up for CP service in the job buffer, which is thus the queue under consideration here. It should be noted that this description is very much si.mplified: In the true system, there are e.g. four different job buffers, corresponding to different priority levels. It is important that jobs do not have to wait too long in the job buffer. Acceptable service delays are part of the specifications of the equipment, and it is clear that a subscriber trying to place a call may hang up if he considers the delay unacceptable (this is particularly true for automatic calls generated by other stations). As shall presently be outlined, each call generates many different jobs, and if the caller hangs up, those jobs already completed will be lost. Hence as delays increase in an overload situation, an increasing proportion of the work performed by the CP is subsequently lost.
B. Bengtss on
1226
In order to maintain the functions of the system, the load thus has to be limited. The only available control variable is the acceptance of arriving calls: Some calls may have to be blocked out, i.e . not given a dialling tone. Once a call has been accepted, it must be given service according to the following : Each call generates in the order of magnitude of 60 jobs in the CP. Very roughly, these can be divided into three groups: (i)
Before the caller hears the dialling tone, some tests and other functions must be performed. This accounts for about 15 jobs.
(ii)
After the caller has dialled the number, the lines have to be tested and the connection established. Before the recipient's telephone sounds, another 30 jobs have to be completed.
(iii) When the conversation is over, it takes about 15 more jobs to restore the lines to their original status, etc. The time it takes to dial a number is about 5-10 seconds. If we accept a call, we thus see from (i), (ii) above that we have to perform some 15 jobs almost immediately, and then another 30 jobs in about 10 seconds. Further away in the future, an additional 15 jobs will have to be done. Suppose now that we have a sudden increase in the arrival rate of new calls. Then if we accept as many as we can, there will be an "explosion" in the demand for processor capacity after 10 seconds, and the performance will be unacceptable.
continue as before. Otherwise step up or down the acceptance rate by unity. The dynamics of the system is thus accounted for by a cautious control: Increases of the acceptance rate are small (the maximum rate is about 70 calls/second), and they are rare enough that the effect of each one can be separately observed. Since the CP is active on several priority levels, its capacity for call handling is not constant. This is one reason why it is not possible to assign a fixed value to the desired acceptance rate. Another reason is that under normal load, short "bursts" of arriving calls will occur. These should of course not be delayed. On the whole, this load regulation works very well. In a normal situation, it accepts approximately as many calls as is possible, which is the objective of the control. Simulations indicate, however, that its transient properties are not optimal, which seems natural with regard to its cautious behaviour. In section 5.5 we shall compare its behaviour to a simple linear controller. 5.3
A Linear Dynamical Model
We shall now use the ideas of section 3 to obtain a linear dynamical model of the system: The demand for processor capacity generated by each call was summarized in (i)-(iii) of section 5.1. This demand can be visualized according to Fig . 2. O~mCl/7d
rar
'proc~~~or
Basically, the problem of control can thus be seen as a prediction problem: If we accept a new call now, then will the situation be acceptable 10 seconds from now? We must clearly take the dynamics of the system into account.
CCl,POCliy
~
r-
hICl61t~;,
5.2
The present load regulation is called LOAS (originally "LOAd Supervision") and is described in Karlstedt- Wildling (1979). It is based on the steady-state behaviour of the job buffer. Suppose first that a fixed number of calls per second are accepted. Clearly the expected queue length in the job buffer - and thereby the waiting times increases as the rate of accepted calls increases. Hence, given what waiting times are acceptable, it is possible to determine what fixed number of calls per second can be accepted. We call this the acceptance rate. Now, a quantity which is closely related to the waiting times, is the CP load, i.e. the proportion of time that the CP works. The CP load is easy to measure, and hence the acceptance rate can easily be checked against the CP load. If the acceptance rate is changed, it takes about 10 seconds before the full effect of this begins to show up in the CP load: cf (i)-(ii) in section 5.1. The philosophy of the present control can be summarized as follows (it should be noted that several other rules are active for security reasons): (1)
(ii)
£)IO///"'j c tJ/7/7~ C lio/7
Present Control Strategy
Keep the number of accepted calls per second constant during 10 seconds, so that the result can be seen in the CP load. After each 10-second-interval, check the CP load against a predetermined band .
(iii) If the CP load is within the band, then
'\.Olo/lm9
\
tone
(tJ/7v~r~oli'o/7
Ii
Oi~cor7/7~ct
ACCI!''pICl/7C~
Fig . 2
Demand for processor capacity generated by one call.
The time it takes to dial and to complete the conversation, respectively, can be viewed as two random variables, whose distributions are known. Hence we can calculate the probability density function for the processor demand of a call. Its general shape is as in Fig. 3. This is thus the demand of an "average" call.
O~mo/7(1
IOr
'prOC~~.5or
CCl,PClCl ly
L -______________________-=__ 11i77~
Fig. 3
Average demand for processor capacity.
Now, the differences between the present set-up and what we had in section 3 are that first of all each customer now returns twice to the queue (Fig. 2), and secondly his "size" changes as he is recycled. (We identify each group of jobs with the
1227
Mod e lli ng and Co n trol of Fe edba ck Que ui ng Sys tems service of a customer). It is clear that the same modelling approach as in section 3 can be applied to the present problem, however, and since the delay distributions are known, we can again obtain a linear stochastic model with state-dependent approximately Gaussian, noise. As to the first moments of this system, the dynamics of the feedback loop are given by the "step response " of Fig. 3. Given the distributions of the delays, it is easy to write down explicitly a discrete-time model of suitable order, and with a suitable sampling interval. Note that under heavytraffic conditions we can interpret the firstmoments (deterministic) model as resulting from the Law of Large Numbers just as the Gaussian disturbances in the model of section 3 stemmed from the Central Limit Theorem.
The model is the following : As before, we denote the input to the feedback loop by u and its output by v. Then we have the following first-moments model for the delay x(t+l) = Ax(t) + Bu(t)
(5.la)
v(t) = Cx(t)
(5.lb)
where A and B are as in (3.7) and C is a row vector, whose elements depend on the delay distributions; see Almhem-Claesson (1982) for numerical values. Note that the interpretation of x is different in (5.1) compared to (3.5): In (3.5), x contains the number of customers of each age, and in (5.1) it consists of old values of u. 5.4
A Simple Linear Controller
Given the model 5.1 of the feedback dynamics, the discussion in section 4 suggests a control strategy according to the following :
(0
(i)
The set-point for the job buffer queue length z is z=lO, and for the acceptance rate" it is X=70 calls/second (the latter parameter is technical, and depends on the desired CP load and on the higher-priority work rate). The sampling interval is 0 . 5 seconds.
(ii)
The penalty function for the LQG formalism is as follows: With notations according to (i) we have the penalty function (5.2) The result will be called the LQ regulator.
Generally speaking, both regulators work well as long as the call rate is constant. In a normal situation, when the system is not overloaded, both regulators accept arriving calls virtually immediately. We remarked in section 5.2 that the call handling capacity will not be exactly constant, so that the probing strategy of LOAS will have an advantage in assuming essentially nothing about the system structure. It turns out, however, that the LQ regulator is robust enough that its performance is not noticeably affected by a varying capacity. The same remark holds for the dynamics of the feedback loop: Different types of calls have different demands for attention by the processor, which results in variations of the feedback dynamics. The situation of interest here is when a step in the arrival rate occurs. In Figures 4 and 5, such a step occurs at time t=50, when the arrival rate increases from 25 to 90 calls/second . Figures 4 and 5
Results after sudden increase in the arrival rate at time t=50.
Estimate the state of the feedback loop by simply simulating (5.1). This is the same as taking v=O in (3.9); cf the remark following (3.11).
(it)
section, with the following specifications.
Solid curves : LQ regulator Dashed curves: LOAS regulator Ito
Approximate the job buffer with a firstorder linear system as in (4.1), i.e. a discrete-time integrator.
(iii) Consider the estimates in (i) as accurate and use them in the linear control law resulting from an application of the LinearQuadratic Gaussian formalism to the total system. (This requires, of course, that a "set-point" for the entire system has been selected i.e. reasonable values of the job buffer queue length, the acceptance rate etc). In point (ii) above, the rate-of-change of the queue length is approximated by the difference between the total input ( ",, +V" in Fig. 1) and the processor capacity. This is a good approximation if the queue is rarely empty, which is the case when a good control is needed. At point (iii), the control law determines the number of new calls to accept during the next sampling interval . We shall consider the simplest way of effectuating this, namely to accept them equally spaced over the sampling interval. 5.5
51!C017ch
Fig. 4
Actual acceptance .
75 (', I I
60
I
I
I
I
I
95
Comparison of Performances
In this section we briefly present a simulated experiment, where the present control strategy (LOAS) was compared to the very simple one of the previous section. The model used in the simulations was a fairly accurate one, as opposed to (5.1), which was used to derive the control strategy. We shall consider the linear regulator of the previous
/5 O+---~---r~~~~~~£L3LT-------.
o Fig. 5
100
15tJ
Queue length in job buffer.
B. Bengtsson
1228
As the step occurs, both regulators initially accept all the arrivals as before. Since the LQ regulator accounts for the dynamics, it rapidly compensates for the predicted peak demand at t-60, and starts to decrease the acceptance rate. After some oscillations, it quickly stabilizes the acceptance rate. With the LOAS regulator, the following happens: The maximal acceptance rate is very high in order to allow fast fluctuations in the arrivals. Hence, all arrivals are accepted during the first 10 seconds. As a consequence, the demand for processor capacity becomes very large as these calls become ready to be connected. Hence the queue length in the job buffer grows far beyond the security limit. This, in turn, implies that virtually no new calls are accepted during the next 10 seconds. Following thiS, finally, the acceptance rate is stepped up from a safe value by one call per second in every 10 seconds. After a couple of minutes the appropriate rate has been found. The gain in the number of accepted customers when using the LQ controller instead of LOAS is about 10% during the first 100 seconds after the step (7000 and 6300 calls, respectively). Note that after the transient the queue lengths are larger with the LQ regulator: This obviously follows from the larger acceptance rate. 6
CONCLUSIONS
We have considered queueing control problems of feedback type; see Fig. 1. A simple approach to the modelling and control of such systems in heavy-traffic conditions is suggested. The crucial assumption is that the delay of each .. customer" in the feedback loop is independent of the others. Applications of the Central Limit Theorem or the Law of Large Numbers then result in linear dynamical models. In the former case, the model has state-dependent Gaussian noise, and in the latter case it is deterministic. The advantage of these models is that they capture the dynamical aspects of the system in a simple way, which is well suited for direct application of ideas from Control Theory . This is illustrated with the problem of load regulation in a telephone exchange: A very simple controller based on the suggested approach is seen to perform very well compared to the control presently in use . REFERENCES Almhem, P. and A. Claesson, (1982). Load Regulation in a Telephone Station. Department of Electrical Engineering, Link~ping University, Link~ping, Sweden. LiTH-ISY-EX-0338. (In Swedish). Astr~m,
K. J. (1970). Introduction to Stochastic Control Theory. Academic Press, New Yo r k.
Bengtsson, B. (1982) . On Some Control Problems for Queues. Link~ping Studies in Science and Technology, Dissertations Department of Electrical Engineering, Link~ping University Link~ping, Sweden. Jazwinski, A. H. (1970). Stochastic Processes and Filtering Theory. Academic Press. Karlstedt, T. and K. Wildling (1979). Call Handling and Load Regulation in an SPC System. Proceedings of the 9:th International Teletraffic Congress, Torremolinos, Spain.
Kleinrock, L. (1975). Queueing Systems. Vol Computer Applications. Wiley-Interscience (New York). Ljung, L. (1979). Asymptotic Behaviour of the Extended Kalman Filter as a Parameter Estimator for Linear Systems. IEEE Trans Automat Contr vol AC-24 , pp 36-50 .
11 :