- Email: [email protected]
Simulation study of switched circuit communication networks using learning automata routing
Mathematics and Computers in Simulation XXIV (1982) 281-287 North-Holland Publishing Company
281
SIMULATION STUDY OF SWITCHED CIRCUIT COMMUNICATION AUTOMATA ROUTING
NETWORKS USING LEARNING
M.S. CHRYSTALL and P. MARS School of Electronic
and Electrical Engineering,
Robert Gordon’s Institute of Technology,
AB9 1 FR, United Kingdom
Traffic Routing in switched circuit communication networks is considered using both alternate or fixed rule and learning automaton routing strategies. The simulation studies using a microprocessor based real-time simulator are shown to provide confumation that the automaton routing scheme performs at least as well as the optimal fixed rule. In addition, studies under failure mode conditions, including link, node and focussed overloads, conclusively demonstrate the superior performance of learning automata routing.
1. Introduction The routing algorithm of a switched circuit communication network is used to set up and establish new calls in the system. Conceptually this involves a multi-stage decision process based on the availability of network facilities and the behaviour of the calling population. It therefore is subject to operation in a non-stationary environment and as such requires the ability to modify the routing strategy to suit the offered network conditions. Conventional routing schemes, e.g. telephone network routing, adopt a fixed strategy which performs the same routing laws under all conditions. However, there is a growing trend towards routing schemes which modify their routing policy to accommodate changing network conditions [ 1,2,3]. Among the many approaches that have been suggested one which appears to have growing interest is concerned with schemes based on adaptive or learning concepts
[lOI. The reason for interest in adaptive routing schemes can be further elaborated as follows. Network hardware (and software) are prone to fault conditions caused by adverse operating conditions and other external influences. Network traffic behaviour is also subject to alteration with peak and off-peak periods and additionally, abnormal loads imposed by holidays, natural disasters and other unpredictable events. An adaptive routing algorithm should have the capa03784754/82/0000-0000/$02.75
0 1982 North-Holland
bility of identifying and reacting to these problems and hopefully maintaining an adequate service for the user. On the other hand, the optimally designed fixed rule although satisfactory under the designed conditions cannot cope with the abnormal and consequently the user suffers a degradation in service. The aim of this paper is to describe and demonstrate an adaptive routing scheme and also present the simulation technique used in the study. In the experimental work the performance of the adaptive routing scheme is compared with that of a conventional fixed scheme and through selective fault injection, the selforganising ability of the adaptive scheme is demonstrated.
2. Switched circuit communications
networks
Any switched circuit network consists of a collection of switching centres (or nodes) interconnected by links, a link containing a number of lines, each capable of providing a physical communication path. In a modem system these resources are used in a circuit switched fashion, i.e. lines are electronically switched to form a complete circuit between the call source and destination. This entire path is set up by a special signalling message which traverses the network, holding open lines through which it has passed. The set-up path is then allocated for communication, and only when the source releases the circuit will the
282
M.S. Chrystall, R Mars / Switched circuit communication
lines become available for use by other calls. The statistical properties of the traffic conditions are of particular interest for network modelling. In this study the behaviour of calls arriving at a source node is assumed to be a Poisson process with the average point to point arrival rate given by Xii calls/ min. The probability of K arrivals at node i, destination j in time T is given by P[Kij arrivals in T
sec.] = (AiiT)K e-“iiT K!
’
Also of interest is the mean call hold times or service time. This is assumed to be exponentially distributed with a mean value of l/p mins/call. Therefore, for messages of length r minutes the following probability density function is considered:
networks
Design of an alternative path scheme requires an off-line optimisation process such as dynamic programming and given the network topology and traffic behaviour an optimal routing rule can be constructed. The optimal rule operates to use the network facilities in the most efficient way. This leads to a greater throughput of calls and consequently a better utilisation of the network. The operation of this algorithm is simple to comprehend and considers a routing table or matrix at each node. Routing matrix at node I: Sequence 1s; 2nd
01 . 1 r.
rD2
3rd...
rD3
1
. . .
1
f(r) = peew. To complete the description of the network, the various routing laws by which calls move from node to node must be considered. As it is in this area where the study is founded the next section describes both the conventional and adaptive schemes.
3. Network routing schemes In this section three routing algorithms are presented, two of a fixed policy and one adaptive scheme. The alternative path scheme is representative of the conventional routing algorithm in a telephone network. Another fixed algorithm considers a probabilistic routing behaviour, the call route taking the nature of a random walk. Finally an adaptive scheme where the routing probabilities of the previous scheme are modified according to the outcome of the routed call is described. 3.1, Fixed rule alternative path scheme The fixed rule alternative path scheme [4] is an example of an algorithm with a rigid routing policy. Despite the inability to reconfigure to suit the network conditions this scheme has definite advantages, namely simplicity and optimality under the designed conditions.
Each element in row D of the matrix indicate all allowable next nodes for a call at node I destined for D. The decision process of the scheme involves attempting to connect the call to the first choice in the sequence rDl (the most direct path) and if unsuccessful sequentially tries all other alternative rD2, rD3, .. . (back bone routes). If all these fail the call is deemed blocked and ejected from the network. This technique employs an extremely logical and simple solution to the routing problem. It is however restricted by the lack of feedback and the wellordered behaviour of the decision mechanism. 3.2. Probabilistic or random routing scheme This scheme represents an attempt to improve on the inflexible decision policy of the previous scheme. Again a routing matrix at each node is required. Routing matrix at node I: Probability of choice
In addition to this information
a further matrix must
MS. Chrystall, P. Mars / Switched circuit communication networks
be utilised to store the probability tain next node: PDi = probability
of selecting a cer-
of selecting rDi as the next node.
The operating procedure of this scheme involves a random selection of the next node from an allowable set. If the first choice proves to be unsuccessful the remaining alternatives are again considered (after suitable renormalisation of the probabilities) until either the call is successfully routed or is rejected. This scheme offers an approach which makes fuller use of all possible paths between the source and destination. In the next technique an improvement of this basic scheme is introduced by feeding back the outcome of the call, culminating in a modification of the choice probabilities. 3.3. Learning automata routing scheme Central to the adaptive routing scheme described in this section lies the notion of a stochastic learning automaton [9]. Such a device operates to pick an optimal action from a set of allowable actions. This action is performed on a random environment which in turn produces a response, this being fed back to the automation. The automation can then update its internal state in accordance with the received response thus modifying the future output behaviour; Consider the automaton/environment configuration shown in Fig. 1. Mathematically the environment is represented by an input set a(n), an output set /3(n) and a set of penalty probabilities C(n) as shown in Fig. 2(b).
b Fig. 2. (a) Automaton; (b) Environment.
The output set a(n) is a simple binary response (O,l), reward or penalty, good or bad, and as such presents an ideal feedback arrangement for the routing application. For a study of the automaton consider the quintuple co(n), tin), p(n), Fco(& p(n)), G @(n)B As in the environment /I(n) and o(n) represent the input and output sets. (See Fig. 2(a).) Internally the automaton is described by p(n), the action probability vector, P(n) = (hr, -*.>P,) the transition function
or reinforcement
AUTOMATON
and the output
function
o(n) = G Hn)) In this application an automaton with a simple stochastic output function and the following reinforcement scheme is employed.
P&n
+ I)=(1
j#i
Penalty on action q Pitn + l)=(l
-b)PjCn),
Pj#i(n + l)=Pj(n)
0< a, b< 1.
(p(n) = 0):
-a)Pj(n),
pi(tl + 1) = 1 - CPj(n
L--p-/J&
scheme
p(fl+ 1) = F@(n), p(n))
Reward on action Cui
Fig. 1.
283
+
+ 1) ; (/3(n) = 1):
284
MS. Chtystall, P. Mars / Switched circuit communication networks
Through the learning parameters a, b this scheme can be configured to have different learning behaviours. The application of the above learning concepts to network routing is an extension of the random routing scheme. The random selection of a next node corresponds to the automaton choosing an action, the outcome of the call, success or failure, instigating a reorganisation of the schemes routing structure. Etablishing a path involves a chain of routing decisions as the call is passed from node to node. The feedback policy is implemented when a call reaches the destination or is blocked at some intermediate node. For a successful call all automata along the path are rewarded equally. In the failure case all automata involved are punished irrespective of initial good decisions. This scheme can be considered as a self-tuning probabilistic routing scheme, the routing strategy being continually revised according to the conditions met by the calls. In the steady state the routing probabilities will converge such as to provide the optimum performance. If, however, the network or traffic status changes the learning controllers gain experience of the new conditions through their own behaviour and modify their action probabilities accordingly.
4. Simulation techniques The complex topology and multiplicity of traffic sources prevents rigorous mathematical analysis of a communications network. The most realistic method of studying the behaviour and performance of the network under different routing schemes is therefore a computer simulation. This is also the most convenient technique for implementing the various fault and abnormal conditions which disrupt network operations. 4.1, Simulation package The simulator design used for this study employs a three program package implemented on a PDP 11/03 microcomputer and consists of: (1) A network specification and simulation data structure construction program; (2) The actual simulation program; (3) An analysis program used to calculate and dis-
play network parameters of interest (i.e. blocking probabilities) directly from a log file produced during the simulation phase. An investigation of a particular network commences with the construction of the simulation data structure by the first program of the package. This program is operated off-line, and during execution the user is prompted to input topological data for the network. During the simulation phase the simulation data structure is used as a basis for a node by node simulation of the specified network. Calls enter the network from a random call source (hardware implemented) and are allowed to futer through the data structure representing the network. When a particular call is either blocked or successfully completed, a block of data is written to a log file in such a manner as to indicate the individual history of the call. On completion of the simulation the log file contains a complete record of all calls using the network during the experiment. With the complete log fde stored on disk the user can utilise the analysis program to extract the relevant information from the experiment. This multi-program package offers significant improvements in simulation flexibility compared with the single program implementation. The use of a data structure technique also proves advantageous with the possibility of simulating networks of different topologys and sizes, without any changes in the software. 4.2. Simulation program In a circuit switched network, service is provided by setting up a complete path of connected lines from the source node, through intermediate nodes to the destination. This is achieved by the simultaneous operation of several switching centres receiving calls and if necessary, routing them to further sections of the network. Due to this simultaneous behaviour the system simulation is quite difficult to implement on a digital computer. In this simulation program the problems introduced by the parallel nature of the network are partially alleviated by the use of a real-time program consisting of three pseudo-parallel processes. These take the form of a main program process, real-time clock process and a call arrival process. As the program cannot be processing all nodes at the same time, the
MS. chrystall, P. Mars / Switched circuit communication networks
concept of a process queue is introduced. Calls arriving at a node are stored in this queue until the program reaches the node at which point the calls are removed and processed accordingly. In addition to the process queue the concept of a holding data structure must be introduced. This area of memory is used to store information related to calls currently holding in the network. This is very important as without this information nothing is known on the state of the network, i.e. how many calls are in prog ress, which lines are in use. The main process of the program is responsible for system set up and the switching behaviour of the network nodes. This involves sequentially moving from node to node removing calls from the process queues and allocating senders. If no senders are available, calls must be inserted in a nodal first in/first out stack, where they can wait for a sender. A sender is essentially part of the nodal switching hardware and is responsible for setting up calls on an outgoing link from the node. At any time during the execution of the main process, the real-time clock or call source can interrupt the network computer. When this occurs the appropriate service routine is selected and performed. Upon execution of the real-time clock service routine all network functions of time are processed. For instance, calls holding in the holding data structure are timed down and if found to be complete are removed from the network, freeing facilities for subsequent calls. Other conditions affected by the real-time clock are the senders and sender queues. When a call is timed out of a sender it is routed to the next node. As calls in the sender queues are timed out a check is made on the availability of senders. If senders are still not available the calls are blocked. Calls arrive in the network when the call generator interrupts the network processor. When this happens ,
I CALL GENERATOR SC!k85
SO”KP
destmolio" ,"kprr"pt
f
Fig. 3. Hardware configuration.
-I _
NETWORK MODEL DEClV03 P
285
the call source and destination numbers are read from the generator hardware and an entry is made in the process queue of the source node. Additionally space in the holding data structure is allocated for the new call. The hardware configuration for this simulator is shown in Fig. 3. Time scaling is possible through constrained variations of the clock rate. 4.3. Simulations and log data structures As stated previously, the simulation data structure is used to carry the network topology to the simulation program. This has the general form of a ring data structure, with nodal beads connected together with a continuous loop of pointers. Each bead contains information and a memory allocation to perform the function of the sender queue, senders and the process queue. Note that although the senders and sender queue are actual network hardware, the process queue is a functional requirement of the simulation program. Supplementary to the nodal data, information on the trunk capacities of the various links is stored in a further data block. As the simulation proceeds this data is under continual alteration and at an instant represents the availability of lines between various nodes in the network. The log data structure is charged with the task of providing a record of the network traffic carried during the simulation. As calls finish in the network (successful completion or blocked) the holding data structure representation of the particular call is written to a disk file. On completion of the simulation the log fde contains a complete record of all calls, and is ordered on the basis of call termination. 4.4. Call generator The call generator provides a source of random call interrupt for the network computer, each interrupt representing the birth of a new call. When the generator decides that a call has arrived, it presents two binary numbers to the network indicating the source and destination of the call. Such an arrangement permits the use of a single call source for the network, this having the advantage of a reduction in cost and complexity. A fundamental task in the call generator design was finding an efficient algorithm for the generation
286
MS. Chtystall,P. Mars /Switched circuit communication networks
of the Poisson distributed interrupt pulses. One successful method has been found in the following technique. From the call arrival rate hii (in calls/min), the probability that a call will arrive in 1 second is found (provided X,t %- 1). This probability is then used for comparison with a random number (derived from a pseudo random binary sequence [6] to decide if an interrupt pulse should be generated. This algorithm has proved to be adequate for the purpose and provided the maximum value of h is restricted no problems will be encountered. Operation of the entire call generator is as follows. After initilisation, the program waits for a pulse from the real-time clock. When this arrives the arrival probability for the first call source is read from a program mable call stack. This probability is then used in the method described above to reach an arrival decision. If the outcome dictates a new call the network processor interrupt line is set and a call source and destination is obtained from the stack. This process is maintained until all entries in the stack have been covered. After this the program waits for another real-time clock signal and the entire sequence repeats using the same data. This system has been implemented on an Intel SDK-85 microcomputer system using a small but versatile program. Subsequent modifications to the system allows for dynamic switches from one set of data to another thus enabling the simulation of non-stationary arrival statistics.
for analysing the behaviour of the network/routing scheme combinations is the network blocking probability. This is the ratio of calls unable to be serviced (destination not reached) to the total number of calls. Blocking probability
=
No. of blocked calls Total No. of calls
g-node hierarchical network This network, along with its associated call sources is shown in Fig. 4. The design is based on the logical paths available for communication between two users in different network regions of a conventional hierarchical structured telephone network [4]. 1: Normal network conditions In this experiment the network is allowed to operate under the engineered loads with no induced faults. From the results obtained, Fig. 5, it can be seen that the learning scheme converges to give comparable results with the fixed rule. It therefore can be stated that the learning scheme can match the performance of the optimal fmed rule under normal conditions. Experiment
Experiment 2: Fault on link L 78 By failing link L 78, i.e. LT8 = 0, it can be shown that the fixed rule breaks down for calls going from 1 to 8. Further analysis reveals that the optimal fmed rule for calls destined to node 8 can only access 30
5. Simulation results Previously the study of learning automata routing has concentrated on very simple networks [7,8] with a view to studying the behaviour of the individual controllers. To demonstrate the value of such schemes a set of results for an 8-node automata controlled network is presented. In line with the earlier work the performance of the adaptive scheme is compared with that of the optimal fmed rule under normal conditions. The results presented here also includes a group of experiments which have been carried out to model the network under fault conditions and it is under these circumstances where an adaptive scheme proves useful. In this study the performance criterion selected
.
c Fig. 4. 8 node hierarchical network.
M.S. Chqutall, P. Mars / Switched circuit communication networks
normalcandltiom Lmk L,falled Node 1 failed Ovwlca!rode7
Fig. 5. Results - blocking probability.
lines whereas the learning scheme can converge on 80. The results obtained (Fig. 5) reflect this problem, the fwed rule blocking probability showing a substantial reduction is network performance. In this case the adaptive scheme copes better by utilising sections of the network untried by the fmed rule and thus producing a lower blocking probability. Experiment 3: Fault on node 7 In this experiment node 7 is eliminated by removing links LIT, LJ7, L6, and LT8. Again the learning scheme makes better use of the available facilities and out performs the fmed rule. Experiment 4: Traffic overload at node 7 In order to highlight the problem of traffic overloads an additional call source is added at node 7 (h,s). The results tend to suggest that the addition of such sources effectively reduces the number of lines available for the main stream traffic and causes a reduction in the performance of the network. As in the previous conditions the adaptive routing scheme displays an ability to overcome this problem.
6. Conclusions The learning automata routing scheme offers a very practical and efficient technique for adaptive routing in a circuit switched network. In particular the results of the simulation studies show the scheme to be competitive to the optimal fixed rule under normal conditions, but will significantly improve performance under abnormal conditions. Computer simulation has been found to be a very useful tool in this application and through the various fault injections has permitted a study of the routing schemes under non-stationary conditions.
281
As a consequence of the promising results obtained it is proposed to continue the work with in-depth experiments on much larger networks. This will be possible with the development of a more advanced simulation package which will allow the simulation of networks containing upwards of 20 nodes. The present work will also be extended to an initial investigation into the use of stochastic learning automata routing schemes in packet switched networks.
Acknowledgment The authors wish to gratefully acknowledge the support of a UK Science Research Council grant and the benefit of numerous discussions with professor K.S. Narendra of the Department of Engineering and Applied Science, Yale University, U.S.A.
References [ 1 ] P. Baran, On distributed communications networks, IEEE Trans. Comm 12 (1964) l-9. [ 21 B.W. Boehm and R.L. Mobley, Adaptive routing techniques for distributed communications systems, IEEE Trans. Comm. Tech. (1969) 340-349. [ 31 R.M. Glorioso, Engineering Cybernetics (Prentice-Ha& Englewood cliffs, NJ, 1975). [4] M.T. IIBIs, Telecommunications Switching Principles (ABen and Unwin, London, 1979). [5] P. Mars and M.S. ChrystaB, Real-time telephone traffic simulation using learning automata routing, Technical Report S and SI 7909, Yale University (1979). [6] A.J. Miller and P. Mars, Theory and design of a digital stochastic computer random number generator, Trans. IMACS 19 (1977) 198-216. [ 71 K.S. Narendra, P. Mars and M.S. ChrystaII, Simulation study of telephone traffic routing using learning algorithms Part II, Technical Report S and SI 7907, Yale University (1979). [ 81 KS. Narendra and D.M. McKenna, Simulation study of telephone traffic routing using learning algorithms Part I, Technical Report S and SI 7806, Yale University, (1978). [ 91 K.S. Narendra and M.A.L. Thathachar, Learning automata - A survey, IEEE Trans. Systems, Man Cybemet. 4 (1974) 323-334. [lo] K.S. Narendra, E.A. Wright and L.C. Mason, Applications of learning automata to telephone traffic routing problems, IEEE Trans. Systems, Man Cybemet. 7 (1977) 785-792.
281
SIMULATION STUDY OF SWITCHED CIRCUIT COMMUNICATION AUTOMATA ROUTING
NETWORKS USING LEARNING
M.S. CHRYSTALL and P. MARS School of Electronic
and Electrical Engineering,
Robert Gordon’s Institute of Technology,
AB9 1 FR, United Kingdom
Traffic Routing in switched circuit communication networks is considered using both alternate or fixed rule and learning automaton routing strategies. The simulation studies using a microprocessor based real-time simulator are shown to provide confumation that the automaton routing scheme performs at least as well as the optimal fixed rule. In addition, studies under failure mode conditions, including link, node and focussed overloads, conclusively demonstrate the superior performance of learning automata routing.
1. Introduction The routing algorithm of a switched circuit communication network is used to set up and establish new calls in the system. Conceptually this involves a multi-stage decision process based on the availability of network facilities and the behaviour of the calling population. It therefore is subject to operation in a non-stationary environment and as such requires the ability to modify the routing strategy to suit the offered network conditions. Conventional routing schemes, e.g. telephone network routing, adopt a fixed strategy which performs the same routing laws under all conditions. However, there is a growing trend towards routing schemes which modify their routing policy to accommodate changing network conditions [ 1,2,3]. Among the many approaches that have been suggested one which appears to have growing interest is concerned with schemes based on adaptive or learning concepts
[lOI. The reason for interest in adaptive routing schemes can be further elaborated as follows. Network hardware (and software) are prone to fault conditions caused by adverse operating conditions and other external influences. Network traffic behaviour is also subject to alteration with peak and off-peak periods and additionally, abnormal loads imposed by holidays, natural disasters and other unpredictable events. An adaptive routing algorithm should have the capa03784754/82/0000-0000/$02.75
0 1982 North-Holland
bility of identifying and reacting to these problems and hopefully maintaining an adequate service for the user. On the other hand, the optimally designed fixed rule although satisfactory under the designed conditions cannot cope with the abnormal and consequently the user suffers a degradation in service. The aim of this paper is to describe and demonstrate an adaptive routing scheme and also present the simulation technique used in the study. In the experimental work the performance of the adaptive routing scheme is compared with that of a conventional fixed scheme and through selective fault injection, the selforganising ability of the adaptive scheme is demonstrated.
2. Switched circuit communications
networks
Any switched circuit network consists of a collection of switching centres (or nodes) interconnected by links, a link containing a number of lines, each capable of providing a physical communication path. In a modem system these resources are used in a circuit switched fashion, i.e. lines are electronically switched to form a complete circuit between the call source and destination. This entire path is set up by a special signalling message which traverses the network, holding open lines through which it has passed. The set-up path is then allocated for communication, and only when the source releases the circuit will the
282
M.S. Chrystall, R Mars / Switched circuit communication
lines become available for use by other calls. The statistical properties of the traffic conditions are of particular interest for network modelling. In this study the behaviour of calls arriving at a source node is assumed to be a Poisson process with the average point to point arrival rate given by Xii calls/ min. The probability of K arrivals at node i, destination j in time T is given by P[Kij arrivals in T
sec.] = (AiiT)K e-“iiT K!
’
Also of interest is the mean call hold times or service time. This is assumed to be exponentially distributed with a mean value of l/p mins/call. Therefore, for messages of length r minutes the following probability density function is considered:
networks
Design of an alternative path scheme requires an off-line optimisation process such as dynamic programming and given the network topology and traffic behaviour an optimal routing rule can be constructed. The optimal rule operates to use the network facilities in the most efficient way. This leads to a greater throughput of calls and consequently a better utilisation of the network. The operation of this algorithm is simple to comprehend and considers a routing table or matrix at each node. Routing matrix at node I: Sequence 1s; 2nd
01 . 1 r.
rD2
3rd...
rD3
1
. . .
1
f(r) = peew. To complete the description of the network, the various routing laws by which calls move from node to node must be considered. As it is in this area where the study is founded the next section describes both the conventional and adaptive schemes.
3. Network routing schemes In this section three routing algorithms are presented, two of a fixed policy and one adaptive scheme. The alternative path scheme is representative of the conventional routing algorithm in a telephone network. Another fixed algorithm considers a probabilistic routing behaviour, the call route taking the nature of a random walk. Finally an adaptive scheme where the routing probabilities of the previous scheme are modified according to the outcome of the routed call is described. 3.1, Fixed rule alternative path scheme The fixed rule alternative path scheme [4] is an example of an algorithm with a rigid routing policy. Despite the inability to reconfigure to suit the network conditions this scheme has definite advantages, namely simplicity and optimality under the designed conditions.
Each element in row D of the matrix indicate all allowable next nodes for a call at node I destined for D. The decision process of the scheme involves attempting to connect the call to the first choice in the sequence rDl (the most direct path) and if unsuccessful sequentially tries all other alternative rD2, rD3, .. . (back bone routes). If all these fail the call is deemed blocked and ejected from the network. This technique employs an extremely logical and simple solution to the routing problem. It is however restricted by the lack of feedback and the wellordered behaviour of the decision mechanism. 3.2. Probabilistic or random routing scheme This scheme represents an attempt to improve on the inflexible decision policy of the previous scheme. Again a routing matrix at each node is required. Routing matrix at node I: Probability of choice
In addition to this information
a further matrix must
MS. Chrystall, P. Mars / Switched circuit communication networks
be utilised to store the probability tain next node: PDi = probability
of selecting a cer-
of selecting rDi as the next node.
The operating procedure of this scheme involves a random selection of the next node from an allowable set. If the first choice proves to be unsuccessful the remaining alternatives are again considered (after suitable renormalisation of the probabilities) until either the call is successfully routed or is rejected. This scheme offers an approach which makes fuller use of all possible paths between the source and destination. In the next technique an improvement of this basic scheme is introduced by feeding back the outcome of the call, culminating in a modification of the choice probabilities. 3.3. Learning automata routing scheme Central to the adaptive routing scheme described in this section lies the notion of a stochastic learning automaton [9]. Such a device operates to pick an optimal action from a set of allowable actions. This action is performed on a random environment which in turn produces a response, this being fed back to the automation. The automation can then update its internal state in accordance with the received response thus modifying the future output behaviour; Consider the automaton/environment configuration shown in Fig. 1. Mathematically the environment is represented by an input set a(n), an output set /3(n) and a set of penalty probabilities C(n) as shown in Fig. 2(b).
b Fig. 2. (a) Automaton; (b) Environment.
The output set a(n) is a simple binary response (O,l), reward or penalty, good or bad, and as such presents an ideal feedback arrangement for the routing application. For a study of the automaton consider the quintuple co(n), tin), p(n), Fco(& p(n)), G @(n)B As in the environment /I(n) and o(n) represent the input and output sets. (See Fig. 2(a).) Internally the automaton is described by p(n), the action probability vector, P(n) = (hr, -*.>P,) the transition function
or reinforcement
AUTOMATON
and the output
function
o(n) = G Hn)) In this application an automaton with a simple stochastic output function and the following reinforcement scheme is employed.
P&n
+ I)=(1
j#i
Penalty on action q Pitn + l)=(l
-b)PjCn),
Pj#i(n + l)=Pj(n)
0< a, b< 1.
(p(n) = 0):
-a)Pj(n),
pi(tl + 1) = 1 - CPj(n
L--p-/J&
scheme
p(fl+ 1) = F@(n), p(n))
Reward on action Cui
Fig. 1.
283
+
+ 1) ; (/3(n) = 1):
284
MS. Chtystall, P. Mars / Switched circuit communication networks
Through the learning parameters a, b this scheme can be configured to have different learning behaviours. The application of the above learning concepts to network routing is an extension of the random routing scheme. The random selection of a next node corresponds to the automaton choosing an action, the outcome of the call, success or failure, instigating a reorganisation of the schemes routing structure. Etablishing a path involves a chain of routing decisions as the call is passed from node to node. The feedback policy is implemented when a call reaches the destination or is blocked at some intermediate node. For a successful call all automata along the path are rewarded equally. In the failure case all automata involved are punished irrespective of initial good decisions. This scheme can be considered as a self-tuning probabilistic routing scheme, the routing strategy being continually revised according to the conditions met by the calls. In the steady state the routing probabilities will converge such as to provide the optimum performance. If, however, the network or traffic status changes the learning controllers gain experience of the new conditions through their own behaviour and modify their action probabilities accordingly.
4. Simulation techniques The complex topology and multiplicity of traffic sources prevents rigorous mathematical analysis of a communications network. The most realistic method of studying the behaviour and performance of the network under different routing schemes is therefore a computer simulation. This is also the most convenient technique for implementing the various fault and abnormal conditions which disrupt network operations. 4.1, Simulation package The simulator design used for this study employs a three program package implemented on a PDP 11/03 microcomputer and consists of: (1) A network specification and simulation data structure construction program; (2) The actual simulation program; (3) An analysis program used to calculate and dis-
play network parameters of interest (i.e. blocking probabilities) directly from a log file produced during the simulation phase. An investigation of a particular network commences with the construction of the simulation data structure by the first program of the package. This program is operated off-line, and during execution the user is prompted to input topological data for the network. During the simulation phase the simulation data structure is used as a basis for a node by node simulation of the specified network. Calls enter the network from a random call source (hardware implemented) and are allowed to futer through the data structure representing the network. When a particular call is either blocked or successfully completed, a block of data is written to a log file in such a manner as to indicate the individual history of the call. On completion of the simulation the log file contains a complete record of all calls using the network during the experiment. With the complete log fde stored on disk the user can utilise the analysis program to extract the relevant information from the experiment. This multi-program package offers significant improvements in simulation flexibility compared with the single program implementation. The use of a data structure technique also proves advantageous with the possibility of simulating networks of different topologys and sizes, without any changes in the software. 4.2. Simulation program In a circuit switched network, service is provided by setting up a complete path of connected lines from the source node, through intermediate nodes to the destination. This is achieved by the simultaneous operation of several switching centres receiving calls and if necessary, routing them to further sections of the network. Due to this simultaneous behaviour the system simulation is quite difficult to implement on a digital computer. In this simulation program the problems introduced by the parallel nature of the network are partially alleviated by the use of a real-time program consisting of three pseudo-parallel processes. These take the form of a main program process, real-time clock process and a call arrival process. As the program cannot be processing all nodes at the same time, the
MS. chrystall, P. Mars / Switched circuit communication networks
concept of a process queue is introduced. Calls arriving at a node are stored in this queue until the program reaches the node at which point the calls are removed and processed accordingly. In addition to the process queue the concept of a holding data structure must be introduced. This area of memory is used to store information related to calls currently holding in the network. This is very important as without this information nothing is known on the state of the network, i.e. how many calls are in prog ress, which lines are in use. The main process of the program is responsible for system set up and the switching behaviour of the network nodes. This involves sequentially moving from node to node removing calls from the process queues and allocating senders. If no senders are available, calls must be inserted in a nodal first in/first out stack, where they can wait for a sender. A sender is essentially part of the nodal switching hardware and is responsible for setting up calls on an outgoing link from the node. At any time during the execution of the main process, the real-time clock or call source can interrupt the network computer. When this occurs the appropriate service routine is selected and performed. Upon execution of the real-time clock service routine all network functions of time are processed. For instance, calls holding in the holding data structure are timed down and if found to be complete are removed from the network, freeing facilities for subsequent calls. Other conditions affected by the real-time clock are the senders and sender queues. When a call is timed out of a sender it is routed to the next node. As calls in the sender queues are timed out a check is made on the availability of senders. If senders are still not available the calls are blocked. Calls arrive in the network when the call generator interrupts the network processor. When this happens ,
I CALL GENERATOR SC!k85
SO”KP
destmolio" ,"kprr"pt
f
Fig. 3. Hardware configuration.
-I _
NETWORK MODEL DEClV03 P
285
the call source and destination numbers are read from the generator hardware and an entry is made in the process queue of the source node. Additionally space in the holding data structure is allocated for the new call. The hardware configuration for this simulator is shown in Fig. 3. Time scaling is possible through constrained variations of the clock rate. 4.3. Simulations and log data structures As stated previously, the simulation data structure is used to carry the network topology to the simulation program. This has the general form of a ring data structure, with nodal beads connected together with a continuous loop of pointers. Each bead contains information and a memory allocation to perform the function of the sender queue, senders and the process queue. Note that although the senders and sender queue are actual network hardware, the process queue is a functional requirement of the simulation program. Supplementary to the nodal data, information on the trunk capacities of the various links is stored in a further data block. As the simulation proceeds this data is under continual alteration and at an instant represents the availability of lines between various nodes in the network. The log data structure is charged with the task of providing a record of the network traffic carried during the simulation. As calls finish in the network (successful completion or blocked) the holding data structure representation of the particular call is written to a disk file. On completion of the simulation the log fde contains a complete record of all calls, and is ordered on the basis of call termination. 4.4. Call generator The call generator provides a source of random call interrupt for the network computer, each interrupt representing the birth of a new call. When the generator decides that a call has arrived, it presents two binary numbers to the network indicating the source and destination of the call. Such an arrangement permits the use of a single call source for the network, this having the advantage of a reduction in cost and complexity. A fundamental task in the call generator design was finding an efficient algorithm for the generation
286
MS. Chtystall,P. Mars /Switched circuit communication networks
of the Poisson distributed interrupt pulses. One successful method has been found in the following technique. From the call arrival rate hii (in calls/min), the probability that a call will arrive in 1 second is found (provided X,t %- 1). This probability is then used for comparison with a random number (derived from a pseudo random binary sequence [6] to decide if an interrupt pulse should be generated. This algorithm has proved to be adequate for the purpose and provided the maximum value of h is restricted no problems will be encountered. Operation of the entire call generator is as follows. After initilisation, the program waits for a pulse from the real-time clock. When this arrives the arrival probability for the first call source is read from a program mable call stack. This probability is then used in the method described above to reach an arrival decision. If the outcome dictates a new call the network processor interrupt line is set and a call source and destination is obtained from the stack. This process is maintained until all entries in the stack have been covered. After this the program waits for another real-time clock signal and the entire sequence repeats using the same data. This system has been implemented on an Intel SDK-85 microcomputer system using a small but versatile program. Subsequent modifications to the system allows for dynamic switches from one set of data to another thus enabling the simulation of non-stationary arrival statistics.
for analysing the behaviour of the network/routing scheme combinations is the network blocking probability. This is the ratio of calls unable to be serviced (destination not reached) to the total number of calls. Blocking probability
=
No. of blocked calls Total No. of calls
g-node hierarchical network This network, along with its associated call sources is shown in Fig. 4. The design is based on the logical paths available for communication between two users in different network regions of a conventional hierarchical structured telephone network [4]. 1: Normal network conditions In this experiment the network is allowed to operate under the engineered loads with no induced faults. From the results obtained, Fig. 5, it can be seen that the learning scheme converges to give comparable results with the fixed rule. It therefore can be stated that the learning scheme can match the performance of the optimal fmed rule under normal conditions. Experiment
Experiment 2: Fault on link L 78 By failing link L 78, i.e. LT8 = 0, it can be shown that the fixed rule breaks down for calls going from 1 to 8. Further analysis reveals that the optimal fmed rule for calls destined to node 8 can only access 30
5. Simulation results Previously the study of learning automata routing has concentrated on very simple networks [7,8] with a view to studying the behaviour of the individual controllers. To demonstrate the value of such schemes a set of results for an 8-node automata controlled network is presented. In line with the earlier work the performance of the adaptive scheme is compared with that of the optimal fmed rule under normal conditions. The results presented here also includes a group of experiments which have been carried out to model the network under fault conditions and it is under these circumstances where an adaptive scheme proves useful. In this study the performance criterion selected
.
c Fig. 4. 8 node hierarchical network.
M.S. Chqutall, P. Mars / Switched circuit communication networks
normalcandltiom Lmk L,falled Node 1 failed Ovwlca!rode7
Fig. 5. Results - blocking probability.
lines whereas the learning scheme can converge on 80. The results obtained (Fig. 5) reflect this problem, the fwed rule blocking probability showing a substantial reduction is network performance. In this case the adaptive scheme copes better by utilising sections of the network untried by the fmed rule and thus producing a lower blocking probability. Experiment 3: Fault on node 7 In this experiment node 7 is eliminated by removing links LIT, LJ7, L6, and LT8. Again the learning scheme makes better use of the available facilities and out performs the fmed rule. Experiment 4: Traffic overload at node 7 In order to highlight the problem of traffic overloads an additional call source is added at node 7 (h,s). The results tend to suggest that the addition of such sources effectively reduces the number of lines available for the main stream traffic and causes a reduction in the performance of the network. As in the previous conditions the adaptive routing scheme displays an ability to overcome this problem.
6. Conclusions The learning automata routing scheme offers a very practical and efficient technique for adaptive routing in a circuit switched network. In particular the results of the simulation studies show the scheme to be competitive to the optimal fixed rule under normal conditions, but will significantly improve performance under abnormal conditions. Computer simulation has been found to be a very useful tool in this application and through the various fault injections has permitted a study of the routing schemes under non-stationary conditions.
281
As a consequence of the promising results obtained it is proposed to continue the work with in-depth experiments on much larger networks. This will be possible with the development of a more advanced simulation package which will allow the simulation of networks containing upwards of 20 nodes. The present work will also be extended to an initial investigation into the use of stochastic learning automata routing schemes in packet switched networks.
Acknowledgment The authors wish to gratefully acknowledge the support of a UK Science Research Council grant and the benefit of numerous discussions with professor K.S. Narendra of the Department of Engineering and Applied Science, Yale University, U.S.A.
References [ 1 ] P. Baran, On distributed communications networks, IEEE Trans. Comm 12 (1964) l-9. [ 21 B.W. Boehm and R.L. Mobley, Adaptive routing techniques for distributed communications systems, IEEE Trans. Comm. Tech. (1969) 340-349. [ 31 R.M. Glorioso, Engineering Cybernetics (Prentice-Ha& Englewood cliffs, NJ, 1975). [4] M.T. IIBIs, Telecommunications Switching Principles (ABen and Unwin, London, 1979). [5] P. Mars and M.S. ChrystaB, Real-time telephone traffic simulation using learning automata routing, Technical Report S and SI 7909, Yale University (1979). [6] A.J. Miller and P. Mars, Theory and design of a digital stochastic computer random number generator, Trans. IMACS 19 (1977) 198-216. [ 71 K.S. Narendra, P. Mars and M.S. ChrystaII, Simulation study of telephone traffic routing using learning algorithms Part II, Technical Report S and SI 7907, Yale University (1979). [ 81 KS. Narendra and D.M. McKenna, Simulation study of telephone traffic routing using learning algorithms Part I, Technical Report S and SI 7806, Yale University, (1978). [ 91 K.S. Narendra and M.A.L. Thathachar, Learning automata - A survey, IEEE Trans. Systems, Man Cybemet. 4 (1974) 323-334. [lo] K.S. Narendra, E.A. Wright and L.C. Mason, Applications of learning automata to telephone traffic routing problems, IEEE Trans. Systems, Man Cybemet. 7 (1977) 785-792.