Rearrangeable switch fabric for fast packet switching

packet switching

Rearrangeable switch fabric for fast packet switching V Catania, S Cavalieri and L Vita propose a self-reconfiguring rearrangeable switch fabric based on knockout switch concepts

The knockout switch is a switch fabric proposed for fast packet switchinD Under certain non-uniform traffic patterns (i.e. a 'hot spot contention') the switch exhibits an increase in the packet loss rate. A rearrangeable switch fabric based on knockout switch concepts is proposed which, on the occurrence of "hot spot" traffic, reconfigures itself to guarantee a very low packet loss rate. The new switch architecture is presented, and parameters affecting performance are analysed. An implementation based on dynamically reconfigurable VLSI components is proposed.

functions and discussing the performance increase that can be obtained along with certain key issues relating to management of the switch and detection of traffic conditions which cause reconfiguration. A possible implementation based on reconfigurable VLSI components is proposed, and some basic circuits are presented. An evaluation of the proposed architecture is given regarding the level of performance in terms of packet loss rate and the specific performance of the hardware. Conclusions are finally drawn.

Keywords: packet switching, VLSI, fault tolerance

KNOCKOUT SWITCH CONCEPTS

The knockout switch 1 is one of the most popular switch fabrics for fast packet switching. It is classed as a 'fabric with disjoint-path topology and output queueing '2, as each input has a non-overlapping direct path to every output which eliminates the possibility of blocking or internal contention. It is able to maintain a high level of performance in different kinds of traffic conditions 3. Its performance, however, is degraded when a percentage of traffic entering the switch inputs is directed towards a single output. Exploiting the redundancy necessary to increase switch fault-tolerance, we propose a rearrangeable knockout switch architecture which can automatically absorb the redundant resources, assigning them to the overloaded output, thus increasing performance. The paper is organized into five sections: the main knockout switch concepts are outlined, and the rearrangeable switch architecture is presented, defining its

The knockout switch is an N input N output packet switch with all inputs and outputs operating at the same bit rate. Packets of predetermined length reach the N inputs in a time-slotted fashion. The header of each packet contains the address of the output port to which it is to be sent, followed by an activity bit which indicates the presence (logic 1) or absence (logic 0) of a packet in the current time slot. This address is used by the knockout switch to send each packet to the appropriate output. The knockout switch uses a matrix of interconnected switches so that each input is directly connected to each output (see Figure 1). This means that no switch blocking occurs when packets are to be sent to different outputs. Switch congestion only occurs when several packets arrive from different inputs during the same time slot and are to be sent to the same output. The bus interface associated to each input is basically made up of three parts (see Figure 2):

Istituto di Informatica e Telecomunicazione,Facolta di Ingegneria, Universitadi Catania,VialeA Doria6, 95125 Catania,Italy Paperreceived:18 April 1990.Revisedpaperreceived:29 February1991

(1) Packet filtering zone. (2) Concentrator. (3) Shared buffer.

01 40-3664/91/008451-1 0 © 1991 Butterworth-Heinemann Ltd vol 14 no 8 october 1991

451

packet switching

Figure 1.

Bus

Bus

interface

interface

interface

1

2

3

Bus

\

0

•

•

0

•

•

•

2

3

N

4

Concentrator

o

n

t

/

Q

entering packets in a fixed number L. If M > L, there will be a loss of M - L packets inside the concentrator, as these packets are discarded. In order to provide this function, the concentrator is constructed with basic 2 input x 2 output switch blocks. According to conventional priority logic, if there is a single packet arriving at the inputs, it is forwarded through the left hand side output, whereas if a match occurs between the two inputs, the winner emerges from the left hand side of the switch element and the loser emerges from the right hand side. In Figure 3 we have an example of an 8 : 4 concentrator with these 2 x 2 blocks. The D elements each cause a one-bit delay to guarantee simultaneous outputs. The L outputs in the concentrator are then connected to the shared buffer, formed of two parts: the Shifter with L inputs and L outputs, and L buffers. The shifter's task is to place the packets arriving from the concentrator in each time slot in the buffers in such a way as to implement an Linput/single-output first-come-first-serve (FCFS) queueing discipline. The shifter is managed by a simple state machine based on the equation':

S(i + 1) = (S(i) + K(i))

Figure 2.

Bus interface architecture

The filtering zone has N filters, one for each input, containing the address of the output the interface is connected to. Their task is to filter input traffic, only letting traffic destined to their output pass. All the M packets (M < N) that get past the filter enter the concentrator with N inputs and a fixed output L (l << N). The concentrator's job is then to concentrate the

452

Bus

interface

Knockout switch architecture Ou~ut

1

•

mod L with S(1) = 0

where S(i) is the state of the shifter and K(i) is the number of packets that have arrived from the concentrator in the i th time slot. Packets buffered will reach the output, one by one, by means of a server which implements a tokenpassing service discipline. The main idea which characterizes the knockout switch is that the probability of packet loss due to output congestion can be kept below the loss expected from other sources, such as channel errors. An assessment of knockout switch performance under the assumption of uniform traffic pattern shows that with L = 8 and a 90% load, the probability of packet loss is less than 1 in a million, with an arbitrary N 1.

computer communications

packet switching

Figure 3.

8 X 4 concentrator

There is, however, an unbalanced traffic condition, called 'hot-spot traffic', which causes a sharp degradation in knockout performance 3. Under this condition the L number has to be greatly increased to keep the probability of packet loss within acceptable limits.

REARRANGEABLE KNOCKOUT SWITCH ARCHITECTURE In the authors' proposal there are two main reasons for a reconfigurable knockout switch architecture. The first concerns the fault-tolerance required of the switch fabric, whereas the second concerns an increase in switch performance in case there is a hot spot at an output.

vol 14 no 8 october 1991

Fault-tolerance The modular structure of the switch lends itself to highly efficient and economical solutions in order to increase the level of fault-tolerance. Rather than duplicating the whole switch fabric, redundancy can be introduced, at the bus interface level. In this way, as all the bus interface modules are the same, the spare bus interface can replace a faulty module. Figure 4 shows a basic scheme for substituting a faulty module with the redundant one. Of course, it is first necessary to diagnose the fault (this task can be performed by the entity supervising functioning of the switch fabric), and then inform the controller. The latter activates switches YY, isolating the faulty bus

453

packet switching

N

c o n t r o

I I e r

Figure 4.

Knockout switch with a 'spare module'

interface and inserting the spare bus interface in its place. By means of a writing operation, the controller also fixes the reference address of the spare module packet filters at the value of the replaced module's output. After these actions, the spare module is to all effects able to substitute the faulty module.

Increase in knockout switch performance As pointed out previously, 'hot spot traffic' conditions cause a degradation in overall performance. Yoon et al. 3 obtained an expression for the probability of packet loss which showed that in hot-spot conditions, an appropriate choice for the number/_ of concentrator outputs and shared buffers in the bus interface leads to an improvement in overall performance; i.e. when the hot spot fraction h increases, the probability of packet loss can be reduced by increasing L3. For instance, if in uniform traffic conditions with a 90% load we assume a concentrator with N = 128 inputs and L -- 8 outputs, in the event of a hot spot condition the number of outputs available will be insufficient to keep the probability of PaCket loss below 1 0 E - 6. If these outputs could somehow be doubled, a loss probability of 10E - 6 would be ensured up to hot spot conditions with h = 0.04. This value would be ensured up to h = 0.1 if the concentrator outputs and the number of shared buffers could be quadrupled. Some other non-uniform traffic patterns could be considered to stress the performance of the switch. The analysis made by Yoon et al.3 examines three further traffic patterns besides 'hot spot traffic'. The first is the case in which all the traffic entering a switch input is directed towards the same output. The other two analyze the case in which groups of sources communicate preferentially

454

with certain destinations. In all three cases it is shown that the difference in terms of packet loss, as compared with the case in which traffic is uniform, can be considered almost negligible. This confirms the conclusion that the knockout switch only needs a significant increase in parameter/_ when there is a 'hot spot contention'. These authors propose increasing / by exploiting the resources available on the spare module. To this purpose we have defined a hardware architecture which allows the resources offered by the spare module to be combined with those of each bus interface 3. When the switch control system becomes aware of a hot spot, it uses appropriate commands to cause reconfiguration of the spare module and the bus interface in the hot spot condition. In the n e w configuration, although the bus interface and spare module remain as two separate entities, they form a single bus interface with a sufficient number of concentrator outputs and shared buffers to guarantee low probability of packet loss. As reconfiguration of bus interface modules is rather too complex to achieve by using traditional hardware components, it is implemented with dynamically reconfigurable VLSI components. These components, described in greater detail below, are highly suitable for the concentrator and the shifter, as they have a parallel structure organized according to an array whose elements are programmable cells.

Hot spot detection and management issues In order to ensure efficient use of the resources of the switch fabric it is equipped with a management system, the functions of which include fault management, traffic and performance management, configuration management and a signalling management system. A specific function of the traffic and performance management is that of detecting hot spot contention conditions, estimating their duration and, if necessary, asking the configuration management to reconfigure the output involved. If td is the estimated duration of the hot spot and tr is the time required to reconfigure the output, increasing its resources, the traffic and performance management only activates reconfiguration if: td >> 2tr

This relation must be respected to prevent the system from oscillating on account of traffic bursts the duration of which is of the same order of magnitude as the reconfiguration time. It should be noted that the time considered in the relation is twice the amount of reconfiguration time to take into account the time the system takes to return to its original state. As we shall point out in the following paragraph there are various traffic configurations that can cause a hot spot contention for time intervals ranging from a few minutes to hours. Conversely, in the implementation proposed, the value tr is in the range of a few milliseconds. In order to detect a hot spot and estimate its duration, the traffic and performance management moves in two directions. The first involves interaction with the signalling management system, through which it is possible to

computer communications

packet switching estimate the most significant parameters of the connections in transit through the switch. These parameters include information about the traffic statistics for each connection (e.g. average bit rate, burstiness, etc.). In the hypothesis of connection-oriented traffic (e.g. voice and video), in fact, a signalling phase always precedes activation of the connection; in this phase, besides establishing the routing path between the source and the destination, the amount of resources required is also communicated. On the basis of that information about all the active connections, the traffic and performance management can therefore detect the existence of a hot spot contention and foresee its duration (in relation to the kind of connection). Another direction the traffic and performance management takes for the detection of a hot spot is that of collecting traffic statistics and updating an MIB (Management Information Base). This allows identification of periods of time during which there is an increase in traffic on a certain output.

The first concerns management messages exchanged between the nodes of the network, which in some cases are configured as a concentration of traffic towards the network control centre. The second, in a B-ISDN environment for instance, regards temporary increases in traffic on a certain physical path, due to the remapping of connections in the even of faults.

HARDWARE

STRUCTURE

Bus i n t e r f a c e a r c h i t e c t u r e

Each new bus interface has two functioning modes: the normal mode and the hot-spot mode. In the normal mode, the bus interface functions in a standard way with N inputs and one output obtained from the L outputs of the concentrator and the shared buffer (see Figure 2). In hot spot conditions, the bus interface is reconfigurated as shown in Figure 5. Each concentrator is reconfigured

Some application scenarios

In this subsection we shall consider some application scenarios, all relating to computer networks or, more generally, to packet switching telecommunication networks in which hot spot contentions occur quite frequently. The first general consideration to be made is that a shared-resource environment like that of computer networks, can potentially present problems of resource contention and therefore hot spot traffic. An example is access to a file server on the part of several terminals located in geographically remote areas. There are various application environments which have this characteristic. In the LAN environment, for instance, data and programmes are frequently concentrated in a large file server and there are a number of discless workstations which access this file server to load their files. These operations are frequently concentrated in certain periods of time, thus causing a hot spot contention on the file server. In banking applications, at certain times every day there is a high concentration of traffic moving from a number of terminals distributed in the various branches towards the bank control centre to update the databases it keeps. In these cases, and in many other similar cases, collecting traffic statistics would allow the network switch fabrics to foresee the amount and duration of hot spot contentions. If we consider the use of the knockout switch in a B-ISDN environment, in which not only data but also voice and video are transported using the asynchronous transfer mode (ATM) technique, besides the collection of traffic statistics, an essential role is played by the information obtainable from signalling procedures to open connections (as pointed out above). As far as the duration of a hot spot contention is concerned, in the case of voice and video traffic it is quite obvious that on average it can last for a matter of minutes or even tens of minutes. When the contention concerns data traffic, considering the examples given above, the average duration is at least a few minutes. To conclude our discussion, two further cases can be considered as possible causes of hot spot contention.

vol 14 no 8 october 1991

Module i

Spare Module 7

f-

.~ N/2 Iower l -" input line I

~

I I

Concentrator//

2L

N/2 upper input line

\

I

~ L-bus

Spare / concentrator /

/2L

Adapter

SU-bus

Buffer

I

I

Shifter

Buffer

T-bus

O ut~l~put i

Figure 5. module'

Knockout switch reconfiguration with a 'spare

455

packet switching with N/2 inputs and 2L outputs. It is possible to verify 1 that the average number of switch elements in a concentrator (supposing N ~>L) is equal to N * L. So reconfiguration of an N-input L-output concentrator into an N/2 input and 2L output one does not require resources other than those already available. In order to obtain an N input 4L output concentrator, an adapter has been devised to interconnect the outputs of the two concentrators. As shown in Figure 5, the shifter is reconfigured into 4L inputs and 4L outputs. For this purpose we have assumed using the resources available on the spare module so as not to make the size of each bus interface excessively large. Of the 4L shifter outputs, only L are sent on an external bus (Su-bus) and are connected with the shared buffer of the bus interface. The remaining 3L buffers are obtained from the resources available on the spare module. The 4L X 4L shifterwill fill the 4L buffers (situated in the two bus interfaces as outlined above) according to the algorithm described above. In order for the 4L shared buffers to be served according to an FCFS discipline, the two servers belongingto the two bus interfaces have to be synchronized. This is achieved by passing a token from one set of shared buffers to the other by means of the token bus, in order to obtain a single output.

IX) BLOO<

\

CONFr.,URABLE LOGIC BLOCK

INTERCONNECT

AREA

a,

G

Implementation As we mentioned in the previous section, for the knockout parts that have to be reconfigured in the hot spot mode, LCA (Logical Cell Array) components are used s. In this section we give a brief description of the LCA components and then present the reconfigurable architecture. For reasons of simplicity, we will show a concentrator with 8 inputs and 2 outputs which is reconfigured into one with 8 inputs and 4 outputs, an adapter which, as we pointed out above, has to be coupled to the concentrator, and a 3 input/output shifter. The LCA is a VLSI circuit made using CMOS technology. Its internal architecture is user-programmable and consists of three configurable elements (see Figure 6a): IIO blocks, logic blocks, and interconnections. The logic functions and interconnections are achieved by means of a bit map memorized in an array of static memory cells inside the component. Several bit maps can be defined, corresponding to distinct LCA functions. An internal logic allows automatic loading of a certain configuration, and it is possible to change the bit map, and therefore the logic function, at any time. Each inputoutput block (IOB) provides an interface between the functions inside the component and its external. It is possible to define each block as input, output and bidirectional. Whichever the case, it is possible to programme compatibility with the TTL or CMOS logic levels. The configurable logic blocks (CLB) are placed in a matrix at the centre of the component. Each CLB has a combinatorial logic section, two flip-flops, and an internal control section (see Figure 6b). There are five logic inputs, a common clock input, a reset input and a data input for

456

Loc,c -VA~Ae~ES --

i¢ .~

,!

te, Ar~

( CROCK--

......

2I!

........

,~; -1. (~/mcE)

(XOCK --" ,h nES~T - -

,[il

! Figure 6. structure

(a) Logic cell array; (b) configurable logic block

the flip-flops, and two outputs. By means of an internal look-up table, it is possible to define a wide range of logical functions involving the CLB's logical variables. More complex functions can be obtained by interconnecting several logic blocks. Implementation of the concentrator Each 2 X 2 contention switch is made up of a control circuit which chooses which path a packet in arrival has to follow during the time slot, according to the priorities described previously. Two edge-triggered flip-flops have been used for the control circuit (see Figure 7). These are synchronous with the arrival of the activity bits, and for this purpose there is an external clock triggered to each time slot edge. This guarantees that when the activity bits

computer communications

packet switching

Q

CL

5

: i

I

CL

Figure 7.

2 X 2 contention switch

edge-triggered flip-flops which are synchronous with the activity bit, and a switching circuit. Here again, the presence of flip-flops guarantees that the path assigned to each packet remains unchanged throughout the time slot. Implementation of the shifter Figure 10 shows an electrical scheme of the circuit of the 3-input 3-output shifter. It is basically made up of three functional blocks:

Figure 8.

Implementation on LCA of a 2 X 2 contention

switch

arrive, the control signals established on the basis of these bits remain unchanged until the next time slot. These signals directly control a switching circuit, essentially made up of three AND ports. In this way, the path assigned to each packet remains unchanged throughout the time slot. As far as the clock signal is concerned, the clock signals for each layer have to be derived from it as it is necessary to take into account the inevitable propagation delay for the activity bits through the same layers. This is achieved by providing delay blocks on the clock signal between one layer and another. Figure 8 shows the 2 x 2 contention switch mapped onto an LCA component. Implementation of the adapter The adapter created through the Logic Cell Array concerns the connection between two 4-output concentrators and an 8-output shifter. The functioning logic is that of directing the right hand side concentrator outputs to the first left hand side input whose activity bits are not active. In this way, the active right hand side concentrator outputs are compacted starting from the first left hand side concentrator output with a low activity bit. In Figure 9 an electrical scheme of the adapter is shown. It is possible to note the presence of a control circuit made up of five

vol 14 no 8 october 1991

• an interconnection networkwhich allows each inputto be directed towards the chosen output according to the shifter's functional logic. The path is determined by three control signals which remain unchanged throughout each time slot; • a state memory which allows the state in the i th time slot (i.e. the presence or absence of a packet in each buffer) to be memorized. This information is obtained by memorizing the activity bits as they leave the shifter. They will be used in the next time slot (i + Ith), thus configuring the interconnection network. The memory is made with positive edge-triggered flip-flops and driven by a clock (CIk2) synchronous with the arrival of the activity bits leaving the shifter. • a configuration memory whose task is to memorize for the duration of a time slot the configuration of the interconnection network described above. It is made up of three positive edge triggered flip-flops driven by a clock (Clkl) synchronous with the arrival of the activity bits at the shifter input.

REMARKS

ON EVALUATION

The rearrangeable switch fabric we propose can be evaluated using the analytical approach followed by Yoon etal. 3as it is essentially based on the same concepts as the knockout switch. We can therefore deduce that the most

457

packet switching I0

CLK

Ii

I2

I3

I4

I6

I5

I7

[

U0

U1

U2

U3

U4

U5

U6

U7

Figure 9. Adapter: electrical scheme

I2

I1

13

CLKI.PAD

CLK2.PAD

01

02

03

Figure 10. Shifter implementation

458

computer communications

packet switching 0

40000

80000

I

I

I

120000

160000

I

200000

1

I

P/C-SILOS 3C.8 * 0 50000

I00000

l

150000

l

200000

l

250000

t

l

CLK ~

i1 [

I

I

~2[ JI

I

I

I

2

I

1

I0 [

I

m

I

!

-1

I

~

•

i

I

•

•

•

2

3 I2 [

~[

•

_1

I

I

I

4

s 13 [

I

Ls[ 0

50

I

I

i00

150

200

250

I

I

I

I

c~L[ - - [ I1 [

12 [

~J----I

w.[

6

z4[

5

J

z5[ 1

I6 [

2

I7 [

•

•

•

n

•

•

•

•

!

l

6

• i

•

• !

•

u

•

•

•

•

7

_l__

8

•

i

9

z

J

L

LS [

5 6

Figure 11. (a) Input, (b) output relationships and time constraints of a 2 x 2 contention switch for 40 Mbit/s bit rate serious condition leading to an increase in the packet loss rate is that of a traffic pattern producing a hot spot contention. Even in this condition, the architecture proposed is able to maintain a lower packet loss rate (10E - 6) than the knockout switch, thanks to the increase in the concentrator outputs and the resources in the shared buffer provided by the reconfiguration described above. The use of LCA components gives reconfiguration times of around 20 ms, thus making the switch able to cope with a number of traffic conditions determining a hot spot contention, like those described above. The performance of the hardware designed for the implementation has been assessed using a simulation tool called SILOS, integrated in the system of development of the LCAs 6. The hardware was simulated analyzing both its logical and temporal behaviour. The results obtained show that the whole architecture can function correctly up to bit rates of over 30 Mb/s for each switch input. Figures 11, 12 and 13 depict some simulation tests performed for the circuits described above. They show the logical and temporal behaviour of the activity bits of the packets passing through the circuits. Figures 11 a, 12a and 13a allow analysis of the logical behaviour of the circuits when subjected to various input patterns; Figure 11b, 12b and 13b obtained from the previous ones by expanding the time scale, show the time relations between input and output.

CONCLUSIONS

The solution proposed has the advantage of exploiting redundant resources normally used for fault-tolerance in order to increase knockout switch performance. Implementation is made possible by using dynamically reconfigurable VLSIs. The architecture presented can

vol 14 no 8 october 1991

uo [ - ! ~ ul [ J l ~

!~__dR__|__R

!J--I

u2 [ J ~ u~[ | | u4[ |

!

i---i

- ! 1--1--4

12 t3

!

l

n

I

!

l

l--|

u5 [ u6 [ uT[

! ....

"

14

-

16 17

.......

* P/C-SILOS 3C.8 * 0 5O

i

18

100

150

200

250

I

I

;

l

zo[ -3 I n[ - J - - q ]2[ z3[ ~4[

s --J

i~[ ---J

" I

6

I

7

I6[

8

z~[

9

uo[

J

ul [ u2 [ u3 [

;--] [--1 i--]

I

lz 12 13 14

U4 [

15

us[ u6[

16 t7

U7 [

18

Figure 12. (a) Input, (b) output relationships and time constraints of an Adapter for a bit rate of 40 Mbit/s

easily be extended in the presence of more than one redundant module. In this case, it is possible to increase even further the outputs L of the concentrator connected to the hot spot output. The reconfiguration time of the LCA components (about 20 ms) allows the proposed solution to scale up efficiently to many application fields which potentially lead to hot spot contention.

459

packet switching 0

soooo

I

I

CK1. pAD [

I

I

100000 150000 200000 250000 I

•

•

I •

•

I •

•

I

•

cK2.P~[ ~

1 1

CK1-PAD [

2

CK2.PAD[

~1[

!

•

•

•

I

~

•

•

I

I

3

i1 [

z2[ I3[

I

•

•

•

•

I

•

_1

•

I

4

12 [

I

oi [ - I I J 02[ J l - - I 03[ l Figure 13.

•

I

I

•

l

•

I

!

I

i

R

!

!

I

_=

!

I-

!

!

-=

25050

25100

I j--

I

25150

I

i

25200

25250

I i 2

J--I

3 4

J--I

•

5

[3 [

l

7

oi [

J--I

-=--

0

o2[

]--I

!--

s

03[

[--[

5

(a) Input, (b) output relationships and time constraints of a Shifter for a bit rate of 40 Mbit/s

ACKNOWLEDGEMENTS

This work has been supported by "Progetto Finalizzato Sistemi Informatici e Calcolo Parallelo" of CNR.

REFERENCES

Yen, Y, Hluchyj, M G a n d A c a m p o r a , A S 'The knockout switch: a simple, modular architecture for high performance packet switching', IEEEJ. Selected Areas in Commun. (October 1987) pp 1274-1283

460

25000

l

2 Ahamadi, H and Denzel, W E 'A survey of modern high performance switching techniques' IEEEJ. Selected Areas in Commun. (September 1989) pp 1091-1103 3 ¥oon, H, Eiu, M T and Lee, K ¥ 'The knockout switch under nonuniform traffic' Proc. IEEE GLOBECOM '88 Hollywood, FL, USA (November 1988) 4 Catania, V, Cavalieri, S and Vila, / 'A reconfigurable knockout switch for high performance packet switching' Proc. 32nd Midwest Symposium on Circuit and Systems Urbana, IL, USA (14-16 August 1989) 5 XILINX: The Programmable Gate Array Data Book 6 XILINX: P/C SILOS, Users' Manual

computer communications