SKALP: Skeleton architecture for fault-tolerant distributed processing

Exrensibilit;p. .e. the possibiiit:/ to change system functionah y and performance without changing the system design. The extensibility is achieved bs the way of pools of resources at different Ievefs of the s.Wem (communication lines, hal:lware (a small number of different board types, a board of a type being interchangeable with another one), of adaptability to changes in hardware technology (for instance the 8 bits to 16 bits CPUs evolution). Integrity, A hardware fault-tolerance management is integrated to the kernel cf SCALP and provides the mechanisms needed not only by the hardware gradeful degradation but also by the software graceful degradation. The robustness is an important characteristic attempted in SKALP. Exchal;lge of fault-tolerance infomation bet>\een the clearly sepacattd SKALP and application levels allows the design of safety-orrented and/or availability oriented appiications. The performance requirements are xhieved mainly by the way of pools of resources. Implementation of the basic ‘functions (communication mechanism, etc.) in the hardware/firmware (thus reducing run-time overhead), and adaptability of SCALP to rhr technological evolution permit a n.,table increase of performance. A relatively small cost is obta!ned by us& standard circuits. especially microprocessors. In the framework of a close cooperation between the Computer Architecture Group of IMAG* in Grenoble and the Architecture and Command Group of CNET** in Lannion, SCALP is the low-level component of a project called CANOPUS whose completion is to get a system fur the control rbf a telephone switching system. But the hardware architecture, fault-tolerance management and software kernel are designed in ??

??

1Nowwith

Opfios .%A., Zirst, 38240 Meylan, France. *Now with TBlimBcanique, Z&t, 33240 Meylan, France.

order to obtain a general sLeleton which may be speclaiized by producing an application-oriented software and additional peripheral hardware. A large class of dedicated systems may thus be generated. The latter sections are concrrned with particular aspects of the skeleton: hardware and software communication mechanism and architecture, hardware fault-tolerance.

Standard Statiorr

2. Trae Hardware Architecture

The system is a federation or loosely coupled processors with a relatively smah rati ot c?crh~rtgcd information. The SKALP architecture is d k~cal ne?work of stutions which con~munrzatc by serial lines (Fig. 1). The local network has an exchange wa.y, called “highway” which is a pool of several autor,omous serial lines working in a traffic sharing mode. Each highway line is a serial shaled bus with the entirely distributed control (no cenrralized allocation); the mechanisms, existing OI each station for each line, are strictly identical. This “shared mode highway” communication mechanism, called LISA, is a very important SKALP feature which was chosen since it fits decentralized control, extensibility, integrity, performance requirements. The number of highway lines depends on the throughput and availabihty requirements. Each network station has a multi-microprocessor structure (Fig. 2): - a set of couplers, one to one with re:.pect to lines; the couplers implement the LISA communication mechanism; alt the network couplers are strictly identical. - a main treatment processor, whose hardware and software kernel are identical for every station; this treatment processor performs the application-oriented software. Depenchng on particular applications, special purpose processors may be added to the station, as for exampie a fast signal processor or a timing processor for real-time applications. Thus each station has a distributed structure. The pro-

Modularity of :~erformancc b!: IIX numhcr or! lines of the hic,hwayr and by the r~urr:b~~-.>t stations; Hardware gra<:8eful :Jegradatir,rt: ir 01 ~p!mrr devices are ab:;olutely independcrri I i ~~LIIIS~~Z may loose some coupling devices but ir 1113)

“_-----

l-

f

--_--

1

_

1

L--_._-.

Tic) 2 Str~ctwe

r

at a Standard Station for 3 Lir es and 128 K Bytes Memcry

(5)

01 ienred applications by an adequate choice ol the number of each resource (communication lines, identical stations); Maptability to hardware technology evolution: if’ some parts of the system are independent ecough functions evolution

to support

the integration

of some

in cusforj&e&rr chips or/and the towards more powerful1 stundurd

chips; (6) Easy use for dedicated systems: if facilities are @,eneral and powerful1 enough to be used by a large class of applications. The choice of the number of each implemented pool resource (communication lines, identical stations) depends on the performance and availabilit:! requirements of a particular application, The numbe: of implemented resources is greater

than the strictly ncce~sary number which depends on performance requirf3nents. The total number is determined in such a way that the rrumber of nonfailed resources is Iarge enough to wait until the next maintenance, with a gi-ben “probabiiity”. More precisely, the distribution of the cumulative down-time is determined for an expected total USC of the dedicated system. How to handle the dcrivation of that distribution for a perio~~ctrl maintenance and pools of resources. is developed in [4]. A periodical maintenance, i.e. a maintenacc which only takes place at predetermined times, is recommended for systems based on pools. IE the computed cumulative down-time does not fit the availability requirements, rh 1 designer may either increase the total number of resourcch, or da:cl-east the period of maintenance. If we want to compare OK system tci iimililr fault-tolerant systems, we must point ou! some details. For example, compared to the above aentionned SlFT system, SKALP is a network made of iitdej~ndenf modules (which are in fact stations: there is no possibility for a treatment processor to read directly data hold in any memory). Compared to the FTMP [9], SKALP does not involve hardware voting. Compared to the building blocks of Rennels, Avizienis % Ercegovac [ 191, SKALP does nor involve duplication of CPU. Some other remarks could aIso be made to compare SKALP to one or several of these systems, such as an entirely distributed control for communications and a modularity useful tar the design of several dedicated systems, due for erample to the extensibility and to the welt separated IeveIs of the skeleton, including hardware and software kernel, and of the application. 2.2. LZSA: the Communicurion

Several autonomous serial lines make up the exchange way called *‘highway”. Each station may send/receive on any line. Each line is a serial bus with no central control. Each bus is supporting allocation, synchronization (timing), and data on the same physical serial linr. ‘The LISA mechanism determines the dialogue at low lewls of the system: elementary allocation oi’ the bus i\ obtairwd by mean; of a “seif-allocation” mechanism alliikJg0~L5 tn 1151. Each stati;~ has a “free-line” detector .arrala ~‘conilict” dcrc~3or. Deirc~ciou 2.2. 1. “I+iw-lid The line can bc in two states: !. “‘free-line”: no starion ti’anstrtits or3 ihv linl:; ;i rrCw rransmi3sion cizn 5tiXi3; 1. ’ :>u5v-linc:“: .Yre (or cevCrelr station(s) emits on :i;~ line; all now transmissiom are prohibited. l hc l‘rco-lint detector is an allocator performing rhc “I’rrst coming-first served” politics; but it is posib!e IC! have several “first coming”; then, the allo~~~rion is continued by th: conflict detector (see below). Xlorc precisely. tte free-line detection works ah t’ol!~:ns: at timr I. tqe line is declared free if irs state d::es not ~hangc d~rrEnp the interval time It-TOW, tl. .Is btation \cill +t.irt to emit when it has detected the free line. The TCCC duration j, such that, during the emission of a message, the time between two change% of the state of the line is always less than TCPCC.

Mechanism

In the distributed systems, the mechanism for communication has a crucial effect on both the qualitative “flavor” and the qualitative performances of the system. The LISA communication mechanism (from French: LIgne Skrie Autoaliouce), has the following features: (a) common exchange way: the bus (a passive broadcast medium), (b) distributed control over aII the stations.

This &p&on is active during the ef’fcr!rraintorniatii~rt tion by the station” f “transported by the line”. A station detecting a conf%cI Gl‘ni Ic) emit. Thus, the other (concurrent) stations may

By acting on the addresses recognized by a station, it is possible to consider two types of messages: (1) messages for an unique destinator, (2) global message (broadcast message): for several destinators. The implementation of the LISA mechanism is a specialized processor, the coupler. Each coupler of each srztion works in haifduplex mode on the highway’s line associated to it. The coupler performs also several functions relative to the fautt-tolerance (see Section 3. Some practical data are given in Section 5).

3. Fault-Toierance

Features

Fault-?oierance in the system is based on a self der~ction of failures at the level of microcomputers and CPU boards, and on wired sigrrcrls inside a station. These mechanisms are used to partly create exceptional and abnormul conditions which in turn are used to update the SKALP configurelion and to produce reports from SKALP to rhe application software for reconfiguration actions in the application. Report; f!:o~ the application software to SKALP are also used. It is assumed that the communicating stations exchange informations (no one.wsy communication). 3 1. &GC Mechanisms B&c mechanisms exist at the levels of microcorn. puters and CPU boards, and inside the stat&. 3.1. I. Microcomputer Level A special effort has been made on the study of tk detection of fakes of microcomputers ig SKALP. The aim is to detect failures on-line {i.c. concurrently with standard programs), without using massive redundancy (TMR, .._) or dup.ication/comparison. A methodology has been developed (initial partial results were presented in [2]), in ordrr to study the detection by the occurence of abno: mal events in the software and by periodically executed specific test algorithms. Abnormal events can be: ?? the end of a watch-dog, ?? an attempt to execute a non-implemented operation-code.

_._,_ _ ._

ETRE

:

ETAR DEM END 3ER.H CNX

: : : : :

ACNF

:

FSI RCP TSO TXACT TCP TxENA SFR RKACT

: ; : : : : i :

These events :%ect failures that rffecr the program sequencing whiie periodical test algorithms are used to test failures that do not affect, in

genera!, prrtgrsrn icqr~erxin~. %CC*il::i: i9f ikr,:. tion Of arrothcrabnwma~ Wt;‘;llhw :riQii’ci:?;lib; plementcd:

detection) for failures affecting the hearr of the CPU die. For example, if a failure affects the slack pointer register of the M68OO, an error latency will be about 13 instructions and the frequency of the occurrence of the above event will be l/50 instrucEons. ?? To determine, by functional simulation, the characteristics of the detection parameters followir?g the occurence of one event “there is an error nn the program counter”, and to extrapolate the preceding results, using modeling, in order to get results as a function of the frequency of the above event. If f denote the frequency of the event, the mean detection time is given by: E[detection

0.822f+O.O63 time] = ---

b.Gd8f’i0.027Jf’

the size of the sample being 500. ?? To determine for failures which do not create the event, the “test-programs” which wifl be periudicatly executed (for faiium afftcting some parts of the data processing section, like operators,

and for failures affecting some parts of the control section, like the sequencing of ADD, SUB, . _. instructions). Regarding the data processing section, fault-hypotheses depending on the physical structure (contacts aluminium-diffusion, Frecontacts polysilicon-diffusion, etc . _) and on failure mechanisms, have been taken into account. With regard to the control section, we have made on-line oriented functional hypotheses for the sequencing of differentiated instructions. No:e that only failures which do not create the above mentioned evect have to be tested.

d 7 CI drteminc to

be anaig&

Ihe upliolu~

t(J fkc w:ftc/k&$

tMpfm~l~~ JflJ

ffiilt

am

fb ~~fWhc’rt~~

cxwmd test protywrr~. ‘I’lds hf jxrint t6 link& 10 the rest of the ch~pa csf ;t board. AH other chips of u board are tesfa by periodically executed test prqrarns, as each CPU board contains at Ieast one CPU. The paramclcrs are: ?? the period of a watch-dog; ?? the length of rhr execution 01’:lre waarch-dog: ?? the diffcrcrtt fxriods of the differerrt te+t programs a!&oclatCd to lhc parts ol’ fhc tfic ot’ the microprocessor which have tu bc periodically tested and with the different chips of the bnard; ?? the Icngth of the exccruion of rhc tL%tj~ogrz~m*: ?? the occurence probabijiriti of 0 f4ifrm oJ! f!JC differcm psru of I trr die of rhv rirkrul#r,oc~~Lr. and on the different chip% of the lto~d. ?? the detection time, associated to the WjI end IOC detectors. We distinguish two diffcrcnr problc:t~~. Probfem I: What LIC the optin~;lI jli’rir%!L. built that the mean detection ~irne is mlaimir;d. urr&-r the constraint of a @en ovcrheud in C-PI1 tirnc: Problem 2: What arc the optimal periods, such that the overhead is minimized under the constraint of a given mean detectian time? Nore that these problems are not duaj problems; they have only the property of exchanging the constraint and the function. These problems have been solved lclassicnl Lagrange multipliers sufficient), and arc analytical/geometric solufions have been given I.\]. These solutions allow to pururnefri:e the dctcctlon time/overhead in CPU time spent for lkilurc dcrc~tion, as a function of fhc applic;ltion rcyuircmontc. Nevertheless. the managing of fault-tojerancc in the high levels will have to take into account that detection time. Alf the failure detectors for a CPU board act (directly or not) on a wired J#ihre signal which immediately stops the CPU (and disconncctc physically a coupler from his communization jrrrc if the CPU board is a coupler).

3.12. Sfaiion Level A station is composed of several CPU boards, each of them having a wired failure signal. which may be observed by all other CPU boards of the station.

again AVAIL when a message from him will be received, since a particular communication line is not undefinitely not selected for the emission of a message. The SKALP configurazion is thus aulomaticail.v updared> in case offuilur: or overload (if adequate policy is used for the cho;ie of a line for emission). It is clear that only emission/reception of messages may be useful for SKALP configuration updating, since a station which dots not communicate with other stations has not to take care of the system configuration. Morcover. except during messages exchange phases with other stations, the exactness of the set of assumptions is not importamt. Hence, the view that a CPU boa.rd has of the system configuration may, or may mt, correspond to the real states of the CPUs bclon$ng to other stations (and this is not due to detecrton time of failures or information transmission delays). 3.2.2. Reporting and Applimtion Reconfiguration Reports to the application heve to be produced in order to inform it about: - loss of information in a ccaupler CPU board, - loss ot information in a treatment CPU board or loss of the CPU board itsL.lf The reports have to be produced: - by a unique reporter (in order to avoid cascaded recovsry/rcconfiguration actions in the application); - iff they are necessary (in order to avoid intempesrive reconfiguration actions). It is clear that exceptional conditions detected outside a station cannot be ured, since two kinds of events have to be reported: - effective failure/repair of a treatment CPU transition or a board (a real TEST-rFAILED real ABSENT-+AVAIL transition) or a loss of information (a real AVAIL-TEST transition of a coupler or a treatment CPU board); - an erroneous disconnection of a CPU board (while running programs): a real AVAIL-’ ABSENT transition or a real TEST-tABSENT t. ansition. It is again clear that only a CPU board having a exact view of state transitions may report: only a CPU board belonging to the station where the transitions occur. That CPU board is the reporfw

3ZJ

dcl’lncdin 3.1.2. Recovery of lost lnf~~mI3rknby mvans of Jupli* cation in different statiorn for roStb;rck. and hcncC corfsisrency lowcct

probfems

due

(added

to the

levtls

emapstdation the

detection

donho

time

crfcct

in

[I&

rcconfigurable portion\ of sofwate, and urnmic& of the

of

appiicatlon

operations

to

concerning

the change

or

the sy4tCm

configuration in the oppllcation ~ofrwarc (no inconsistent

States

when

sysrem

canfipuriltion

changes from one state ro another 116))bave to bc provided

by the application

applications ccrning of

uGn8 zhc W*

may hilvc dil f’crcnt spcrilrcaiicmr

the lorr of information,

primitives

recover These

\oftwpre

SKALPtcvcl.JndcrJ,diJTcwnttyph al

ports of the

at

data

app~iCatio0

and/or

primitives

app:ications

etc.. Ievcl

processe%

will bc useful

is

in

order

to

iJt ~~PZIW.

for wJJJc

(availadlity-0ricJWd

CoJJ-

. ‘I JJc rtntl~

&SC*

~$1

onC\).

3.3. Abnorml Cvrrdifiuns As are

exceptional

conditions,

abnormal

de[ecled inside and outside a

only

condition4

station.

but not

SKALP level may derecr them: the applica.

(ion software tions,

may

Abnormal

dctcct

conditions

some

abnormal

Condi-

have in this case to bc

reported 10 the SKALP levei.

3.3.~. Detection Inside

a

which

is not

stalion,

a

confirmed

AVAIL-•‘tFSf by

the

tranGtion

dctcctlon

i.e.the

of

a

concerned CPU board bccamr% again AVAIL, is an abnormal condition. Inside or outside a station, everything not normal detectedby the applicationsoftwarc is an abnormal condition. Inside or outsJdc a statian. something like a mismatch in cheek-sums, etc.. dctccted by SKALP level is an abnormal cotta dition. Only suppositions may be made for the occurrence of abnormal conditions. These cone ditions may be due to: - a not yet detected failure in a CPU board {due to the detection time), - a permanent but not tested failure because out of the failure hypotheses (test programs arc failure,

always

derived

- a transient

under

faiIurc,

faihtre

hypotheses),

8 Court+

322

~w*qtcnnn, wpt. ._

confi(rb:ation

eta/

1 Skeleton Architecture for Fad-Tolerant

updating

and reconfiguration

.__.___.--..-__l

Exceptional - real state transttions

detected

using exceptional

and abnormal

-_______

Abnormal

_ reception,

by SKALP,

- everything

of

towards

etc..

transitions

AVAIL

- outside a station by the repotter

- all state transitions

may be re.

ported to SKALP

by broadcasted

messages for SKALP nsjWJr:,

‘;:alrll~~rrratIur~ r’0114Wq

conFigu-

ration according

cerned station,

to the emission/

reception. - report of the apparent

AVAlL

to the apptication.

by an

appllcdtion-defined

station,

reconfiguration

of treat-

the con-

if detection

by the application/outside

TEST-ABSENT. or ABSENT-_AlLED

- report to SKALP/to

station.

TEST transition of the tast coupler CPU board of a station

TEST-FAILED

by SKALP

a station

^___

- updatmg the SKALP cofigu-

ration updating, - ail transitiox AVAIL-TEST AVAIL-ABSENT transitions

by SKALP, AVAIL-TEST

detected

- inside/outside

CPU board,

_~.._~~.~ . _.I_~__._ -..___. reports are made

detected

of check-sums

detetted

- not confirmed

by SKALP,

a supposed

abnormal

- mismatching

CPU board, - failed en-&ion,

corldidons

by the application,

a supposed

TEST, FAILED or ABS,ENT

inside a station

conditions.

condit,ons _I_message frop

by SKALP

Oistrtbut~dPfocessiflg

application

actions

is done a

for accounting--test/

reconfiguration conditions.

as for exceptionai

for

in the

software.

ment CPU board, have to be snd

reported to the application. These report5 will be eventually

Apr.X3ia!#Or\

followed

rrihnrl$~u’aiWrr

acttons irecovery. npplicnti9n

by recontiguration etc

.i

in the

software.

_.._._,...-. ._...._ ____.___._ ‘i-:!t,!?i J’.l,:, I-Jifcb i‘f’ii bnnrd enterlng the system _ . ~~__~ ..._ I”..~~.~,~__

ih.c to suppose the others tu be AVAIL.

_--

-

matically of communication lines overload, and finally: they are simple. Lastly, they are oriented towards local networking rather than mechanisms derived frorl the Preparata-Metze-Chien model 1171.

4. Software Gtructun: #. 1. The Overall

Sojtware

Sfruclwe

The main notions us:;d in the system are the processes and the events. The processes are sequential. The application ii a set of processes implemented over the stations of the SKALP network. The stations have a microprocessor-oriented realization and, as in [lj, each slation is mulriprogrammed.

The interactions between a process and its environment are the events (even&& accompanied by parameters). The handling, detection and

circulntion

of the evcurs bctwecn the ~IOWWS WC

performed by the hardvvarc and software skcIcton. while their treatment has IO bc performed by the processes. Thus. the synchronization

par’ has beru la.c~r-

ited, because of the class of apphcationr anemp:cJ hy IIIC skeleton architccturc. Thcsc appliratiort-, have a high

&JJVX

of intcractizn

b~t\rcen di+trtnt

pTCWSSt!S.

The software implrmcnlcd NI the cq&r~ forms the LiSA communication rnc~+~n~

t’~t.

Section 2. I) and sonic f’un~lion~ cwwrning

rlw

f- 1,

failure deteclion and reporting (cf. Section 3).

The software kerncf implemented on the t:c.t~ment processor is the support of the applicatinnoriented software. The kernel has IWO parts: an acti1.r part (the “scheduler”). and a passive part (a SC~Iof printitives offered IO the application). The scheduler is the main entity for the pro
the schcdulcr aHocatc~

the CPU by activating the corresponding process. Nofe: This “corresponding prh’css” is easy to be determined destinator

because of the explicit

or implicit

process associated with the major pllrt

of the events: each message has a dc%tittator process field, a physical faihtrc is alw~~gs dedicated to a specialized process . . .

an

cwnr

After the processing of the event, the activated process must give the control back (by using an cxplicit wait primitive) to the scheduler. Several primitives are offered to the application: send message, allocate memory, process end, wait, activate process, create process, initiate timer, etc. An original mutt_al-exclusion mechanism is also offered to the application

[12]. This mechanism is

especially adapted for distributed

processes since it

5. Prototype

Characleristics_COncluslon

sy.;tem is presently being Carried Out, serial lines. When the prototype definition was in progress, the 16 bits microprocessors were not available. Hence, our network prototype is based around the M680O family of MOTOROLA. Two types of boards have been developed for a standard station: - a coupler CPU board involving about 60 ICs a MCBEBOO, which performs the ! ;3A communication mechanism. Each coupler, thus each highway’s line, has an exchange rate of 300 Kbit/ set on 250 m. - a treatment CPU board involving about 70 ICs around two MC68300; one of the MC68BOO microprocessors is specialized for real time applications, the other is the main treatment processor. The addressing rsrnge of the MC6800 treatment microprocessor has been extended to 128Kbytes and 1024 processes may be supported on a station. Without fault-tolerance management, the software kernel is about 4 K bytes: without specific test programs for parts of the die of microprocessors, and without test programs for other chips of the bnar ds. the fault-tolerance process is about I .5 K bytes. Eight staticns connected by two serial line-s are currently running in Lannion. Due to thz well defined interface between coupler and treatment CPU boards, the availability of the 68000 microcomputing unit led to the rfesigl! of a treatment CPU board based GC this Eircuil, The board is using the debugging facilities of MOTOROLA. the MAXBUG, but neitheT the kcrrlcl software nor the fault-tolerance mechanisms (hardware and software) ate iinplemented yet. Bus connectors being just e.ctended, a station may be provided either with a Cf 00 treatment CPU board, or with d 68000 treatmedrt CPU board, ?%&?ain due to the well defimd interface between soupier and treatment CPU bzztrds, the availability r:~t it S~+jCS~i,i chip implementing the LISA comr~~~ti~!rtrorrm~chnnism will lend to the design of a I>~‘I+ c<)~&Y CYU board. This chip will integrate in B lb; pins chip ate inlerface implementing the LISA mechanism and an ETHERNET-like mechanism A firototype involving

two

for large size packets and long distance communications [EFC 801. The use of this chip for the design of a new coupler CPU board will involve a drastic decrement of the nKk_ber of ICs. Indeed, a prototype entirely based on 6800 is useful for experiments. but a computing power well balanced between coupler and treatment CPU boards, woufd be more attractive for an exrensive use of the system. These hardware developments illustrate the facilities offered by the SKALP system: while some hardware developments are made, both the software kernel and application software may be simultaneously developed. References [l

I Y.P.

Chien, Multitasking executive simp//fies realtime microprocessor system design, Computer Design,

Januaryl980. 121 8. Counais, Some resutls about the efficiency of s!mpte mechanisms for the detecriun of microcomputer ma/functions, 9th Fault-tolerant computing symposium, Madison, June Xl/22

1979, U.S.A.

131 6. Courtois, Optimal frequencies for concwrent and periodical testing of logical systems, 2nd International conference on reliability and maintenability_PerrosGuirec, Tregastel. September 8112 1980, France. 141 B. Courtois. Safety, availability and maintenance evaluation forredundant systems. to appear in Digital Processes. 151 EFCIS (Etude et fabrication de circuits lntegres sptiaur), LISA: circuit de communication sur ligne s&e ~5 mdcanisme d’allocation distribu.Ge, Projet da specification, 1980, Grenoble. Enslow. What is a “disrributen’ data processing sysrem?, IEEE Computer Magazine. January 1978.

[61 P.H.

171 A. Guyor Et N. Harris. Motorola MW coprocessor ICOPJ, Computer Systems Laborators/. Dept. of Electrical Engineering, Stanford University, 1980. [81 C.A.R Hoare, Communicating sequential CACiVl. vol. 21, no. 8, August 1978.

processes,

t9l A.L. Hopkins, T. Basil Smith, 111, J.H. Lala, FTMP_ A highly reliable fau/t-tolerant multiprocessor for aircraft, Proceedings cf the IEEE, vaI. E6, no. 10, Dcrober 1978.

L101 R.

Levin, Program sfrucfures for exceptional condition hand/&g, Ph.D. thesis, Carnegie Metlon University, 1977.

[I 11 M. Marinescu. LISA: a communication mechanism for bcal networks, 1st Conkrence on distributed computing systems, Huntsville, October 1979, U.S.A. 1721 M. Marinescu, A primirive distributed mut&e/ exclusion mechanism, IMAG Report no. RR 153, February 1979 (Submitted for publication). 1131 S.R. McConnel, D.P. Siewiorek. f&M. Tsao, The measuremenf and analysis of rrensierrt emvs in digital

121

-