- Email: [email protected]
A data acquisition and experimental control system for the opal experiment at the CERN LEP collider
Nuclear Physics B (Proc . Suppl.) 23A (1991) 395-401 North-Holland
395
A DATA ACQUISITION AND EXPERIMENTAL CONTROL SYSTEM FOR THE OPAL EXPERIMENT AT THE CERN LEP COLLIDER S. tii10TTON C;I Id.N, ;urol)ran Or~aniza(ion fnr Nuclrar ResearcJl, 1211 C:enew 23, .Srrilzerland
`l'hr Ol'A1, data acquisition (1)A(L) system is a distril)uted rnulli-processor environment for the collection, prucc~ssíng and data-Oow rnana.grment of the signals generated by l~t independent subdetectors tvhosP pllrpc~se is to record e+e - interartiuns near and shove the 7.° mass . 'The diversity of detection techniques is reflected l)y the wide range of iront-end read-out hardware ilnplementecl using t~r CAIIIAC, ('ASTRl1S, and ~'111~, standards . To facilitate the management of the data from the Snl)det .ectl)IS by utilising standard software thrunghcnlt the s`stern, a standard architecture was impclsrd at the Local S,yst.rlrl Crate (T,SC) a .nd above thereby- providing a. unifi)rm interface with the central O(':1T . I)A(Z chain . each T.SC cclrlsrsts of a VA'11'~ cra.t r runtaíning a. standstel arrangement of a)mrnercial alld p11rpOSr-1)nllt rlrmlules InCIndÍng 68Qtm renes ( ;I'U's running the OS-9 operating S~St~elrl . 'Triggering a.nd data-Qow beyond the f .SC is monogeel hf similar ~'A1T: crate and bulk (lala transfer fmrn the detectc)r to tbr Ol'A1, On-line ~'At 8700 uses an ul)tical fibre link. 'l'hr raw data are recorded conto IRNI 3480 cartridges frc)m the VAX fur subsequent c)íP-line processing.
1
Introduction
`I'he 01'A1. cxperimrnt at LI ;1' [1.] is a grneral pure+c- inl,eract,ic)ns lx)sc apparatus for tlic dct .cctic)n of nrar and ahovc thc 7,° mass . 'l'he app .~rat,us ccnrsists uf I 1 indc " rcndent, suhdet,rcl,ors clesigncd I,o mea.surc sl)rci(ic clrarac(cristics c)f t,hrsc intrractians . 'hhis rcpc)rt . dcscrihcs t.hr OPAI, 1)A(Z chain whosr funr.t.iun is I,o rrad c)ul, thc olat.a fmrn rarh of tlle suhclcl,cctors, to mcrge a .ll thc data corrrsT)unding t,o cme c+c, - inl,rractiun and t.c) rrcord i,he cc)rrlplcrtc cvrnt for svhsc"glrrllt, uÍT linr recunsl,rucl,ion . `i'hr rrpurt concent.ratcs c)n Lllc crrrl,ral rorrllx)nrnl.s uf Lhr appa .ra.tus I)rgirlllirlg al. I,llr sul)dcl,rcl,wrs' hc)cal S}- sl.c m (~ra~~s (I,S("s~ whcrr part.ia.l cvcnl .s .rrc accluirrd and ending at ihe ()l'Al . On~_line VAX 8700 where the cornplete cvenl,a arr recordccl . As a . nlca.slrrr cvf Ihe I)crfi)rrrlancc which rnust I)c " acvliicvcd 1)y thc I)A(,~ sysl,crn onc s)rulld nol,c (,ha.t. c+w __, lradrc)n intcrncl,iuns at. thr cxl)c"cl,col ratc clf t,llc I~I';l' design larninusil,}~ c)f 1 .(i x 1(T'I~cm-ZS " -~ is alu)ul . (T .511z . 'l'hr avrragr sizc elf hadrelvic E`vcnl,s is ahc)11t 200k1i wliilc" rlmn-liaclrc)nic ivlrracf.ic)ns and I)tlclcgre)Irrel rvrrlts e'clnl.ril>ul,r 711)cllrl, 701cR I)cr rurnl, . l." clrcovs .11 . 'I hc rsll,io clf Ilaclrclnic Lcl ncln-hadrclic Ille~ I)cmk is al)cwrl, 1 :2 flrrrrfi)rc Lllr cysl,rln rrlllsl, I)r crll)rll)Ir clf mlslrlirlirlg rlvrrrlgr eli11R1 ralcw mf a5(1 k R a- v rlsslrrrring rll)11111 rr Iln<'kf;rcllrrrd rvrnls lu~r hrldrc)uic'
intcractiem . The system must also pace s1lÍíicirnt bu(frring capacil}- to accommodate randurn fluctuaLiuns in the trigger rats. I1.eliability, ease of maíntcnance, ease of operation, c(fec'tive rnunit.c)rirlg and g()ucl i)ookkrcping are also important cunsidrralions fc)r a. cc)rnl)lrs 1)~1(,~ system and (hese problems hac-e Deen addressed in the current irnplernerltatic)n of Ilrr ()l'A(, 1)AC,Z design . Sectic)n 2 describes the design criteria in more detail and the irnplernentaf ion is discussed in Scctic)n 3 followed l)y concluding remarks in Section ~1 .
2
Design Criteria
ll. i s apl)r~)prial.c (,u sunrrYrarisr scnne uf thc grneral :rquirrrnrnts c)f a T),~(,~ syst.rrr) in which pcrfi)rrnance is I,IIr l)rirllary consiclrra .t .iun . (,c)cxl prrfc)rrrlancr ulrans t .llal. I,l)c sysl .rnr is calrlblc uf running rclial)ly for Ix"riuds elf rnarly hours sustaining L(IC (lala-flcm- rat.r nrcc,cwry Lc) rccurd all t .he inl,cr.rct .ic)ns uf inl,crcst vrith rninirnal dcad-lirnc and 1\'1tE1 lrr)rrlri)d1 mutsidc inl.crvcnf .iwrl 1)y t.llr e)prrat,clr . 'l'licrr arc nlany cc)nlril)11lclry Glrt,c)rs lo Ilie, ylc)el lx" rfilrrllancr of thc sysl.e" )n uf wlvicll liigll clal ;r-(lcnr ralrs .Inol slrrrll rc"acl out dcadl.irnr .Irc" c)rlly Lwv) . ('Irarly Illr llardwarr and Sc)fl .wam rlllrsl lu~ rrlial)Ir lort irl addil ic)rl 1llr rrrclr rrl)c~rl ing . clalrl-(Imv rlrclnilwrirll; .Ind rc " c'cwrry I)rwcc" clvrrs rllusl I)r w(licirrll, tn rrlal)Ir flrr cll)rrlll,llr wit lil I Ir clr rlcl dr-
0020=taf132/O1/~0 :1 .fr0 ~ 1fl01 ~= i~lacvic~r Scicnct" I'ulrlierhrre I1 .V . (North-Ilnlland)
II
3~
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
tailed krrawlcdge of the 1)A(,Z system tm recover from t.hc occasional inevitable errors. The work (" f designing and cuuslructirrg the rea(lout .-tntl front!-ertd cleci.runics fur the indepencíent sal)dclecaurs in a large c(~Ilabora.tive project, is tti=picallj' (listrilntte(l over several, often remote, insl,it.utes. In or(ler to fa('.tlrtat .e this stage in t1rC cminntlSSlunlrrg mf the (~fAl. det(~ctur, an early (lccision was taken to allow each subdet.ccaur t.o uce whatever technology ~ti'as appropriate t.o their needs fur LTre front-end elect.rmnics . 'l'he resulting (lirersit.y of detection techniques an(1 mad-out, hardware !!tent lrrrplleS tire need for smrne standar(lisat,iorr of i.he interface heiwcen the subdetecteer read-out. artd the cenlrl-1 T)A(ti chain in order t.o facilit.at.e t,hr merging of t.lre partial events from tire sttlxleterturs within i.hc central C)1'I\~. DAC,Z SyStetra and to preserve I,he gcrreral rcquircrnents outlined abcwe. ~\aü.lr such a. srherne it, becomes possible to rrse the same data.-(low management. software throughout the sysi.errr witlt only minor ntuclificat,iuns . 1n addition, central soft~~'arc services can be easily nraintaincd aaul distributed . St.andardisalimn is also required ín ut.her areas, to a dist.riT)uled, nrulti- proccswm system like the Ol':\I, vA(~ system a reliable conrrnunicat-ion nel.work is re gtrired t.u interconne(a the various processors. 'his is, of course, mf fundarnenl.al imtwrtanc:e t.o the cxperirnent since event. transmission, conl,rol rrressages, error reporting, rncmit,mring an(1 st'at'us infi)rrnal ivm all del)end mn it .. 'Che 1)est. use of rrranpmwcr is achieved by rrrinirnising the nlrntl)Ir of nel.wmrk prmtmcmls sm that resmurces can he cuncenlrat .r-cl un a. singly. starn(lard which pervades I,he wllmle sysl.enl . \\'hc"I,her I,his is t)11SSIt)Ie Irl 1)raclice, llcm~evc"r, clel)erlds mn the har(1ware in use and il.s sul)1)orl. fmr any given net,wc~rk 1)rcvl.or-ul . ()!.her
Imple
eníaíion
"l'his scclimn dc~scrilu~s I,he ac'1.11a1 i1111)lc " rrlental,imn wf t,llr (>l':\I~ I)r\(",Z Sysl.cnt srs il, cxisl,ec) fwr I,llc I~l!(1 rl1n . Aftc~r discllasing sc!1r1c uf t.l1e illfr' .rel.rocf,rrr'(t, i11cllrclirrg rlc~fvvcfrkini; srtld dalst~~flwv sufl.warc, eftcll mf I,hc princil)al (worywarols wf Llle c'c~rllrsrl I)A(,~ 4yalcrll will I)c cliarllsaccl wil.ll irlclicrrl,c)rs nf 111c lu~I'firrlrlancc
obtained so far . T3lfore doing so, it is hclpfirl tm give an cn-ervíew of the data-flm to give a l)el,ter trndersl.anding of huw Lach component, ills into the s}"st,eln .
3.1
roerview
Aftrr each c+(~ - inrnch-crossing, t.hc signals gcnera.tcd irr a subset of the ()t'AG sub(lct.erturs are currlT)ined in the ()l'Al~ rentra.l trigger systerrr where 1)rograrnnrrblc logrc IS trse(1 to decide which cvcnts are to be read out . If the event is to be read out a. !,rigger signal is broadcast t.o the Sllbdete.(aUTS rISr11g a. d(~(lic:ated bus. T~url.her triggers are inhibited until a.11 suh(lelect,mrs have read out the (vent into local buffers . if the (vent. is réjer-te(1, a reset signal is broa(l(:ast to allow the subdeter.tors l,o reset their hardware befurc the next bunch rrmssing. 'I'lle single level trigger reduced the (vent, rate from the -l5 kllz hunch-crossing frequency t,o typic:a.lly 2112 under nurma .l 1)canr condit,iuns. ()n rcreiving a f.riggrr signal, the data. frc)rr1 the sul)delectors ace digitised and read out using a varict,y of front-end elecl,ronics irni)Icrnented using the (~AR~TAC, rns"r~uS and \'loll ; st.a.nda.rds. T'he'y are then transferred via an appropriate ínterfa.cc to the so-called I,ocal SS°stern Crate (T,S(') whc"-re I,hetir are t)rmccGSed . 1'rorn í111S stage mT1R'ar(1S the (laia-flog is essentially the same for all sul)(ictrctors ; a Silua.t,iun made pmssil)le by i,hc use of a. sl,anclarcl c()Irfigurat,ivn of hardware within the T~S(~ whirl! permit's I,lre use of standard data-flow rna .nagernr-nt sc)fl.ware (I,he "Skeleton") Lhrmnghout tltc system . In sc~rne cases t,hr. gecnrlet,ry of a sul)dcl,ectur is such that, il, is ccvlr~e.r.icstt I,m read it. c)ut, in Lwu halves. In this case Lhc" Gul)def,rcl,mr usually has Ivvm l .S(~'s Iarh (ledicat.ed I,u the react-out, of urle half wf the snl)dcl,er.tc>r. ()ne cif I,hc I~SC's is dcsigtlat,ed the "mast,er" and is responsibl~" fc~r I,he rncrging of Lhc data frcnn the t,wm halves and fi)r file I,ransfer of t,i1c clata into t he cent.rrl 1)A(ß chain . '('11e cornl)lct,e ~vent,, ('.mT151SI,lllg c-)f 1,11(" data. from each of the ()f'~\T . sul)detecl,urs, is hull! in I,i1e T;vent Rtrilcler (I~;VF3) frotrl rotten, it, is proc~ssed 1)y I,hc fílter crate whir.!! prrfmrrrrs Irrl,tern recugrlit,ic)11 cm 1,111 evrltt,s ill a 1111r11 .í-1)rmcessmr systcrrt in mrdcr I,u classify I,llerll and ol)I,iwnallti~ rc;jecl, c .r presrale 1).rckgrmnrld events . I;vcrrl,s are transferr~d frc1111 l,hc I~then c'ral,r Lcr the 'l'y) cral,c 1lairlg aí1 ol)t,ic}tl fibre while 1,1ík cvcnl. I,ra11sfcr ltrmlcu'crl Irel.wecn t Ikon crrrlcs is esl,Ftl)lishcd I)y tlrc t .ralrl!:111iayimrl srlrcl src'knwvlr(lgurrrcrrl. of cvcnl . drscril)-
S. Wotton/A data acquisition and experimental control system for tire OPAL experiment I,ors over Tahernet using the 7~CP/11' protocol . 7'he 'Top cra.t,e prrfiwms Borne rcfcrrnla.tting of the raw data. before injecting theta into the VAX 8700 using a fast, parallel, VMT~ to VAX link. The VAX is responsible for file recording and optional streaming of the data. onto disk err cartridge . Fssentia .lly all of the data, flow from the LSC's up to the ~'AX is implemented using the VMß standard for the llardwa.re a.nd using the US-9 operating system with i()11;T'IIAIV [2], (' and assembler fur I,he software.
3.2
Networking
A~Tost of the int.cr-system communication in Ui'A1, is based on Tahernet . I)I~CNr'I' is the principal protocol between dE~VICCS Wlt,lllrl the Un-line VAX cluster while VO'I'S is usrd under both VMS and US-9 to interconnect the VAX a.nd VA4I: operating systems . VU'i~~S IS Irsed rnairlly for the transmission of control a.nd error nlessa.ges using the CA"I'S [3] protocol. OS9N r'T' is used for some of the communication between the VM ~ OS-9 systems . In addition, TCP/IP pla tys an important rôle in the data-flow protocol as well a.s supporting a.n additional general purpose network service between VMS and VME. The proliferation of networking protocols within UPAL is a. fa.r from icíea.l situation since it places Borne strain on the resources available for Lhc rna .inte»a.ncr of the systrm. Neverthelrss the situation was, to sulnc extent, una.voida.ble if the desired flrnctionalit,,y was to be achieved.
3.3
Skeleton
The UI'AT~ "Skeleton" is a. strife of software whirls runs under t,hr ()S-9 operating system providing file data.-(lcrw ma.na.gement, servicrs for the subdetectors . The main services provided are buffer management a.nd control c>f the trigger hardware in the T.SC . The data lnllfer is organised as one or more pipe'irles or "streams" into whicll evenl,s from the front,end are injected by a. "Irroducer" Irrucess . 1`he events pass f,hrough srvera,l srquentia.i "sta.ges" where data forrnat .ting, longing and preprocessing arc perfcrrrned by "nlcrdif, ; ::Is nonsurner" processes Irefcrre being extrrlctc(1 froar I,lar: firra.l stage soy bile l~;cart i3uildcr. Addil,icrnal "rlc.rtl-~rncrdifying ccrnsarrler" Irr(1(:CSSC'S a.re typically used by the subdctec:tcrrs to rrlcnlitor the da.l,a. c;nalil.y. In general dal,a is buffered at rrlch StagC until tiler sulrsrcllrrnl, stage is ready Lo rrceivc il.. 'l'hc nurnIrrr of st,trl;rg, I,hr rrrrrrrcrry aIIcuWrd fur lruffcring find
397
the operations períormcd at each stage are indivídua.lly configurable by the subdetectors but the Corrnat of the data a.t the final stage must be ín tllc standard UPAT. íorma.t (7.I:BItA [.1]) .
3.4
Trigger
The UPAT, trigger system [`r] combines trigger input signals which it rrceives between l fps and l~l fps after each bunch-crossing from 6 of the 14 subdetectors to derive a global trigger decision 7 fcs before the next bunch-crossing . The time between bunch-crossings at hI~P is 22.2 fcs . 'The decision ís made using progra.mrna.ble logic ~rhich provides a large degree of flexibility enabling the trigger logic to be caçíly adapted to changing run conditions . 'l'he triger interrupt or reset signal is generated in a purpose-built module called the Global `Trigger Unit (UTU) K"hírh then distributes these, and other, synchrunisat.iorl signals visa a dedicated bus to one or nrnre T.ocal Trigger Units (L'Tti's) for each subdetectur . 'I`he T.TU then generates the interrupt winch triggers read-out or reset of the corr~pondi:ng detector . In addition to the special modules required to generate the trigger signals, the VME Trigger crate contains the same modules that are found in the standard T .ocal System Cra .tc (see below) since the trigger is read out like all the other subdetectors .
3.5
Local System Crate
'The sl,àndard Local System Crate (LSC) comprises the following hardware : ® V M1 i~, Crate ® last intelligent Controller (I'1C 8230 [6]) ® fa,hcrnet, interface (T.Ti.'T) ® 4 I~IR dual-ported memory (I)PA~18241 [6]) ® VMF Intercunncct Tort (VIP [fi]) ® T~oca.l 'Trigger Unlit (h`I'U) The IIC is l,hc processor which controls the da.ta.ílw+' in tem sulrdet,rctor . It runs the US-~ bulTcr manager tasks which inject the data from tbr front-end, perfcrrrn optional zero suppression or pedestal subtr.~c.t.icrrl, crealtc 1.hc 7,I;RIi.A data. structure and tra.nsfcr Lllr went to the Event Ruildo . 'I'hr I)IR~i Irroxides a rrlull,í-rvrnl, hlllfrr which is accessible. berth try tbr
S. Wotton/A data acquisition and experimental control system for the (.aP9T~ experiment
398
fI(; and t.hr 1)y t,hc fvrnt Ruildrr over the ViP link . The 1~1~1' connrc-ts the LS(' to a local area network arrd iltr 1:1'U synclrn)nises the read-uuf. of the subdetrct.ur with t.hr crnt.rai t rigger system . 'I'hr LS(' may, of roars(?, coni .ain ac}clit,iona.l cont1)ortrnts as rrcluire(1 1)y the indivi(lual sul)detectc~rs but, the interface to the central UA(~ system is the same fur all of i.hent. 'l'Ite rrtaxirrnrrr) susiain~)ble !.rigger rate ancí the dead-tirrtr fur each snl)drt,rr.l .c)r is typically limited by the read-c)trt. hardware, t.hr ever!!. si r and t,hc amount of Cl'U Dower availal)Ir fur (lata preprocessing and therefore rarirs sigrriGcantly between the sul)cíetecturs . 'i~lrr standard funct.iwns of the I .SC' introduce nu significant clra(1-t.itrtr unless tl)"" ntrnrory allocate(} fi)r buffering is inadcduatr . During the 1990 run, t}'pical dead-t,irnrs of 5 50 rns were recorded leading t.o an arrragr dead-t.irne of al)out 30 ms fc)r a 21Iz event rats . 'l'ypically the T,S(' has surplus CI'U power available (luring data taking vrhich can be use(1 for rrlcmitoring of the (lat.a (lualit.y, trlortituring ancí ront.rul of the sul)(letrctur hardware and ut,hrr rrtiscellaneuus functions. While not in use for dal.a taking the 1.SC ca.n be usecf fur pncgrarn (lrveloplrtrnt . and local test or calibration runs . SIr1('r rrlüll\' of Lhc hS< :'s du not use !weal disk st .oragr fmr i)rmgrarn and clat.a Ples, they all have access t.ci so-cancel "nrtvvwrk (licks" which are a virtual disk service running un a V.1~ workst.at.ion . 'l'{lis central (ilr Dace facilit.al.es f.11r lrtckup, tnainl,enance and dist.ril)ul,iun wf sl,andard suft,ware. 3 .fi
Eve>
Builder Filter
Matrix
'1~11r I;verlt, fBlildrr rilt.rr A1atri~ c'cvlsisl,s of tvvo V~11 ; crat.es cc)nt.aining lhe following rrx)dulrs: ® !;vent. Ruilclrr crate - 2x 1~ 1( ; 8230 ._ 5 x d' 11' 82 19 - 1,13'l' l'i~llc" rnrl inlrrfac'r - -1 x I)f'A1 821 I ® (~ ill,rr crnl,r - 1: 1( ; ß230 -1 x Fl(', ß231 ~. 111
v nl l'; :I :In-
I
-~ I,li'l' l';Lllc"rllc "I, ir11 .c " rf:)c'r
- VßI( 8212 rnernot'y-mapped link - 2xT)l'~'1 8211 - ~'lT' 8219 - OUI. 8112 'l'hr function (>f the T~.vent. Builder (I;V13) is to rtrrrge the partial events from each sub(let.ert,or bec+c_ intcract.iun . 'l'he rvenl.s a.re longing t.() a given buffered locally in the sulxlrt.ecaurs' rnenu)ry rrrodules while queues of event descriptors are maintairrecl ín the 1:vent Builder . ~Vhen all the descrvpt.ors fur a. particular event have 1)ern received, the T:vent Builder can calculate the size of the event. and allocate bu(Fer space for it in f.he I;Vß buffers. 'These buffers use the ~1 dual-purte(1 rrlelrlory rn0(1111CS UrgaRrse(1 in colunuls . The sul)-events which corrlprisr a complete event are t.hrn t.ransferrrd from the subdet,ect.urs using ~1 of the ~'II' links in parallel info a single buffer cultrrnn and the l;ß'ß ackncnvledges receipt of the event so that the snbdrtect.ors can release the space a.llocal.rd to 1.he event in !,heir buffers. ()ncc the complrt.e even! is in memory a.nd has been 1)ruperly forrna .tted, the I;~' R signals to the i~ ilt,rr crate that iv . can l)rgin processing 1.he ever!! .. The runt remains in the 1;Vß final-ported rrlemory during 1~ iltrr processing and the 1~ iltrr accesses it, using t.hr fill.(! VIf' (illk. ~~%hen 11ík event, has been t)roc( " SS(?(l, the rllt,er signals Lhe 1;~'R and, if t.hc even!. leas nc)t been flagged as uninteresl .ing I)y i,he fill .rr, 1.11e even!, is t.ransferrrd to I,he 'l'op Cral,e using t,lle ()pt.ical T)alat I~ink ()1)1 . 8112 driver nu)dt11r . The int.rlligrnt, l';thern(" I, controller module (A1~%1~~1T~ 330-1) sul)lu)rl,s 1.11e 'i'('1'~Il' 1)rc)tucc)1 used during ihc t.ransfrr of clat,a I,c) t,lle '('wl) Crat,r (see beluv~) . 'l~hc" evellt~s arc nut necessarily transferred t,u 1,11e l';~'H in sequential trigger number c)rdrr sin(-.e one of the ()PAT. sul)del .ecturs pel'fUrrr7S SUl)Statllt.lal (lata. 1)rocessirlg un 1,11(" evenl,s, dlstrll)utrng 1,he event, allalyis c)vrr 20 (;I'l1's. 'l'hr order il1 which t.he event analysis corlll)Icl,rs (lepends c)n I:he size uf l.he event, and cannc~t,, thrrcfc)rr, I)e gllaranl.ced t,o 1)e srduent,ial . 'l'11is irr11)lirs tllal,
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
Filter Processing
3 .7
rra.rrre suggests, the rilter was conceived as a. means of selectively rejecting e~°ruts which are nut of interest thereroy reducing the number of cartridges required to store the ra,w data and reducing the required ba.nelwidtlr of the curnhcwents in the DA(,~ chain beyond it,. In practice "ac;tive filtering" was only ena.hred fur the last part, of I,hr 1990 run since the trigger system provided adequate rejertiotr of baa;kground events . 'l'he I~filter was I,)rerrfure used primarily Fur event classification and rnunil,vering . Many quantities of Frhy-sira.i interest arc hist.~~grammecí by the rilter including statistics on the types of events which are being recorded . fn additiven Llre 1~fitter supports a colour, un-line event display mhich has proven invaluable for the fast diagnosis of a. wide range of detector or trigger problems. 'l'his Fvenf, display program runs on an Apple Macintosh 11 rnicrc~currrlnrl,er which selectivel}accesses the ec"ent data in the filter crate using a. memory-mapped link (VRIi. 8212). The hill,er processing introduces no significant cíeadtimeinto the system provided there is adequate buffering a.nd CPl1 power . 'I~his situation ca.n he realised lay the addition of more memory and CPU's as required. AS ItS
3.8
Optical Data Link
3.9
Top Crate
'l'he I~ filter crate is situated 100 m underground, close to the sulxletect,ors, hui. the" events arc rrcorcíed from the VAX 8700 ven the surface . 'fhe events arc transferred frurn the ffilter crr,tr. Lw a VA9 r crate on the srrrfa .cr (the 'lb1 Craf.e) using a. duplex opl,ica.1 fibre link. 7'he fibres are c)rivc>~r a .t each c"nd l,y a. Cr,S ()r)í.81 ~12 rrrudrrle which cvenrrc "ct,s to f.he I)MA of a. C1;S rIC 8230. 1)at.a rates of > 211~iRs - ~ ca.n be achieved . 'fhe event, transfer protocol rc~duires that the event, size and other characteristics a.re known before the event is transferred . r(u achieve this, I,he protocol is established between the ODI . modules by the transmission and a.cknowledgernent of event descrir~t,urs using 'fC ;i'/ti', a. prvel,ocol suplx~rted by the M V M f; 330- I mucíules in the rüf,er and 'fop crates.
'l'he VA~1 r: 'l'up (;rake" contains I,lrc" fvellowing rrrvedules: o FIC 8230
® c)~r, 81-12 ® I,lfl' I;I,hrrnc"I, inlc"rfrrcr
39~
® IKON 10089 URII-fit' driver 0
2xUI'11~i 8241
o h4VR~iE 330-t 'hhe purpose of the 'fop Crate is to receive events via t.hc Optical Data Link using the kond of the hair of CLS ODr,$I 12 modules and to transfer them to the VAX 87()0 over a fast parallel lime using tire 1)I?C I)Il.l 1-VV protocol and CUPACK software. 'fhe surplus C1'U ín tl: ..'!c~ Crate was used to reformat the cíata into the reprc~senta-lion {IFI;F, VAX etc. required fur the subsequent event reconstruction and analysis. Ruth formats can be supported sínrultaneuusïy by the 7bp Crate sine tire events cuurcí be transferred to the VAX 8700 using two independent hardware channels of the CUPACK link and injected into two independent buffer manager streams. 'fhe Tup Crate may optionally distribute events t.o a. processor farm for event reconstruction before the events are passed to the VAX although this possíbility was not routinely used during the 1989 and 1990 running periods .
3 .10
VAX 8700
The VAX 8700 receives events from the Top Crate using a DRR-32V' module connected to the BI bus of the ~'AX . ~'he ec"ents are then recorded on íB11í 3180 cartridges using a dual cartridge drive with starkers under the cunt.rol of "Recorder" programs each of whine controls a. single volume (cartridge ur disk file) on the mass storage device . Events are dist,ribnted t,o the Recorders by the Alodel Buffer A~ia.nager [7] v+-bleb supports multiple data streams and which receives events frurn a "Producer" prugrarrr n"hose purpose is to read events from the Dli.B32-W and inject them info one ur more butler manager streams. The streams a.rc software configurable and arc therefore easily adaptable to changing requirements . lí ; addition to event recording, the VAX 8700 and on-litre VAX cluster support many of the other services required by the cía.ta acquisition system including: e Skaters di~piays o I~.rror message formatting and rveut.ing o Csrlihrat.iwn cíatabasc rnanagetnrnt 0
11-rcrnit .urirrg
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
4~
® I~un contrul ® ßookkerping ® (~S-9 net,work cíisk srrver
3.11
R.un Control
li.un control is perfar~ncd on tire ~~i\t 8700 irslrrg a knuwlecígc-based s)-stcrn writt.rn in Prolog and using ul)ject:oriented prugratruating t.cchnigatcs [8]. 1`he variuals phases of a run and the t.ransit,iuns between there are represented by finite Si.ate i\iaehines (hSh~i's) organised in a hierarchical struci.trre cnrrespanding to the corrapurrrnts of file (íat.a a('glllst .lt.l()n sj'Strrra . (",c)rrrmaai(ls assor.iai,ed with transitions between i.he slates of an i~S1~I rcprrsctrt ing a given rornl)onrnt tier sent i,u that component as text sl,rings using t 1te C~fI'S prot.ucol. ~~hc respwnsc frmrn i.he curnl)onent signalling that tlae trarts11,111r1 11äS sltc'crSSfllll}' cc~rtrplet.ed rs sCTtt, 1)ack to the t';\ ~ using t,hr same protacml . In this way, ihr rein control cngiuc maintains a kncnrleclge of the cilrrcrit state of i,hr rrrn artcí call initiate transü,ioras to any oi,hcr allowed state. Aut,ornatic acrd operator assistr(1 error recovery procecíurcs help to reduce the level of expertise required by the operator to run the experiment. ~1'he operator interacts with the run control cngilir axing menus, panels and a. graphical rcprcscnt.atiwn ~f i.he 1~'S~t states cvt ti VAX Workstation . 'phis bureau inl,crface is based un Dcc~Vincíc)ws graphics .anti mindwving prirnitivc "s.
3 .12
flow Control
( ;uril,rul and rnwriil.wring cvf i.lrc snluíctccaurs' high and Iwv vult.agc puwcr sirl)hlics aaid ml.hcr aaixilia.ry systcrns arc perfc)rrned l)y llrc ()l'A1, Slwv ('.unl.ruls Systcrn [9]. 'l'his ccmsist,s uf ac"ven VI~11~; crates which rec:eive and rnonit,ur signals, but,h ana.lmgur a.nd dig_ it.al, frorn scnsurs clislril)ut.c~d armutid I,Ire a1)pa.ra.tirs. Cuntrul and statrrs corrrrriands ctiri I)r scnt, tc) i.he snlodef,ectwrs cvr uther Slmv (',c~rrt.rc)ls sr,at,icors via ()S-J pi1)rs tis sirnple I,exl, strings . A hurrr,an ínt.c" rface rr5ing thc" A1arínl,ush if,y1)c "rcarcl sc~fl,ware htis I)cen dcvc"Iu1)ed to Gacilíttitc 1.Itc nuiriíl,mríng sincl cunl,rol uf Lhc til)Irirtil,iis ti.nd a sirnl)Ir, mrrrm-drivert inl,crfaacr is tils,m availtible . ln tidclil.iun, tin inf.c" rface (,w Llrc rtin runt,rml c`ilglflr exlst.S R'lllc'.h, íalt,ll~)Ilgll tIWI~ yrl irrrl)Irrttenl,cd, vill allc)w I,Ite sill)clrl,rclvrs' vcvlt .agr Sir1)I)lic "s I,u I)c orrIc~rrrtil .ictill}' sti~nc'hrnnisccl wil li plysims rrrrts . 'l'Irc Slmv (',corl,ruls 5ysl,ctn crrn I)c rrscd (,ca l;rrrrrralc w}trnings aitrd rrlrrrrrr~r fnr aigrrtils Ilrmt. rxccccl I>rcac"t Llrrctihcrlcls,
Such ala.rtrts and wtirnlngs earl also trigger a.uturnatic: tict,icms in order to protect, the apparatus from damage and i,o 1)revent accidents . COl1Ci>1151®1~S The ()l'A1. da.i,a. acquisition s;~stem ha.s operated reliably over an extended period including the 1989 and 1990 runs in which alnu)st l50 t)00 e~c- -a hadrun int,eractiuns were recorded at appruxitnal,cly one quarter of t,lte I.1?1' design lurninusit,y. Dead-t,irnes of less than 10% were achicred a.t. trigger rates c~f about 2112. 'I~he neutral I)A(.~ system is c:urrently being upgraded t0 sllst~llrt the higher da1,a-(low ra.t,es ex1)ected during 1991 and beyond . Li:Iis exprctc"(í t .o appraa.ch cvr exceed the design lurrainusity and t,irere is a. pussibilil,y that, the nurnl)er of e+~ - lnlnrhes will I)e inc:rrased . 'hhe current filter sysi,ern leas 1nsulficient processing i)cnver t,u sustain these higher rates and is I)ring replaced by an Apollo llN 10000 1)rucessing faun while t.lie central trigger sysl,rrn will benefit llama the additicm of fast per-trigger logic f .cl aceornudate the reduced t,irYae bel,ween bunch-crossings when 8-1)unch 1~f~1' upcrat,iun is irnplcrraentcd .
~.eferences 'l'he ()fA1, Cc)llaburtii,ion, 'I~he UI'Ai, T)ri,eci.or at T,1~;1', ('l:ß,lV f'T'l; 90-I I-1, Suhrriil,tcd t,c~ Nucl. Instr. a.ncl it,Icthods, Airgust 19S)0 . [2] 11 . vcln dcr Schtrritt, li:l~(~~68K 1i,ea.l 'l'inic I~mrI,rtin 77 fur 68K f'rucessnrs, i'lrysiktilisclrcs lrrsl,itul, llnivc".rsit.at, Ileidelherg, i~1ay 1989. [:3] 'l'. .1. llrrrrcrs-I~ec, 'l'he CATS 'franspmrl . Service ('tilling Scduencc, i'rugrarn Library', ('T';ß.N, 1)1) Division, 121 I (ácnevra., Switzerland, R . Rrurr, it-i . (i(1t,1S5E'rIS, ,1 . 7,011, 7,I~,R1i.A lyntirriic I)ti.ta Sl,ruc;tirre ancl 11~Ierrrury lllanager, 1'rmgr.~rn l~il)rary, 1)ll ~ivision, ('l?Ii.N, 1211 (ärrirv .a, Swit.zcrland .
[ rr] 'l'he ()l'AT~ 'l'riggcr Sysl,crrt, Nircl . Irrsl.r. ancl (1Tct.limd5 .
1.0
lac sul)rrril.l,ed t,o
[fi] ( ;I';5, Iicntt.c du l'unl-Fiirl .in 70, 121 :3 l'c"I,il,-1,atrcy°, (vcrrevra, Swit.zcrland . ('. Fiwisstii, l'. ~'titrclc V'd'vrr, 'l'irr 111()I)i;l, Fittfü~r ~isrnrigcr (lac"r Wtidr, i'rcigrsirtr I.ilrrsrry, ('i;li.N, i)I) I)iviwiwrr, l'll I (vrrrrvti 23, Wvitrrrlan(I .
S. Wotton /A data acquisition and experimental control system for the OPAL experiment [8] Robin I' . l~liddleton, Conl,rol rrf a. LI;P I;xp~rirnent using a knowledge-Raced System, CNRS, Proceedings of the International Workshop on Soi'tware Engineering, Artificial Intelligence and Expert S,vsf.ems foc iligh l;nergy and Nuclear Physics, r.,yons, I~'rarrce, 19--2~1 A7arch 1990 . [9] A.K . Arnundscn c~t al ., Nucl . lnstr. arul AZethods, A293 (1990) 1 ~15.
403
395
A DATA ACQUISITION AND EXPERIMENTAL CONTROL SYSTEM FOR THE OPAL EXPERIMENT AT THE CERN LEP COLLIDER S. tii10TTON C;I Id.N, ;urol)ran Or~aniza(ion fnr Nuclrar ResearcJl, 1211 C:enew 23, .Srrilzerland
`l'hr Ol'A1, data acquisition (1)A(L) system is a distril)uted rnulli-processor environment for the collection, prucc~ssíng and data-Oow rnana.grment of the signals generated by l~t independent subdetectors tvhosP pllrpc~se is to record e+e - interartiuns near and shove the 7.° mass . 'The diversity of detection techniques is reflected l)y the wide range of iront-end read-out hardware ilnplementecl using t~r CAIIIAC, ('ASTRl1S, and ~'111~, standards . To facilitate the management of the data from the Snl)det .ectl)IS by utilising standard software thrunghcnlt the s`stern, a standard architecture was impclsrd at the Local S,yst.rlrl Crate (T,SC) a .nd above thereby- providing a. unifi)rm interface with the central O(':1T . I)A(Z chain . each T.SC cclrlsrsts of a VA'11'~ cra.t r runtaíning a. standstel arrangement of a)mrnercial alld p11rpOSr-1)nllt rlrmlules InCIndÍng 68Qtm renes ( ;I'U's running the OS-9 operating S~St~elrl . 'Triggering a.nd data-Qow beyond the f .SC is monogeel hf similar ~'A1T: crate and bulk (lala transfer fmrn the detectc)r to tbr Ol'A1, On-line ~'At 8700 uses an ul)tical fibre link. 'l'hr raw data are recorded conto IRNI 3480 cartridges frc)m the VAX fur subsequent c)íP-line processing.
1
Introduction
`I'he 01'A1. cxperimrnt at LI ;1' [1.] is a grneral pure+c- inl,eract,ic)ns lx)sc apparatus for tlic dct .cctic)n of nrar and ahovc thc 7,° mass . 'l'he app .~rat,us ccnrsists uf I 1 indc " rcndent, suhdet,rcl,ors clesigncd I,o mea.surc sl)rci(ic clrarac(cristics c)f t,hrsc intrractians . 'hhis rcpc)rt . dcscrihcs t.hr OPAI, 1)A(Z chain whosr funr.t.iun is I,o rrad c)ul, thc olat.a fmrn rarh of tlle suhclcl,cctors, to mcrge a .ll thc data corrrsT)unding t,o cme c+c, - inl,rractiun and t.c) rrcord i,he cc)rrlplcrtc cvrnt for svhsc"glrrllt, uÍT linr recunsl,rucl,ion . `i'hr rrpurt concent.ratcs c)n Lllc crrrl,ral rorrllx)nrnl.s uf Lhr appa .ra.tus I)rgirlllirlg al. I,llr sul)dcl,rcl,wrs' hc)cal S}- sl.c m (~ra~~s (I,S("s~ whcrr part.ia.l cvcnl .s .rrc accluirrd and ending at ihe ()l'Al . On~_line VAX 8700 where the cornplete cvenl,a arr recordccl . As a . nlca.slrrr cvf Ihe I)crfi)rrrlancc which rnust I)c " acvliicvcd 1)y thc I)A(,~ sysl,crn onc s)rulld nol,c (,ha.t. c+w __, lradrc)n intcrncl,iuns at. thr cxl)c"cl,col ratc clf t,llc I~I';l' design larninusil,}~ c)f 1 .(i x 1(T'I~cm-ZS " -~ is alu)ul . (T .511z . 'l'hr avrragr sizc elf hadrelvic E`vcnl,s is ahc)11t 200k1i wliilc" rlmn-liaclrc)nic ivlrracf.ic)ns and I)tlclcgre)Irrel rvrrlts e'clnl.ril>ul,r 711)cllrl, 701cR I)cr rurnl, . l." clrcovs .11 . 'I hc rsll,io clf Ilaclrclnic Lcl ncln-hadrclic Ille~ I)cmk is al)cwrl, 1 :2 flrrrrfi)rc Lllr cysl,rln rrlllsl, I)r crll)rll)Ir clf mlslrlirlirlg rlvrrrlgr eli11R1 ralcw mf a5(1 k R a- v rlsslrrrring rll)11111 rr Iln<'kf;rcllrrrd rvrnls lu~r hrldrc)uic'
intcractiem . The system must also pace s1lÍíicirnt bu(frring capacil}- to accommodate randurn fluctuaLiuns in the trigger rats. I1.eliability, ease of maíntcnance, ease of operation, c(fec'tive rnunit.c)rirlg and g()ucl i)ookkrcping are also important cunsidrralions fc)r a. cc)rnl)lrs 1)~1(,~ system and (hese problems hac-e Deen addressed in the current irnplernerltatic)n of Ilrr ()l'A(, 1)AC,Z design . Sectic)n 2 describes the design criteria in more detail and the irnplernentaf ion is discussed in Scctic)n 3 followed l)y concluding remarks in Section ~1 .
2
Design Criteria
ll. i s apl)r~)prial.c (,u sunrrYrarisr scnne uf thc grneral :rquirrrnrnts c)f a T),~(,~ syst.rrr) in which pcrfi)rrnance is I,IIr l)rirllary consiclrra .t .iun . (,c)cxl prrfc)rrrlancr ulrans t .llal. I,l)c sysl .rnr is calrlblc uf running rclial)ly for Ix"riuds elf rnarly hours sustaining L(IC (lala-flcm- rat.r nrcc,cwry Lc) rccurd all t .he inl,cr.rct .ic)ns uf inl,crcst vrith rninirnal dcad-lirnc and 1\'1tE1 lrr)rrlri)d1 mutsidc inl.crvcnf .iwrl 1)y t.llr e)prrat,clr . 'l'licrr arc nlany cc)nlril)11lclry Glrt,c)rs lo Ilie, ylc)el lx" rfilrrllancr of thc sysl.e" )n uf wlvicll liigll clal ;r-(lcnr ralrs .Inol slrrrll rc"acl out dcadl.irnr .Irc" c)rlly Lwv) . ('Irarly Illr llardwarr and Sc)fl .wam rlllrsl lu~ rrlial)Ir lort irl addil ic)rl 1llr rrrclr rrl)c~rl ing . clalrl-(Imv rlrclnilwrirll; .Ind rc " c'cwrry I)rwcc" clvrrs rllusl I)r w(licirrll, tn rrlal)Ir flrr cll)rrlll,llr wit lil I Ir clr rlcl dr-
0020=taf132/O1/~0 :1 .fr0 ~ 1fl01 ~= i~lacvic~r Scicnct" I'ulrlierhrre I1 .V . (North-Ilnlland)
II
3~
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
tailed krrawlcdge of the 1)A(,Z system tm recover from t.hc occasional inevitable errors. The work (" f designing and cuuslructirrg the rea(lout .-tntl front!-ertd cleci.runics fur the indepencíent sal)dclecaurs in a large c(~Ilabora.tive project, is tti=picallj' (listrilntte(l over several, often remote, insl,it.utes. In or(ler to fa('.tlrtat .e this stage in t1rC cminntlSSlunlrrg mf the (~fAl. det(~ctur, an early (lccision was taken to allow each subdet.ccaur t.o uce whatever technology ~ti'as appropriate t.o their needs fur LTre front-end elect.rmnics . 'l'he resulting (lirersit.y of detection techniques an(1 mad-out, hardware !!tent lrrrplleS tire need for smrne standar(lisat,iorr of i.he interface heiwcen the subdetecteer read-out. artd the cenlrl-1 T)A(ti chain in order t.o facilit.at.e t,hr merging of t.lre partial events from tire sttlxleterturs within i.hc central C)1'I\~. DAC,Z SyStetra and to preserve I,he gcrreral rcquircrnents outlined abcwe. ~\aü.lr such a. srherne it, becomes possible to rrse the same data.-(low management. software throughout the sysi.errr witlt only minor ntuclificat,iuns . 1n addition, central soft~~'arc services can be easily nraintaincd aaul distributed . St.andardisalimn is also required ín ut.her areas, to a dist.riT)uled, nrulti- proccswm system like the Ol':\I, vA(~ system a reliable conrrnunicat-ion nel.work is re gtrired t.u interconne(a the various processors. 'his is, of course, mf fundarnenl.al imtwrtanc:e t.o the cxperirnent since event. transmission, conl,rol rrressages, error reporting, rncmit,mring an(1 st'at'us infi)rrnal ivm all del)end mn it .. 'Che 1)est. use of rrranpmwcr is achieved by rrrinirnising the nlrntl)Ir of nel.wmrk prmtmcmls sm that resmurces can he cuncenlrat .r-cl un a. singly. starn(lard which pervades I,he wllmle sysl.enl . \\'hc"I,her I,his is t)11SSIt)Ie Irl 1)raclice, llcm~evc"r, clel)erlds mn the har(1ware in use and il.s sul)1)orl. fmr any given net,wc~rk 1)rcvl.or-ul . ()!.her
Imple
eníaíion
"l'his scclimn dc~scrilu~s I,he ac'1.11a1 i1111)lc " rrlental,imn wf t,llr (>l':\I~ I)r\(",Z Sysl.cnt srs il, cxisl,ec) fwr I,llc I~l!(1 rl1n . Aftc~r discllasing sc!1r1c uf t.l1e illfr' .rel.rocf,rrr'(t, i11cllrclirrg rlc~fvvcfrkini; srtld dalst~~flwv sufl.warc, eftcll mf I,hc princil)al (worywarols wf Llle c'c~rllrsrl I)A(,~ 4yalcrll will I)c cliarllsaccl wil.ll irlclicrrl,c)rs nf 111c lu~I'firrlrlancc
obtained so far . T3lfore doing so, it is hclpfirl tm give an cn-ervíew of the data-flm to give a l)el,ter trndersl.anding of huw Lach component, ills into the s}"st,eln .
3.1
roerview
Aftrr each c+(~ - inrnch-crossing, t.hc signals gcnera.tcd irr a subset of the ()t'AG sub(lct.erturs are currlT)ined in the ()l'Al~ rentra.l trigger systerrr where 1)rograrnnrrblc logrc IS trse(1 to decide which cvcnts are to be read out . If the event is to be read out a. !,rigger signal is broadcast t.o the Sllbdete.(aUTS rISr11g a. d(~(lic:ated bus. T~url.her triggers are inhibited until a.11 suh(lelect,mrs have read out the (vent into local buffers . if the (vent. is réjer-te(1, a reset signal is broa(l(:ast to allow the subdeter.tors l,o reset their hardware befurc the next bunch rrmssing. 'I'lle single level trigger reduced the (vent, rate from the -l5 kllz hunch-crossing frequency t,o typic:a.lly 2112 under nurma .l 1)canr condit,iuns. ()n rcreiving a f.riggrr signal, the data. frc)rr1 the sul)delectors ace digitised and read out using a varict,y of front-end elecl,ronics irni)Icrnented using the (~AR~TAC, rns"r~uS and \'loll ; st.a.nda.rds. T'he'y are then transferred via an appropriate ínterfa.cc to the so-called I,ocal SS°stern Crate (T,S(') whc"-re I,hetir are t)rmccGSed . 1'rorn í111S stage mT1R'ar(1S the (laia-flog is essentially the same for all sul)(ictrctors ; a Silua.t,iun made pmssil)le by i,hc use of a. sl,anclarcl c()Irfigurat,ivn of hardware within the T~S(~ whirl! permit's I,lre use of standard data-flow rna .nagernr-nt sc)fl.ware (I,he "Skeleton") Lhrmnghout tltc system . In sc~rne cases t,hr. gecnrlet,ry of a sul)dcl,ectur is such that, il, is ccvlr~e.r.icstt I,m read it. c)ut, in Lwu halves. In this case Lhc" Gul)def,rcl,mr usually has Ivvm l .S(~'s Iarh (ledicat.ed I,u the react-out, of urle half wf the snl)dcl,er.tc>r. ()ne cif I,hc I~SC's is dcsigtlat,ed the "mast,er" and is responsibl~" fc~r I,he rncrging of Lhc data frcnn the t,wm halves and fi)r file I,ransfer of t,i1c clata into t he cent.rrl 1)A(ß chain . '('11e cornl)lct,e ~vent,, ('.mT151SI,lllg c-)f 1,11(" data. from each of the ()f'~\T . sul)detecl,urs, is hull! in I,i1e T;vent Rtrilcler (I~;VF3) frotrl rotten, it, is proc~ssed 1)y I,hc fílter crate whir.!! prrfmrrrrs Irrl,tern recugrlit,ic)11 cm 1,111 evrltt,s ill a 1111r11 .í-1)rmcessmr systcrrt in mrdcr I,u classify I,llerll and ol)I,iwnallti~ rc;jecl, c .r presrale 1).rckgrmnrld events . I;vcrrl,s are transferr~d frc1111 l,hc I~then c'ral,r Lcr the 'l'y) cral,c 1lairlg aí1 ol)t,ic}tl fibre while 1,1ík cvcnl. I,ra11sfcr ltrmlcu'crl Irel.wecn t Ikon crrrlcs is esl,Ftl)lishcd I)y tlrc t .ralrl!:111iayimrl srlrcl src'knwvlr(lgurrrcrrl. of cvcnl . drscril)-
S. Wotton/A data acquisition and experimental control system for tire OPAL experiment I,ors over Tahernet using the 7~CP/11' protocol . 7'he 'Top cra.t,e prrfiwms Borne rcfcrrnla.tting of the raw data. before injecting theta into the VAX 8700 using a fast, parallel, VMT~ to VAX link. The VAX is responsible for file recording and optional streaming of the data. onto disk err cartridge . Fssentia .lly all of the data, flow from the LSC's up to the ~'AX is implemented using the VMß standard for the llardwa.re a.nd using the US-9 operating system with i()11;T'IIAIV [2], (' and assembler fur I,he software.
3.2
Networking
A~Tost of the int.cr-system communication in Ui'A1, is based on Tahernet . I)I~CNr'I' is the principal protocol between dE~VICCS Wlt,lllrl the Un-line VAX cluster while VO'I'S is usrd under both VMS and US-9 to interconnect the VAX a.nd VA4I: operating systems . VU'i~~S IS Irsed rnairlly for the transmission of control a.nd error nlessa.ges using the CA"I'S [3] protocol. OS9N r'T' is used for some of the communication between the VM ~ OS-9 systems . In addition, TCP/IP pla tys an important rôle in the data-flow protocol as well a.s supporting a.n additional general purpose network service between VMS and VME. The proliferation of networking protocols within UPAL is a. fa.r from icíea.l situation since it places Borne strain on the resources available for Lhc rna .inte»a.ncr of the systrm. Neverthelrss the situation was, to sulnc extent, una.voida.ble if the desired flrnctionalit,,y was to be achieved.
3.3
Skeleton
The UI'AT~ "Skeleton" is a. strife of software whirls runs under t,hr ()S-9 operating system providing file data.-(lcrw ma.na.gement, servicrs for the subdetectors . The main services provided are buffer management a.nd control c>f the trigger hardware in the T.SC . The data lnllfer is organised as one or more pipe'irles or "streams" into whicll evenl,s from the front,end are injected by a. "Irroducer" Irrucess . 1`he events pass f,hrough srvera,l srquentia.i "sta.ges" where data forrnat .ting, longing and preprocessing arc perfcrrrned by "nlcrdif, ; ::Is nonsurner" processes Irefcrre being extrrlctc(1 froar I,lar: firra.l stage soy bile l~;cart i3uildcr. Addil,icrnal "rlc.rtl-~rncrdifying ccrnsarrler" Irr(1(:CSSC'S a.re typically used by the subdctec:tcrrs to rrlcnlitor the da.l,a. c;nalil.y. In general dal,a is buffered at rrlch StagC until tiler sulrsrcllrrnl, stage is ready Lo rrceivc il.. 'l'hc nurnIrrr of st,trl;rg, I,hr rrrrrrrcrry aIIcuWrd fur lruffcring find
397
the operations períormcd at each stage are indivídua.lly configurable by the subdetectors but the Corrnat of the data a.t the final stage must be ín tllc standard UPAT. íorma.t (7.I:BItA [.1]) .
3.4
Trigger
The UPAT, trigger system [`r] combines trigger input signals which it rrceives between l fps and l~l fps after each bunch-crossing from 6 of the 14 subdetectors to derive a global trigger decision 7 fcs before the next bunch-crossing . The time between bunch-crossings at hI~P is 22.2 fcs . 'The decision ís made using progra.mrna.ble logic ~rhich provides a large degree of flexibility enabling the trigger logic to be caçíly adapted to changing run conditions . 'l'he triger interrupt or reset signal is generated in a purpose-built module called the Global `Trigger Unit (UTU) K"hírh then distributes these, and other, synchrunisat.iorl signals visa a dedicated bus to one or nrnre T.ocal Trigger Units (L'Tti's) for each subdetectur . 'I`he T.TU then generates the interrupt winch triggers read-out or reset of the corr~pondi:ng detector . In addition to the special modules required to generate the trigger signals, the VME Trigger crate contains the same modules that are found in the standard T .ocal System Cra .tc (see below) since the trigger is read out like all the other subdetectors .
3.5
Local System Crate
'The sl,àndard Local System Crate (LSC) comprises the following hardware : ® V M1 i~, Crate ® last intelligent Controller (I'1C 8230 [6]) ® fa,hcrnet, interface (T.Ti.'T) ® 4 I~IR dual-ported memory (I)PA~18241 [6]) ® VMF Intercunncct Tort (VIP [fi]) ® T~oca.l 'Trigger Unlit (h`I'U) The IIC is l,hc processor which controls the da.ta.ílw+' in tem sulrdet,rctor . It runs the US-~ bulTcr manager tasks which inject the data from tbr front-end, perfcrrrn optional zero suppression or pedestal subtr.~c.t.icrrl, crealtc 1.hc 7,I;RIi.A data. structure and tra.nsfcr Lllr went to the Event Ruildo . 'I'hr I)IR~i Irroxides a rrlull,í-rvrnl, hlllfrr which is accessible. berth try tbr
S. Wotton/A data acquisition and experimental control system for the (.aP9T~ experiment
398
fI(; and t.hr 1)y t,hc fvrnt Ruildrr over the ViP link . The 1~1~1' connrc-ts the LS(' to a local area network arrd iltr 1:1'U synclrn)nises the read-uuf. of the subdetrct.ur with t.hr crnt.rai t rigger system . 'I'hr LS(' may, of roars(?, coni .ain ac}clit,iona.l cont1)ortrnts as rrcluire(1 1)y the indivi(lual sul)detectc~rs but, the interface to the central UA(~ system is the same fur all of i.hent. 'l'Ite rrtaxirrnrrr) susiain~)ble !.rigger rate ancí the dead-tirrtr fur each snl)drt,rr.l .c)r is typically limited by the read-c)trt. hardware, t.hr ever!!. si r and t,hc amount of Cl'U Dower availal)Ir fur (lata preprocessing and therefore rarirs sigrriGcantly between the sul)cíetecturs . 'i~lrr standard funct.iwns of the I .SC' introduce nu significant clra(1-t.itrtr unless tl)"" ntrnrory allocate(} fi)r buffering is inadcduatr . During the 1990 run, t}'pical dead-t,irnrs of 5 50 rns were recorded leading t.o an arrragr dead-t.irne of al)out 30 ms fc)r a 21Iz event rats . 'l'ypically the T,S(' has surplus CI'U power available (luring data taking vrhich can be use(1 for rrlcmitoring of the (lat.a (lualit.y, trlortituring ancí ront.rul of the sul)(letrctur hardware and ut,hrr rrtiscellaneuus functions. While not in use for dal.a taking the 1.SC ca.n be usecf fur pncgrarn (lrveloplrtrnt . and local test or calibration runs . SIr1('r rrlüll\' of Lhc hS< :'s du not use !weal disk st .oragr fmr i)rmgrarn and clat.a Ples, they all have access t.ci so-cancel "nrtvvwrk (licks" which are a virtual disk service running un a V.1~ workst.at.ion . 'l'{lis central (ilr Dace facilit.al.es f.11r lrtckup, tnainl,enance and dist.ril)ul,iun wf sl,andard suft,ware. 3 .fi
Eve>
Builder Filter
Matrix
'1~11r I;verlt, fBlildrr rilt.rr A1atri~ c'cvlsisl,s of tvvo V~11 ; crat.es cc)nt.aining lhe following rrx)dulrs: ® !;vent. Ruilclrr crate - 2x 1~ 1( ; 8230 ._ 5 x d' 11' 82 19 - 1,13'l' l'i~llc" rnrl inlrrfac'r - -1 x I)f'A1 821 I ® (~ ill,rr crnl,r - 1: 1( ; ß230 -1 x Fl(', ß231 ~. 111
v nl l'; :I :In-
I
-~ I,li'l' l';Lllc"rllc "I, ir11 .c " rf:)c'r
- VßI( 8212 rnernot'y-mapped link - 2xT)l'~'1 8211 - ~'lT' 8219 - OUI. 8112 'l'hr function (>f the T~.vent. Builder (I;V13) is to rtrrrge the partial events from each sub(let.ert,or bec+c_ intcract.iun . 'l'he rvenl.s a.re longing t.() a given buffered locally in the sulxlrt.ecaurs' rnenu)ry rrrodules while queues of event descriptors are maintairrecl ín the 1:vent Builder . ~Vhen all the descrvpt.ors fur a. particular event have 1)ern received, the T:vent Builder can calculate the size of the event. and allocate bu(Fer space for it in f.he I;Vß buffers. 'These buffers use the ~1 dual-purte(1 rrlelrlory rn0(1111CS UrgaRrse(1 in colunuls . The sul)-events which corrlprisr a complete event are t.hrn t.ransferrrd from the subdet,ect.urs using ~1 of the ~'II' links in parallel info a single buffer cultrrnn and the l;ß'ß ackncnvledges receipt of the event so that the snbdrtect.ors can release the space a.llocal.rd to 1.he event in !,heir buffers. ()ncc the complrt.e even! is in memory a.nd has been 1)ruperly forrna .tted, the I;~' R signals to the i~ ilt,rr crate that iv . can l)rgin processing 1.he ever!! .. The runt remains in the 1;Vß final-ported rrlemory during 1~ iltrr processing and the 1~ iltrr accesses it, using t.hr fill.(! VIf' (illk. ~~%hen 11ík event, has been t)roc( " SS(?(l, the rllt,er signals Lhe 1;~'R and, if t.hc even!. leas nc)t been flagged as uninteresl .ing I)y i,he fill .rr, 1.11e even!, is t.ransferrrd to I,he 'l'op Cral,e using t,lle ()pt.ical T)alat I~ink ()1)1 . 8112 driver nu)dt11r . The int.rlligrnt, l';thern(" I, controller module (A1~%1~~1T~ 330-1) sul)lu)rl,s 1.11e 'i'('1'~Il' 1)rc)tucc)1 used during ihc t.ransfrr of clat,a I,c) t,lle '('wl) Crat,r (see beluv~) . 'l~hc" evellt~s arc nut necessarily transferred t,u 1,11e l';~'H in sequential trigger number c)rdrr sin(-.e one of the ()PAT. sul)del .ecturs pel'fUrrr7S SUl)Statllt.lal (lata. 1)rocessirlg un 1,11(" evenl,s, dlstrll)utrng 1,he event, allalyis c)vrr 20 (;I'l1's. 'l'hr order il1 which t.he event analysis corlll)Icl,rs (lepends c)n I:he size uf l.he event, and cannc~t,, thrrcfc)rr, I)e gllaranl.ced t,o 1)e srduent,ial . 'l'11is irr11)lirs tllal,
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
Filter Processing
3 .7
rra.rrre suggests, the rilter was conceived as a. means of selectively rejecting e~°ruts which are nut of interest thereroy reducing the number of cartridges required to store the ra,w data and reducing the required ba.nelwidtlr of the curnhcwents in the DA(,~ chain beyond it,. In practice "ac;tive filtering" was only ena.hred fur the last part, of I,hr 1990 run since the trigger system provided adequate rejertiotr of baa;kground events . 'l'he I~filter was I,)rerrfure used primarily Fur event classification and rnunil,vering . Many quantities of Frhy-sira.i interest arc hist.~~grammecí by the rilter including statistics on the types of events which are being recorded . fn additiven Llre 1~fitter supports a colour, un-line event display mhich has proven invaluable for the fast diagnosis of a. wide range of detector or trigger problems. 'l'his Fvenf, display program runs on an Apple Macintosh 11 rnicrc~currrlnrl,er which selectivel}accesses the ec"ent data in the filter crate using a. memory-mapped link (VRIi. 8212). The hill,er processing introduces no significant cíeadtimeinto the system provided there is adequate buffering a.nd CPl1 power . 'I~his situation ca.n he realised lay the addition of more memory and CPU's as required. AS ItS
3.8
Optical Data Link
3.9
Top Crate
'l'he I~ filter crate is situated 100 m underground, close to the sulxletect,ors, hui. the" events arc rrcorcíed from the VAX 8700 ven the surface . 'fhe events arc transferred frurn the ffilter crr,tr. Lw a VA9 r crate on the srrrfa .cr (the 'lb1 Craf.e) using a. duplex opl,ica.1 fibre link. 7'he fibres are c)rivc>~r a .t each c"nd l,y a. Cr,S ()r)í.81 ~12 rrrudrrle which cvenrrc "ct,s to f.he I)MA of a. C1;S rIC 8230. 1)at.a rates of > 211~iRs - ~ ca.n be achieved . 'fhe event, transfer protocol rc~duires that the event, size and other characteristics a.re known before the event is transferred . r(u achieve this, I,he protocol is established between the ODI . modules by the transmission and a.cknowledgernent of event descrir~t,urs using 'fC ;i'/ti', a. prvel,ocol suplx~rted by the M V M f; 330- I mucíules in the rüf,er and 'fop crates.
'l'he VA~1 r: 'l'up (;rake" contains I,lrc" fvellowing rrrvedules: o FIC 8230
® c)~r, 81-12 ® I,lfl' I;I,hrrnc"I, inlc"rfrrcr
39~
® IKON 10089 URII-fit' driver 0
2xUI'11~i 8241
o h4VR~iE 330-t 'hhe purpose of the 'fop Crate is to receive events via t.hc Optical Data Link using the kond of the hair of CLS ODr,$I 12 modules and to transfer them to the VAX 87()0 over a fast parallel lime using tire 1)I?C I)Il.l 1-VV protocol and CUPACK software. 'fhe surplus C1'U ín tl: ..'!c~ Crate was used to reformat the cíata into the reprc~senta-lion {IFI;F, VAX etc. required fur the subsequent event reconstruction and analysis. Ruth formats can be supported sínrultaneuusïy by the 7bp Crate sine tire events cuurcí be transferred to the VAX 8700 using two independent hardware channels of the CUPACK link and injected into two independent buffer manager streams. 'fhe Tup Crate may optionally distribute events t.o a. processor farm for event reconstruction before the events are passed to the VAX although this possíbility was not routinely used during the 1989 and 1990 running periods .
3 .10
VAX 8700
The VAX 8700 receives events from the Top Crate using a DRR-32V' module connected to the BI bus of the ~'AX . ~'he ec"ents are then recorded on íB11í 3180 cartridges using a dual cartridge drive with starkers under the cunt.rol of "Recorder" programs each of whine controls a. single volume (cartridge ur disk file) on the mass storage device . Events are dist,ribnted t,o the Recorders by the Alodel Buffer A~ia.nager [7] v+-bleb supports multiple data streams and which receives events frurn a "Producer" prugrarrr n"hose purpose is to read events from the Dli.B32-W and inject them info one ur more butler manager streams. The streams a.rc software configurable and arc therefore easily adaptable to changing requirements . lí ; addition to event recording, the VAX 8700 and on-litre VAX cluster support many of the other services required by the cía.ta acquisition system including: e Skaters di~piays o I~.rror message formatting and rveut.ing o Csrlihrat.iwn cíatabasc rnanagetnrnt 0
11-rcrnit .urirrg
S. Wotton/A data acquisition and experimental control system for the OPAL experiment
4~
® I~un contrul ® ßookkerping ® (~S-9 net,work cíisk srrver
3.11
R.un Control
li.un control is perfar~ncd on tire ~~i\t 8700 irslrrg a knuwlecígc-based s)-stcrn writt.rn in Prolog and using ul)ject:oriented prugratruating t.cchnigatcs [8]. 1`he variuals phases of a run and the t.ransit,iuns between there are represented by finite Si.ate i\iaehines (hSh~i's) organised in a hierarchical struci.trre cnrrespanding to the corrapurrrnts of file (íat.a a('glllst .lt.l()n sj'Strrra . (",c)rrrmaai(ls assor.iai,ed with transitions between i.he slates of an i~S1~I rcprrsctrt ing a given rornl)onrnt tier sent i,u that component as text sl,rings using t 1te C~fI'S prot.ucol. ~~hc respwnsc frmrn i.he curnl)onent signalling that tlae trarts11,111r1 11äS sltc'crSSfllll}' cc~rtrplet.ed rs sCTtt, 1)ack to the t';\ ~ using t,hr same protacml . In this way, ihr rein control cngiuc maintains a kncnrleclge of the cilrrcrit state of i,hr rrrn artcí call initiate transü,ioras to any oi,hcr allowed state. Aut,ornatic acrd operator assistr(1 error recovery procecíurcs help to reduce the level of expertise required by the operator to run the experiment. ~1'he operator interacts with the run control cngilir axing menus, panels and a. graphical rcprcscnt.atiwn ~f i.he 1~'S~t states cvt ti VAX Workstation . 'phis bureau inl,crface is based un Dcc~Vincíc)ws graphics .anti mindwving prirnitivc "s.
3 .12
flow Control
( ;uril,rul and rnwriil.wring cvf i.lrc snluíctccaurs' high and Iwv vult.agc puwcr sirl)hlics aaid ml.hcr aaixilia.ry systcrns arc perfc)rrned l)y llrc ()l'A1, Slwv ('.unl.ruls Systcrn [9]. 'l'his ccmsist,s uf ac"ven VI~11~; crates which rec:eive and rnonit,ur signals, but,h ana.lmgur a.nd dig_ it.al, frorn scnsurs clislril)ut.c~d armutid I,Ire a1)pa.ra.tirs. Cuntrul and statrrs corrrrriands ctiri I)r scnt, tc) i.he snlodef,ectwrs cvr uther Slmv (',c~rrt.rc)ls sr,at,icors via ()S-J pi1)rs tis sirnple I,exl, strings . A hurrr,an ínt.c" rface rr5ing thc" A1arínl,ush if,y1)c "rcarcl sc~fl,ware htis I)cen dcvc"Iu1)ed to Gacilíttitc 1.Itc nuiriíl,mríng sincl cunl,rol uf Lhc til)Irirtil,iis ti.nd a sirnl)Ir, mrrrm-drivert inl,crfaacr is tils,m availtible . ln tidclil.iun, tin inf.c" rface (,w Llrc rtin runt,rml c`ilglflr exlst.S R'lllc'.h, íalt,ll~)Ilgll tIWI~ yrl irrrl)Irrttenl,cd, vill allc)w I,Ite sill)clrl,rclvrs' vcvlt .agr Sir1)I)lic "s I,u I)c orrIc~rrrtil .ictill}' sti~nc'hrnnisccl wil li plysims rrrrts . 'l'Irc Slmv (',corl,ruls 5ysl,ctn crrn I)c rrscd (,ca l;rrrrrralc w}trnings aitrd rrlrrrrrr~r fnr aigrrtils Ilrmt. rxccccl I>rcac"t Llrrctihcrlcls,
Such ala.rtrts and wtirnlngs earl also trigger a.uturnatic: tict,icms in order to protect, the apparatus from damage and i,o 1)revent accidents . COl1Ci>1151®1~S The ()l'A1. da.i,a. acquisition s;~stem ha.s operated reliably over an extended period including the 1989 and 1990 runs in which alnu)st l50 t)00 e~c- -a hadrun int,eractiuns were recorded at appruxitnal,cly one quarter of t,lte I.1?1' design lurninusit,y. Dead-t,irnes of less than 10% were achicred a.t. trigger rates c~f about 2112. 'I~he neutral I)A(.~ system is c:urrently being upgraded t0 sllst~llrt the higher da1,a-(low ra.t,es ex1)ected during 1991 and beyond . Li:Iis exprctc"(í t .o appraa.ch cvr exceed the design lurrainusity and t,irere is a. pussibilil,y that, the nurnl)er of e+~ - lnlnrhes will I)e inc:rrased . 'hhe current filter sysi,ern leas 1nsulficient processing i)cnver t,u sustain these higher rates and is I)ring replaced by an Apollo llN 10000 1)rucessing faun while t.lie central trigger sysl,rrn will benefit llama the additicm of fast per-trigger logic f .cl aceornudate the reduced t,irYae bel,ween bunch-crossings when 8-1)unch 1~f~1' upcrat,iun is irnplcrraentcd .
~.eferences 'l'he ()fA1, Cc)llaburtii,ion, 'I~he UI'Ai, T)ri,eci.or at T,1~;1', ('l:ß,lV f'T'l; 90-I I-1, Suhrriil,tcd t,c~ Nucl. Instr. a.ncl it,Icthods, Airgust 19S)0 . [2] 11 . vcln dcr Schtrritt, li:l~(~~68K 1i,ea.l 'l'inic I~mrI,rtin 77 fur 68K f'rucessnrs, i'lrysiktilisclrcs lrrsl,itul, llnivc".rsit.at, Ileidelherg, i~1ay 1989. [:3] 'l'. .1. llrrrrcrs-I~ec, 'l'he CATS 'franspmrl . Service ('tilling Scduencc, i'rugrarn Library', ('T';ß.N, 1)1) Division, 121 I (ácnevra., Switzerland, R . Rrurr, it-i . (i(1t,1S5E'rIS, ,1 . 7,011, 7,I~,R1i.A lyntirriic I)ti.ta Sl,ruc;tirre ancl 11~Ierrrury lllanager, 1'rmgr.~rn l~il)rary, 1)ll ~ivision, ('l?Ii.N, 1211 (ärrirv .a, Swit.zcrland .
[ rr] 'l'he ()l'AT~ 'l'riggcr Sysl,crrt, Nircl . Irrsl.r. ancl (1Tct.limd5 .
1.0
lac sul)rrril.l,ed t,o
[fi] ( ;I';5, Iicntt.c du l'unl-Fiirl .in 70, 121 :3 l'c"I,il,-1,atrcy°, (vcrrevra, Swit.zcrland . ('. Fiwisstii, l'. ~'titrclc V'd'vrr, 'l'irr 111()I)i;l, Fittfü~r ~isrnrigcr (lac"r Wtidr, i'rcigrsirtr I.ilrrsrry, ('i;li.N, i)I) I)iviwiwrr, l'll I (vrrrrvti 23, Wvitrrrlan(I .
S. Wotton /A data acquisition and experimental control system for the OPAL experiment [8] Robin I' . l~liddleton, Conl,rol rrf a. LI;P I;xp~rirnent using a knowledge-Raced System, CNRS, Proceedings of the International Workshop on Soi'tware Engineering, Artificial Intelligence and Expert S,vsf.ems foc iligh l;nergy and Nuclear Physics, r.,yons, I~'rarrce, 19--2~1 A7arch 1990 . [9] A.K . Arnundscn c~t al ., Nucl . lnstr. arul AZethods, A293 (1990) 1 ~15.
403