- Email: [email protected]
The use of transputers in the ZEUS online system
316
Computer Physics Communications 57 (1989) 316—320 North-Holland
THE USE OF TRANSPUTERS IN THE ZEUS ONLINE SYSTEM ZEUS Collaboration L.W. WIGGERS and J.C. VERMEULEN NIKHEF-H, P.O. Box 41882, 1009 DB Amsterdam, The Netherlands
Transputers are used in the ZEUS experiment at DESY in several systems for read-out, triggering and data-transport. In this paper we discuss the various architectures and the expected performance figures.
1. Introduction In November 1990 the HERA electron—proton collider at DESY is scheduled to start its operation. ZEUS is one of the two general purpose detectors at that collider. Every 96 ns beambunches cross at the interaction point in the detector. The rate of hard electron—proton interactions is about 1—10 Hz, while the background of beam—gas interactions is much higher. The firstlevel trigger has the task to set a trigger rate of 1 kHz by eliniinating most of the beam—gas background. Programmable logic is applied at this level. At the second-level commercially available microprocessors will analyze the digitized data of the subdetectors. Combining the results of those processors should result in a further reduction of the trigger rate to 100 Hz. At the third-level powerful RISC processors (MIPS R3000) will reduce the rate to 3—10 Hz by an analysis of the whole event, Transputers are applied in the second-level trigger, the read-out of the subdetectors after a second-level trigger and the building of the events out off the data of the subdetectors. In this paper the architecture of those systems will be discussed briefly.
2. Transputers The Transputer microprocessor of INMOS has been developed to be a building block in concur0010-4655/89/$03.50 © Elsevier Science Publishers B.V. (North-Holland)
rent systems. Important properties for our type of applications are: (1) dma-driven data transport over 4 links (1.7 MByte/s per link); (2) multi-tasking with task-switching times of just a few microseconds; (3) a powerful cpu (for the T800: 20 Mips, 1.5 Mflops); (4) support in the OCCAM language for parallel processing and for interprocess communication and synchronization.
3. Front-end read-out 3.1. Strategy
After a first-level trigger the data of the subdetectors are taken from pipelines and the digitized data are stored in buffers in read-out modules, positioned in VME crates in the electronics barrack in the experimental area. About 45% of the 260 000 electronics channels of the subdetectors will be read-out by INMOS transputers, the other 55% by Motorola 680x0 processors. However, most of the data (80—90%) will be transported by the transputer systems in the experiment. In those systems a transputer transports data via a backplane bus from a dual-port memory (DPM) of the front-end module into its private memory for second-level processing and afterwards for further transport to the event builder. Two solutions are pursued for the backplane bus: (a) an extension of
L W. Wiggers, J. C. Vermeulen 3 bidirectional links
T2l2 or 1222
/
dual-port
second-level triggering [2,3]. In the calorimeter read-out a 2TP-module han-
I
1212 or 1222
________
external bus buffers + address generator I
I
backplane bus
Fig. 1. Layout of the controller for the tracking detector read-out, under development at University College London.
the external transputer bus; (b) the VME standard. Solution (a) is used by the groups working on tracking detector read-out, solution (b) by the calorimeter groups. The advantage of (a) is the simpler interface, of (b) the more general potential. 3.2. Transputer bus backplane In fig. 1 the layout of a read-out controller is sketched as planned for the read-out of the tracking subdetectors in ZEUS [1]. A 16-bits T212 transfers data from DPM’s on the front-end cards with block transfers over its transputer bus. A data rate of up to 13 Mbyte/s is achievable. Data in the DPM will be used for second-level trigger calculations by the second transputer. Transputer links connect crates for data transport to the event builder and for triggering purposes.
dles the data of a crate. After a first-level trigger a part of the digitized data in the front-end cards is read by transputer Y into the DPM. Transputer X will process those data to search for clusters of energy. After a positive second-level trigger all data is read and transported via the links of transputer Y to the event builder. Both transputers have access to the VME bus and can issue VME instructions. The DPM is also accessible by another master on the VME bus. The transfer speed for the present prototypes is 7 Mbyte/s for reading by transputer Y and 10 Mbyte/s for direct write into the DPM of the 2TP-module from outside. For test set-ups a prototype series of 20 modules have been made; in spring of 1990 the final production modules will be available. The latter differ in details from the prototypes: (1) the interfacing to the local bus and to the VME bus is handled by ASIC devices, resulting in faster transfers over the VME bus and to the DPM; (2) both transputers have direct access to the DPM, transforming it into a triple port mem(3) ory; single slot occupancy.
3.3. VME bus backplane
4. Data transport
In fig. 2 the layout is sketched for a controller (2TP-module) for use in the calorimeter read-out
4.1. Link speed
4 bidirectional links
The transfer speed over a link depends on the
4 bidirectional links
________
________
reset~ 800 or T425 1T800 or T425 ~reset, analyse, analyse T + + ~evenr j~ event ifl I or 4 MByte output event in I or 4 MByte error in, errorifl _______________ programmable memory I memory ~~programmable + eventin output 1 + oUtput I/O logic X ___________ I/O logic Y output,
r
errorOut Junrper~
inte~pt request
t
dual-port
A
memory local bus 1 I VME-busrntefface VME bus
$
I
317
and triggering, the event builder and the global
3 bidirectional links
I
Transputers in the ZEUS online system
j I.
error out
~ j urnper inte~pt request
Fig. 2. Read-out controller (2TP-module) in development at NIKHEF.
length of the packet link cables; eachback bytefrom sent the an acknowledge has tofor travel receiving to the sending transputer. The transfer speed can be described for a T800 at 20 Mbit/s as: transfer = 100/(60
+
[length(m)/51
x
5)Mbyte/s. (1)
Where Ix] is the smallest integer larger than x. So
318
L W. Wiggers, J. C. Vermeulen
/
for a cable of a length of about 55 m the transfer speed is half that of a zero length cable. The use of T222 transputers without external memory as repeaters can improve the transfer speed for long cables. A T222 receives data, chops them in small packages and immediately copies them to the output. Data from two links can be handled simultaneously. The overhead on the transport is minimal, the overall transfer speed is higher because of the smaller cable lengths. The process is transparent for the sending and receiving transputers.
Transputers in the ZEUS online system 16 Sub. detectors ______
T
T222
I
/
~,
~i4
:
is T222
t
Crates ______
t6
C012
T222
I I
I •
I
•
I
Ti6 ______
CSB 2u
C004
i
In
_____
T-SSC
___________
L
_____
I
_______
4
T T
: T
______
__________
T
is T222
Fig. 4. Layout of the event builder. Data from 16 subdetectors are sent via 3 C004 link switches to 6 2TP-modules in the third-level crates.
After a second-level trigger the data stored in the front-end cards have to be transported to the third-level trigger system via the event builder at an expected frequency of 100 Hz. First the data of the subdetector are assembled in a 2TP-module in the sub-system crate (SSC) of each subdetector. The transputer networks of the subdetectors will have different configurations. Here the configuration for the calorimeter, applying a crossbar switch, is discussed. The data of the 36 read-out crates are sent from the collecting 2TP-modules over a multiplexer to 3 2TP-modules in the SSC’s of the calorimeter. This multiplexer (control and switch box (CSB)) is built around the INMOS crossbar C004 link switch (fig. 3). At maximum 16 crates are connected via the switch to the receiving transputer of the 2TP-module in the SSC. A transputer sends a request for a connection to the
TI
4
C004’s
1 T16
4.2. Transport to the subsystem crate
Read-Out
SThird-Level Crates
__________
I
Fig. 3. Layout of the read-out network of the calorimeter with a control and switch box (CSB) multtplexsng the data of the read-out crates (Ti to T16) to a transputer in the subsystem crate (T-SSC).
2TP-module in the SSC over a link to a controlling T222. This request is latched in an C012 (serial to parallel converter) connected to the external bus of the T222 transputer. After the transport over the C004 the connection is released. The CSB also has a simple uni-directional broadcast circuit to fan-out trigger decisions over many links [4]. For the transport from read-out crate to SSC crate the transfer speed over in total 7.5 m plus linkswitch is about 1 Mbyte/s per link.
5. Event builder Instead of building an event at a central place and transporting it from there to the third-level trigger, a solution is adopted where the event is built in each individual third-level trigger crate. The data is transported from a 2TP-module in the SSC over a switch (fig. 4). As for the calorimeter read-out a request is sent to a controlling T222. This transputer makes a connection to an 2TPmodule in a third-level trigger crate. An event is assembled in the 2TP-module over 4 links in parallel. For each subdetector 3 links are used for transport and at most 12 events are built simultaneously in the third-level crates. total 16 subdetectors are connected to the event builder over 48 links data are flowing into the third-level crates. The transfer speed over the links and link switch is about 0.9 Mbyte/s per
L. W. Wiggers, J. C. Vermeulen / Transputers in the ZEUS online system
6.2. Global second-level trigger
~t~i~t~1-LeveI Trigger
T-(13
In the global trigger the results of the local processors are combined. A final decision is taken, determining whether an event has to be assembled or has to be skipped in the buffers in the read-out
Master
t)I’M
T-Cl
T.XI
•• •
‘I’-C2
T-X4
T-X5
•
, ,
T-X6
•
+5~
••
______
Fig. 5. Structure of the network of trigger transputers (T-xi to T-X36) and the combining transputers (T-Ci to T-C13 plus the master) in 7 2TP-modules.
-
319
.
link. So the maximum throughput of the system is about 40 Mbyte/s.
6. Second-level triggering 6.1. Second-level local processing Five of the ten second-level triggering systems will use transputers. In this section one subdetector trigger, the calorimeter system, will be discussed in more detail. The X transputer of fig. 2 processes part of the front-end data after a first-level trigger, searching for clusters of energy, signaling the presence of leptons or hadron jets. The results of the local processing of the 36 trigger transputers are cornbined to get an overview of the calorimeter. A tree structure (fig. 5) is chosen, since it minimizes the latency in the network compared to mesh structures. Data of 6 trigger transputers are assembled in a 2TP-module. One transputer receives data over 4 links, the other over 2. After combining the results the data is transferred to the next layer where eventually one transputer sends the results to the global second-level trigger. The total latency is about 3.5 ms, assuming a maximum processing time of about 1 ms at each of the tree stages. Simulation studies justify this assumption.
crates. About 10 subdetectors are participating in the local second-level processing. In the global trigger different algorithms run in parallel on 8 trigger transputers in 4 2TP-modules. From every subdetector the data is broadcasted to those processors. The results of the processing are sent from the trigger transputers over the VME bus to a master processor, that takes the final decision. The decision is broadcast to the subdetectors via the event builder.
7. Interface to hosts A general solution has been adopted in ZEUS. The MicroVaxes controlling the subdetectors, are equipped with an Q-Bus to transputer link interface of CAPLIN Cybernetics Corporation. Via such an interface four users can simultaneously connect to four transputers. Programs are developed at the MicroVax and stored on disk. Monitor data is sent from the transputer systems to the MicroVaxes, displayed there and messages sent to the main data-acquisition Vax for control and monitoring.
8. Conclusion The concepts of parallel triggering and read-out with transputers are widely accepted in the collaboration. ZEUS is using the transputers on a large scale (about 300) for many subdetectors. It is the first time in high-energy physics that transputers are applied on such a scale for data-acquisition and triggering. For the different applications various types of networks are applied: switched, tree and daisy chained. To test the concepts of trigger and readout, prototype transputer modules will be used in test set-ups at CERN, RAL and FERMILAB in 1989 and 1990.
320
L.W. Wiggers, J.C. Vermeulen / Transputers in the ZEUS online system
Acknowledgements We want to express our thanks to all colleagues in the ZEUS collaboration participating in the design and development of the transputer systems. References [1] R. Belusevic and G. Nixon, NucI. Instrum. Methods A 277 (i989) 513.
[2] H. Boterenbrood et al., Proc. tnt. Conf. on the Impact of Digital Microelectronics and Microprocessors on Particle Physics, eds. M. Budinich et al. (World Scientific, Singapore, 1988) p. 217. [3] H. Boterenbrood et al., Proc. of VMEbus in Research, eds. C. Eck and C. Parkman (North.Holland, Amsterdam, 1988) p. 109. [4] H. Boterenbrood et a!., Proc. 10th Occam User Group Technical Meeting, ed. A. Bakkers (lOS, Amsterdam, i989) p. 289.
Computer Physics Communications 57 (1989) 316—320 North-Holland
THE USE OF TRANSPUTERS IN THE ZEUS ONLINE SYSTEM ZEUS Collaboration L.W. WIGGERS and J.C. VERMEULEN NIKHEF-H, P.O. Box 41882, 1009 DB Amsterdam, The Netherlands
Transputers are used in the ZEUS experiment at DESY in several systems for read-out, triggering and data-transport. In this paper we discuss the various architectures and the expected performance figures.
1. Introduction In November 1990 the HERA electron—proton collider at DESY is scheduled to start its operation. ZEUS is one of the two general purpose detectors at that collider. Every 96 ns beambunches cross at the interaction point in the detector. The rate of hard electron—proton interactions is about 1—10 Hz, while the background of beam—gas interactions is much higher. The firstlevel trigger has the task to set a trigger rate of 1 kHz by eliniinating most of the beam—gas background. Programmable logic is applied at this level. At the second-level commercially available microprocessors will analyze the digitized data of the subdetectors. Combining the results of those processors should result in a further reduction of the trigger rate to 100 Hz. At the third-level powerful RISC processors (MIPS R3000) will reduce the rate to 3—10 Hz by an analysis of the whole event, Transputers are applied in the second-level trigger, the read-out of the subdetectors after a second-level trigger and the building of the events out off the data of the subdetectors. In this paper the architecture of those systems will be discussed briefly.
2. Transputers The Transputer microprocessor of INMOS has been developed to be a building block in concur0010-4655/89/$03.50 © Elsevier Science Publishers B.V. (North-Holland)
rent systems. Important properties for our type of applications are: (1) dma-driven data transport over 4 links (1.7 MByte/s per link); (2) multi-tasking with task-switching times of just a few microseconds; (3) a powerful cpu (for the T800: 20 Mips, 1.5 Mflops); (4) support in the OCCAM language for parallel processing and for interprocess communication and synchronization.
3. Front-end read-out 3.1. Strategy
After a first-level trigger the data of the subdetectors are taken from pipelines and the digitized data are stored in buffers in read-out modules, positioned in VME crates in the electronics barrack in the experimental area. About 45% of the 260 000 electronics channels of the subdetectors will be read-out by INMOS transputers, the other 55% by Motorola 680x0 processors. However, most of the data (80—90%) will be transported by the transputer systems in the experiment. In those systems a transputer transports data via a backplane bus from a dual-port memory (DPM) of the front-end module into its private memory for second-level processing and afterwards for further transport to the event builder. Two solutions are pursued for the backplane bus: (a) an extension of
L W. Wiggers, J. C. Vermeulen 3 bidirectional links
T2l2 or 1222
/
dual-port
second-level triggering [2,3]. In the calorimeter read-out a 2TP-module han-
I
1212 or 1222
________
external bus buffers + address generator I
I
backplane bus
Fig. 1. Layout of the controller for the tracking detector read-out, under development at University College London.
the external transputer bus; (b) the VME standard. Solution (a) is used by the groups working on tracking detector read-out, solution (b) by the calorimeter groups. The advantage of (a) is the simpler interface, of (b) the more general potential. 3.2. Transputer bus backplane In fig. 1 the layout of a read-out controller is sketched as planned for the read-out of the tracking subdetectors in ZEUS [1]. A 16-bits T212 transfers data from DPM’s on the front-end cards with block transfers over its transputer bus. A data rate of up to 13 Mbyte/s is achievable. Data in the DPM will be used for second-level trigger calculations by the second transputer. Transputer links connect crates for data transport to the event builder and for triggering purposes.
dles the data of a crate. After a first-level trigger a part of the digitized data in the front-end cards is read by transputer Y into the DPM. Transputer X will process those data to search for clusters of energy. After a positive second-level trigger all data is read and transported via the links of transputer Y to the event builder. Both transputers have access to the VME bus and can issue VME instructions. The DPM is also accessible by another master on the VME bus. The transfer speed for the present prototypes is 7 Mbyte/s for reading by transputer Y and 10 Mbyte/s for direct write into the DPM of the 2TP-module from outside. For test set-ups a prototype series of 20 modules have been made; in spring of 1990 the final production modules will be available. The latter differ in details from the prototypes: (1) the interfacing to the local bus and to the VME bus is handled by ASIC devices, resulting in faster transfers over the VME bus and to the DPM; (2) both transputers have direct access to the DPM, transforming it into a triple port mem(3) ory; single slot occupancy.
3.3. VME bus backplane
4. Data transport
In fig. 2 the layout is sketched for a controller (2TP-module) for use in the calorimeter read-out
4.1. Link speed
4 bidirectional links
The transfer speed over a link depends on the
4 bidirectional links
________
________
reset~ 800 or T425 1T800 or T425 ~reset, analyse, analyse T + + ~evenr j~ event ifl I or 4 MByte output event in I or 4 MByte error in, errorifl _______________ programmable memory I memory ~~programmable + eventin output 1 + oUtput I/O logic X ___________ I/O logic Y output,
r
errorOut Junrper~
inte~pt request
t
dual-port
A
memory local bus 1 I VME-busrntefface VME bus
$
I
317
and triggering, the event builder and the global
3 bidirectional links
I
Transputers in the ZEUS online system
j I.
error out
~ j urnper inte~pt request
Fig. 2. Read-out controller (2TP-module) in development at NIKHEF.
length of the packet link cables; eachback bytefrom sent the an acknowledge has tofor travel receiving to the sending transputer. The transfer speed can be described for a T800 at 20 Mbit/s as: transfer = 100/(60
+
[length(m)/51
x
5)Mbyte/s. (1)
Where Ix] is the smallest integer larger than x. So
318
L W. Wiggers, J. C. Vermeulen
/
for a cable of a length of about 55 m the transfer speed is half that of a zero length cable. The use of T222 transputers without external memory as repeaters can improve the transfer speed for long cables. A T222 receives data, chops them in small packages and immediately copies them to the output. Data from two links can be handled simultaneously. The overhead on the transport is minimal, the overall transfer speed is higher because of the smaller cable lengths. The process is transparent for the sending and receiving transputers.
Transputers in the ZEUS online system 16 Sub. detectors ______
T
T222
I
/
~,
~i4
:
is T222
t
Crates ______
t6
C012
T222
I I
I •
I
•
I
Ti6 ______
CSB 2u
C004
i
In
_____
T-SSC
___________
L
_____
I
_______
4
T T
: T
______
__________
T
is T222
Fig. 4. Layout of the event builder. Data from 16 subdetectors are sent via 3 C004 link switches to 6 2TP-modules in the third-level crates.
After a second-level trigger the data stored in the front-end cards have to be transported to the third-level trigger system via the event builder at an expected frequency of 100 Hz. First the data of the subdetector are assembled in a 2TP-module in the sub-system crate (SSC) of each subdetector. The transputer networks of the subdetectors will have different configurations. Here the configuration for the calorimeter, applying a crossbar switch, is discussed. The data of the 36 read-out crates are sent from the collecting 2TP-modules over a multiplexer to 3 2TP-modules in the SSC’s of the calorimeter. This multiplexer (control and switch box (CSB)) is built around the INMOS crossbar C004 link switch (fig. 3). At maximum 16 crates are connected via the switch to the receiving transputer of the 2TP-module in the SSC. A transputer sends a request for a connection to the
TI
4
C004’s
1 T16
4.2. Transport to the subsystem crate
Read-Out
SThird-Level Crates
__________
I
Fig. 3. Layout of the read-out network of the calorimeter with a control and switch box (CSB) multtplexsng the data of the read-out crates (Ti to T16) to a transputer in the subsystem crate (T-SSC).
2TP-module in the SSC over a link to a controlling T222. This request is latched in an C012 (serial to parallel converter) connected to the external bus of the T222 transputer. After the transport over the C004 the connection is released. The CSB also has a simple uni-directional broadcast circuit to fan-out trigger decisions over many links [4]. For the transport from read-out crate to SSC crate the transfer speed over in total 7.5 m plus linkswitch is about 1 Mbyte/s per link.
5. Event builder Instead of building an event at a central place and transporting it from there to the third-level trigger, a solution is adopted where the event is built in each individual third-level trigger crate. The data is transported from a 2TP-module in the SSC over a switch (fig. 4). As for the calorimeter read-out a request is sent to a controlling T222. This transputer makes a connection to an 2TPmodule in a third-level trigger crate. An event is assembled in the 2TP-module over 4 links in parallel. For each subdetector 3 links are used for transport and at most 12 events are built simultaneously in the third-level crates. total 16 subdetectors are connected to the event builder over 48 links data are flowing into the third-level crates. The transfer speed over the links and link switch is about 0.9 Mbyte/s per
L. W. Wiggers, J. C. Vermeulen / Transputers in the ZEUS online system
6.2. Global second-level trigger
~t~i~t~1-LeveI Trigger
T-(13
In the global trigger the results of the local processors are combined. A final decision is taken, determining whether an event has to be assembled or has to be skipped in the buffers in the read-out
Master
t)I’M
T-Cl
T.XI
•• •
‘I’-C2
T-X4
T-X5
•
, ,
T-X6
•
+5~
••
______
Fig. 5. Structure of the network of trigger transputers (T-xi to T-X36) and the combining transputers (T-Ci to T-C13 plus the master) in 7 2TP-modules.
-
319
.
link. So the maximum throughput of the system is about 40 Mbyte/s.
6. Second-level triggering 6.1. Second-level local processing Five of the ten second-level triggering systems will use transputers. In this section one subdetector trigger, the calorimeter system, will be discussed in more detail. The X transputer of fig. 2 processes part of the front-end data after a first-level trigger, searching for clusters of energy, signaling the presence of leptons or hadron jets. The results of the local processing of the 36 trigger transputers are cornbined to get an overview of the calorimeter. A tree structure (fig. 5) is chosen, since it minimizes the latency in the network compared to mesh structures. Data of 6 trigger transputers are assembled in a 2TP-module. One transputer receives data over 4 links, the other over 2. After combining the results the data is transferred to the next layer where eventually one transputer sends the results to the global second-level trigger. The total latency is about 3.5 ms, assuming a maximum processing time of about 1 ms at each of the tree stages. Simulation studies justify this assumption.
crates. About 10 subdetectors are participating in the local second-level processing. In the global trigger different algorithms run in parallel on 8 trigger transputers in 4 2TP-modules. From every subdetector the data is broadcasted to those processors. The results of the processing are sent from the trigger transputers over the VME bus to a master processor, that takes the final decision. The decision is broadcast to the subdetectors via the event builder.
7. Interface to hosts A general solution has been adopted in ZEUS. The MicroVaxes controlling the subdetectors, are equipped with an Q-Bus to transputer link interface of CAPLIN Cybernetics Corporation. Via such an interface four users can simultaneously connect to four transputers. Programs are developed at the MicroVax and stored on disk. Monitor data is sent from the transputer systems to the MicroVaxes, displayed there and messages sent to the main data-acquisition Vax for control and monitoring.
8. Conclusion The concepts of parallel triggering and read-out with transputers are widely accepted in the collaboration. ZEUS is using the transputers on a large scale (about 300) for many subdetectors. It is the first time in high-energy physics that transputers are applied on such a scale for data-acquisition and triggering. For the different applications various types of networks are applied: switched, tree and daisy chained. To test the concepts of trigger and readout, prototype transputer modules will be used in test set-ups at CERN, RAL and FERMILAB in 1989 and 1990.
320
L.W. Wiggers, J.C. Vermeulen / Transputers in the ZEUS online system
Acknowledgements We want to express our thanks to all colleagues in the ZEUS collaboration participating in the design and development of the transputer systems. References [1] R. Belusevic and G. Nixon, NucI. Instrum. Methods A 277 (i989) 513.
[2] H. Boterenbrood et al., Proc. tnt. Conf. on the Impact of Digital Microelectronics and Microprocessors on Particle Physics, eds. M. Budinich et al. (World Scientific, Singapore, 1988) p. 217. [3] H. Boterenbrood et al., Proc. of VMEbus in Research, eds. C. Eck and C. Parkman (North.Holland, Amsterdam, 1988) p. 109. [4] H. Boterenbrood et a!., Proc. 10th Occam User Group Technical Meeting, ed. A. Bakkers (lOS, Amsterdam, i989) p. 289.