- Email: [email protected]
Development of readout system for FE-I4 pixel module using SiTCP
Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
Contents lists available at ScienceDirect
Nuclear Instruments and Methods in Physics Research A journal homepage: www.elsevier.com/locate/nima
Development of readout system for FE-I4 pixel module using SiTCP J.J. Teoh a,n, K. Hanagaki a, Y. Ikegami b, Y. Takubo b, S. Terada b, Y. Unno b a b
Graduate School of Science, Osaka University, 1-1 Machikaneyama, Toyonaka, Osaka, 560-0043, Japan Institute for Particle and Nuclear Studies, High Energy Accelerator Research Organization (KEK), 1-1 Oho, Tsukuba City, Ibaraki-ken 305-0801, Japan
art ic l e i nf o
a b s t r a c t
Available online 2 June 2013
The ATLAS pixel detector will be replaced in the future High Luminosity-Large Hadron Collider (HL-LHC) upgrade to preserve or improve the detector performance at high luminosity environment. To meet the tight requirements of the upgrade, a new pixel Front-End (FE) Integrated Circuit (IC) called FE-I4 has been developed. We have then devised a readout system for the new FE IC. Our system incorporates Silicon Transmission Control Protocol (SiTCP) technology (Uchida, 2008 [1]) which utilizes the standard TCP/IP and UDP communication protocols. This technology allows direct data access and transfer between a readout hardware chain and PC via a high speed Ethernet. In addition, the communication protocols are small enough to be implemented in a single Field-Programable Gate Array (FPGA). Relying on this technology, we have been able to construct a very compact, versatile and fast readout system. We have developed a firmware and software together with the readout hardware chain. We also have established basic functionalities for reading out FE-I4. & 2013 Elsevier B.V. All rights reserved.
Keywords: HL-LHC ATLAS upgrade Pixel detector FE-I4 SEABAS DAQ SiTCP
1. Introduction Around the year 2022 the LHC will undergo a major upgrade in the so-called HL-LHC project. The ultimate goal is to collect 3000 fb−1 of data by 2030. To achieve this goal, the instantaneous luminosity will have to be increased to 5 1034 cm−2 s−1. This increase in luminosity will greatly enhance the discovery potential of the ATLAS experiment but there is a high price to pay. At this luminosity, the number of pile-up events is expected to reach ∼140 per bunch crossing [2]. This harsher radiation environment and higher event rate pose unprecedented challenges to the detectors especially the pixel detector which is closest to the interaction region. Studies have shown that the current ATLAS pixel detector's FE readout IC, FE-I3 cannot cope with the high hit rate environment, leading to unacceptable detection inefficiency. Thus for the ATLAS experiment, a complete replacement of its pixel detector is foreseen. Motivated by this pixel detector upgrade, a newer generation of FE readout IC which can satisfy the criteria of higher granularity, better radiation hardness and lower material budget has been developed. The new readout IC is called FE-I4 [3,4]. It is the largest readout IC produced to date in an high energy physics experiment. The FE-I4 has a total of 26 880 pixels which is about 10 times more than its predecessor, FE-I3. The pixels are arranged in a 80 336 matrix on a 20 18.6 mm2 chip. The larger chip size leads to more efficient system integration and thus reduces the amount of material per detector layer. Furthermore, the larger chip size also leads to a significant manufacturing cost reduction. The pixel size of 50 250 μm2 is about 40% smaller than its predecessor. This n
Corresponding author. Tel.: +81 6 6850 5359; fax: +81 6 6850 5532. E-mail address: [email protected] (J.J. Teoh).
0168-9002/$ - see front matter & 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.nima.2013.05.161
reduction in pixel size significantly reduces the pile-up inefficiency and improves its position resolution. All input/output (I/O) signals of FE-I4 are in LVDS. The nominal I/O bandwidth of 160 Mb/s is four times higher than FE-I3. The current version of FE-I4 is designed in a 130-nm CMOS technology, but in the future it could be replaced by a higher integration density process. By upgrading to FE-I4, ATLAS aims to achieve the same or even better tracking and vertexing performance than the current pixel detector. Ref. [5] has more detailed information about FE-I4. To read out the new IC, we have developed a new readout system by using a general purpose DAQ board named Soi EvAluation BoArd with Sitcp (SEABAS) [6]. The system will act as an electronic test bench for FE-I4 and provide various functionalities for FE-I4 characterization and performance analysis. Besides acting as the electronic test bench, it can also serve as a DAQ system for a variety of sensor modules such as a silicon planar pixel sensor, 3D silicon sensor and diamond pixel sensor. Furthermore, the readout system must be able to read out multiple FE-I4s (quad-module as currently planned). By using SiTCP technology, we aim to design a very compact, flexible and fast multi-chip readout system. At the current stage, we have developed and tested our system for a single chip readout. In this paper we will give an overview of the readout system interface in Section 2 and the implementation details of the firmware and software in Sections 3 and 4, respectively. The test result will then be presented in Section 5.
2. Readout system interface Our readout system consists of a PC, SEABAS DAQ board, SEABAS-USBpix daughter card, USBpix FE-I4 adapter card and single
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
SiTCP processes the TCP/IP, UDP and Ethernet protocols where users need to only optimize the protocols for their specific application. In other words, the communication and data transmission between PC and SEABAS is entirely taken care of automatically by the SiTCP firmware. This significantly simplifies the development of the readout firmware for any further upgrades. The user FPGA is a Xilinx Virtex-4 FPGA (XC4VLX25) with 1296 kb maximum Block RAM capacity. We have designed the firmware such that it is mainly responsible for forwarding command bit streams to FE-I4 and receiving data outputs from FE-I4. Of all the command types of FE-I4, the firmware only manipulates the Level 1 Trigger Command (LV1) and Calibration Pulse Command (CAL). This is important as the timing of issuing the CAL and LV1 commands needs to be very accurate. Furthermore, this implementation is also meant to ensure maximum flexibility for FE-I4 configuration while saving processing time. The block diagram of the firmware components is shown in Fig. 2. First, in the Top_module the SiTCP Communicator block ensures proper communication between a PC, SiTCP FPGA and User FPGA at all times by taking care of the assignment of user interface signals of the SiTCP at correct timing. System clocks of various frequencies are generated in the Digital Clock Manager
FE-I4
LV1 Trigger, CAL Pulse
FIFOs
Signal_Sender
Signal_Reader
Reset Channels_Manager
Cmd_Decoder
DCM 3. Firmware implementation
SiTCP Communicator
The SiTCP firmware loaded in one of the FPGAs is a hardwarebased TCP/IP processor which is specifically designed and optimized for front-end devices Ref. [1]. It is basically functioning like a synchronous FIFO with a circuit size small enough to be implemented on a single FPGA. This makes the system inherently more flexible than the traditional backplane bus systems such as CAMAC, PCI, VME and ATCA. On top of the fast TCP protocol, a slow control mechanism over User Datagram Protocol (UDP) is also provided.
Single FE-I4 module card
USBpix FE-I4 adapter card
User FPGA SiTCP FPGA SEABAS PC Fig. 2. The block diagram of the firmware for reading out FE-I4.
64 pin connectors RJ45 PHY 100 BASE-T Ethernet
FE-I4
NIM I/O Flat ribbon cable
Configuration Relay
Power
KEL 100pin SEABAS-USBpix daughter card connector
Fig. 1. The simplified schematic of our new readout system (single chip setup).
Power (+5V, -5V)
Top_module
FE-I4 module card. The SEABAS DAQ board was originally developed by the Silicon On Insulator (SOI) collaboration at KEK [7]. Fig. 1 shows an overview of the readout system interface. For a single chip setup, the SEABAS DAQ board is connected to the SEABAS-USBpix daughter card. The SEABAS-USBpix daughter card is then connected to the USBpix FE-I4 adapter card via a KEL 100-pin connector. We have borrowed the USBpix FE-I4 adapter card made by the SiLab of University of Bonn [8] as a temporary solution while waiting for a dedicated adapter card to be made to suit our system. The USBpix FE-I4 adapter card and the single FE-I4 module card are connected by a flat ribbon cable. The flat ribbon cable acts both as a data and power supply line. The SEABAS DAQ board and FE-I4 are independently powered from external power supplies. Finally, an Ethernet cable provides the communication link between computer and the readout hardware chain. In this setup, the SEABAS-USBpix daughter card and the USBpix FE-I4 adapter card are used to route the signals essential for operating FE-I4. In particular, those signal lines include configuration command, reference clock, output data from FE-I4, forced reset signal and various others. All the signals sent from SEABAS to the adapter card are single ended. Remember that FE-I4's I/O signal is in the form of LVDS signal. Thus three I/O signals mentioned above, i.e. the configuration command, the reference clock and the output data from FE-I4 are converted to LVDS signals before being fed into the flat ribbon cable. This signal conversion is performed by the LVDS transceivers located on the adapter card. In addition to the LVDS transceiver, the adapter card also hosts the bias voltage regulators for FE-I4. The SEABAS board has two on-board FPGAs. One FPGA is loaded with the SiTCP firmware; the other affords user's configuration. At the moment we are using SEABAS version 1.1 which employs the 100BASE-T standard with a 100 Mbps line bandwidth. The Ethernet physical layer device (PHY) is linked to the SiTCP with the standard Gigabit Ethernet Media Independent Interface (GMII) specified by IEEE802.3. On top of that, NIM I/O ports could be used to provide external signals such as external trigger signals, external analog pulses and external digital pulses for certain FEI4 operations.
Job_Manager_module
238
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
(DCM). Meanwhile, the CMD_Decoder block identifies headers from the data streams that are sent from the control software and decides what to do next. Each header is associated with a specific task, for example, to activate other modules or to assign values to firmware registers by using a payload extracted from the data stream. On the other hand, the Signal_Sender block in the Job_Manager_module is responsible for forwarding or relaying the command bit streams originated from the software to FE-I4 at 40 MHz
FE-I4
SEABAS
SiTCP Controller
Slow & Fast Cmds
4. Software implementation
CMD Generator Decoder
Injection INJ DAQ
Calibration
Tuning
OPR CNFG
without further encoding. While the other commands are sent from the DAQ software, the LV1 and CAL commands are handled directly by this block. As an example, in a typical calibration or charge injection operation this block will send a CAL command followed by a LV1 command with the predefined trigger interval and the latency value. Once the CAL command is received, FE-I4's internal Pulse Generator circuit will generate digital or analog calibration pulses. The LV1 command will then trigger the acquisition of a new hit or event from FE-I4. After the proper initialization of FE-I4, the Signal_Reader block will continuously monitor the input port for any incoming data from FE-I4. Four asynchronous FIFOs are embedded in this block. The FIFOs will read the incoming 10-bit data words at 20 MHz while writing them out at 25 MHz. The 25 MHz readout clock is optimized for the SEABAS 100BASE-T Ethernet version in order to fully utilize the available bandwidth. In the case of multi-chip readout, the FIFO to be read out is automatically determined by the Channel_Manager block. At the moment, however, only one FIFO is used to read out a single FE-I4 by masking the other FIFOs. Lastly, the Reset block is responsible for resetting all registers within the Job_Manager_module and Top_module to their initial values. The reset signal is issued either after each data taking cycle or upon receiving a reset command from the control software.
Data Acquisition
Firmware Ctrl Cmd
Configuration
239
Analysis Tool DOUT
DAQ & Control Software Fig. 3. The simplified block diagram of the software implemented in our readout system.
The development of the DAQ software is mostly decoupled from the development of the FPGA firmware. This allows a much more efficient way of upgrading the system to suit the needs of future generation of FE readout IC. Fig. 3 illustrates the modular structure of the DAQ and the control software. We try to limit the interdependencies between each module to reduce system complexity. This in turn increases its robustness and maintainability. The software is designed to be compatible across multiple platforms and has so far been tested on Linux and MAC OSX machines. The control software has two main classes from which other classes are built upon. These two main classes are the Configuration class (CNFG) and Data Acquisition (DAQ) class. The Operation (OPR) and Injection (INJ) sub-classes are derived from the CNFG and DAQ top level classes, respectively. In addition, there are two stand-alone classes that are independent of the other classes; they are the SiTCP Controller and Data Output (DOUT) classes. As in the firmware, there is a SiTCP Controller class which will establish connection between a PC and the hardware chain. The CMD_Generator module in the CNFG class generates all the “Slow”
Fig. 4. The schematic diagram of the analog pixel of FE-I4, taken from Ref. [5].
240
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
commands and two of the “Fast” commands of FE-I4. Moreover, this module also generates firmware control commands. On the other hand, the Configuration module contains all the configuration routines which conduct an initialization of FE-I4 and intervene between various scanning operations. The calibration and tuning routines contained in the OPR sub-class are responsible to set a uniform parameter response for every pixel. The DAQ class has three major tasks. First is to initiate data taking, second is to handle data storage and third is decoding. The first two tasks are taken care of by the Injection and Data_Acquisition module. As SiTCP adopts only a minimum set of communication protocols, the data taking sequence uses only a handful of standard socket functions without special tuning. The decoder on the other hand is in charge of identifying specific data frames based on the FE-I4 protocol and extracting 24-bit record out of each 30-bit word of the 8b/10b encoded data stream. The primary output format is the ROOT file. By using analysis tools provided in the DOUT class, the user can easily generate various plots or histograms for data interpretation.
will register a hit. The hit record can then be examined in the output 2-D histogram. As can be seen in the hit map shown in Fig. 5, we were able to configure FE-I4 correctly to achieve any desired hit pattern. On top of that, the fact that the hit data matched the selected pixels proved that 8b/10b encoded data from FE-I4 can be decoded properly. To see if the charges can be injected correctly and to test various components of the pixel analog circuit (preamp, shaper, etc.) as well as to test one of the scanning routines, we have carried out the so-called Injected Charge Scan. In the Injected Charge Scan, different amount of charges are injected into the
Injected Charge Scan 1 χ2 / ndf 0.8
1.643 / 1
Prob
0.1999
Threshold 1391 ± 22.5 ENC 5. Test results
170.4 ± 22.1
FE-I4 contains self-calibration circuitries which can either generate digital strobe signals or inject charges. Referring to Fig. 4, the digital strobe signal can be sent via the “DigHit” port while the charges can be injected via two capacitors (CInj1 and CInj2). By exploiting this self-calibration capability we can show that our readout system can configure and read out FE-I4 correctly. We have performed several tests ranging from a basic digital injection test to a full-fledge register tuning. All the tests were performed on a single chip setup with the FE-I4 version A. More information about how to operate FE-I4 can be found in Ref. [5]. The Digital Injection Test is the simplest test for demonstrating that the configuration can be done properly. At the same time it can also test the decoder. In this test the digital strobe signal is sent directly into the pixel's digital readout circuitry to simulate a hit event. If the configuration is done properly, the selected pixels
Eff.
0.6
0.4
0.2
0 0
2000
4000 6000 # injected e-
Occupancy: Digital Injection 100
100
90
90
300
80
80
250
250 70
70
60 50
150
60
200 row
row
200
50 150
40 30
100
40 30
100
20 50
20 50
10
10
0 10 20 30 40 50 col
60 70 80
10000
Fig. 6. The Injected Charge Scan. We extracted the threshold of each pixel by injecting different amounts of charges into the pixel analog circuit multiple times and fit the distribution with the error function.
Occupancy: Digital Injection
300
8000
10 20 30 40 50 col
60 70 80
0
Fig. 5. Occupancy plot of Digital Injection Test. Different ways of pixel selection for testing can be achieved if the registers are written correctly.
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
Threshold Distribution (Before Threshold_Tune) h1 Entries 25506
1000
Mean
2078
RMS
419.4
#entries
800
600
400
200
0 1000
1500
2000
2500
3000
3500
4000
Threshold (e-) Threshold Distribution (After Threshold_Tune) 6000
h2 Entries 25506
5000
Mean
2000
RMS
59.54
241
then yields the pixel's threshold and the noise value. Fig. 6 shows the typical Injected Charge Scan distribution and the error function fit from which the threshold and the noise values were extracted. The expected “S-distribution” at certain threshold setting confirmed that charges can be injected correctly and the Injected Charge Scan scanning routine is working as designed. Finally by combining previously mentioned configuration and scanning routines, more complicated tasks such as Threshold Tuning and Time Over Threshold (TOT) Tuning can be performed. The purpose of the Threshold Tuning is to achieve a uniform threshold value across the whole pixel arrays. Similarly, the Time Over Threshold Tuning is aiming at getting a uniform Time Over Threshold response across all pixels for a fixed injected charge. These operations are performed by adjusting the local tuning registers in the analog circuit of each pixel. Taking Threshold Tuning as an example, it is performed by adjusting a 5-bit local threshold trimming register called TDAC (as shown in Fig. 4). Instead of a brute force linear search, we applied a binary search algorithm which can cover all possible 32 values while consuming less time. The average thresholds before and after the tuning are compared in Fig. 7. In this case the target threshold was set to 2000 electrons. After the tuning, the mean threshold clearly converged to the target and the spread was reduced from ∼420 electrons to ∼60 electrons. This result convinced us that our tuning algorithm has been correctly carried out. At the same time, we are convinced that the tuned threshold spread is less than 100 electrons, which meets the FE-I4's specification.
6. Conclusion
#entries
4000 By employing the SEABAS DAQ board, we have successfully developed a readout system for the new ATLAS pixel detector FEI4 readout IC. The basic software and firmware functionalities were tested. The results matched our expectations and satisfied FE-I4's specification. We will proceed with evaluating the performance of the readout system in terms of speed and extending the system to support multi-chip readout.
3000
2000
1000
References
0 1000
1500
2000
2500
3000
3500
4000
Threshold (e-) Fig. 7. The threshold distribution before (top) and after (bottom) the threshold tuning. After the threshold tuning, the spread in the threshold was reduced from around 420 electrons to 60 electrons.
pixel's analog circuit and the number of hits are recorded. The resultant hit distribution is fitted with an error function which
[1] T. Uchida, IEEE Transactions on Nuclear Science NS-55 (3) (2008) 1631. [2] P. Vankov, ATLAS Upgrade for the HL-LHC: meeting the challenges of a five-fold increase in collision rate, arXiv:1201.5469v1[physics.ins-det]. [3] M. Garcia-Sciveres, et al., Nuclear Instruments and Methods in Physics Research Section A 636 (2011) 155. [4] M. Barbero, et al., Nuclear Instruments and Methods in Physics Research Section A 604 (2009) 397. [5] FE-I4 Collaboration, The FE-I4A Integrated Circuit Guide, Version 11.6, 2011. [6] T. Uchida, M. Tanaka, Development of a TCP/IP processing hardware, in: IEEE Nuclear Science Symposium, NS33-6, 2006, pp. 1411–1414. [7] SOIPIX group homepage: 〈http://rd.kek.jp/project/soi/〉. [8] The USBpix test system homepage: 〈http://icwiki.physik.uni-bonn.de/twiki/bin/ view/Systems/UsbPix〉.
Contents lists available at ScienceDirect
Nuclear Instruments and Methods in Physics Research A journal homepage: www.elsevier.com/locate/nima
Development of readout system for FE-I4 pixel module using SiTCP J.J. Teoh a,n, K. Hanagaki a, Y. Ikegami b, Y. Takubo b, S. Terada b, Y. Unno b a b
Graduate School of Science, Osaka University, 1-1 Machikaneyama, Toyonaka, Osaka, 560-0043, Japan Institute for Particle and Nuclear Studies, High Energy Accelerator Research Organization (KEK), 1-1 Oho, Tsukuba City, Ibaraki-ken 305-0801, Japan
art ic l e i nf o
a b s t r a c t
Available online 2 June 2013
The ATLAS pixel detector will be replaced in the future High Luminosity-Large Hadron Collider (HL-LHC) upgrade to preserve or improve the detector performance at high luminosity environment. To meet the tight requirements of the upgrade, a new pixel Front-End (FE) Integrated Circuit (IC) called FE-I4 has been developed. We have then devised a readout system for the new FE IC. Our system incorporates Silicon Transmission Control Protocol (SiTCP) technology (Uchida, 2008 [1]) which utilizes the standard TCP/IP and UDP communication protocols. This technology allows direct data access and transfer between a readout hardware chain and PC via a high speed Ethernet. In addition, the communication protocols are small enough to be implemented in a single Field-Programable Gate Array (FPGA). Relying on this technology, we have been able to construct a very compact, versatile and fast readout system. We have developed a firmware and software together with the readout hardware chain. We also have established basic functionalities for reading out FE-I4. & 2013 Elsevier B.V. All rights reserved.
Keywords: HL-LHC ATLAS upgrade Pixel detector FE-I4 SEABAS DAQ SiTCP
1. Introduction Around the year 2022 the LHC will undergo a major upgrade in the so-called HL-LHC project. The ultimate goal is to collect 3000 fb−1 of data by 2030. To achieve this goal, the instantaneous luminosity will have to be increased to 5 1034 cm−2 s−1. This increase in luminosity will greatly enhance the discovery potential of the ATLAS experiment but there is a high price to pay. At this luminosity, the number of pile-up events is expected to reach ∼140 per bunch crossing [2]. This harsher radiation environment and higher event rate pose unprecedented challenges to the detectors especially the pixel detector which is closest to the interaction region. Studies have shown that the current ATLAS pixel detector's FE readout IC, FE-I3 cannot cope with the high hit rate environment, leading to unacceptable detection inefficiency. Thus for the ATLAS experiment, a complete replacement of its pixel detector is foreseen. Motivated by this pixel detector upgrade, a newer generation of FE readout IC which can satisfy the criteria of higher granularity, better radiation hardness and lower material budget has been developed. The new readout IC is called FE-I4 [3,4]. It is the largest readout IC produced to date in an high energy physics experiment. The FE-I4 has a total of 26 880 pixels which is about 10 times more than its predecessor, FE-I3. The pixels are arranged in a 80 336 matrix on a 20 18.6 mm2 chip. The larger chip size leads to more efficient system integration and thus reduces the amount of material per detector layer. Furthermore, the larger chip size also leads to a significant manufacturing cost reduction. The pixel size of 50 250 μm2 is about 40% smaller than its predecessor. This n
Corresponding author. Tel.: +81 6 6850 5359; fax: +81 6 6850 5532. E-mail address: [email protected] (J.J. Teoh).
0168-9002/$ - see front matter & 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.nima.2013.05.161
reduction in pixel size significantly reduces the pile-up inefficiency and improves its position resolution. All input/output (I/O) signals of FE-I4 are in LVDS. The nominal I/O bandwidth of 160 Mb/s is four times higher than FE-I3. The current version of FE-I4 is designed in a 130-nm CMOS technology, but in the future it could be replaced by a higher integration density process. By upgrading to FE-I4, ATLAS aims to achieve the same or even better tracking and vertexing performance than the current pixel detector. Ref. [5] has more detailed information about FE-I4. To read out the new IC, we have developed a new readout system by using a general purpose DAQ board named Soi EvAluation BoArd with Sitcp (SEABAS) [6]. The system will act as an electronic test bench for FE-I4 and provide various functionalities for FE-I4 characterization and performance analysis. Besides acting as the electronic test bench, it can also serve as a DAQ system for a variety of sensor modules such as a silicon planar pixel sensor, 3D silicon sensor and diamond pixel sensor. Furthermore, the readout system must be able to read out multiple FE-I4s (quad-module as currently planned). By using SiTCP technology, we aim to design a very compact, flexible and fast multi-chip readout system. At the current stage, we have developed and tested our system for a single chip readout. In this paper we will give an overview of the readout system interface in Section 2 and the implementation details of the firmware and software in Sections 3 and 4, respectively. The test result will then be presented in Section 5.
2. Readout system interface Our readout system consists of a PC, SEABAS DAQ board, SEABAS-USBpix daughter card, USBpix FE-I4 adapter card and single
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
SiTCP processes the TCP/IP, UDP and Ethernet protocols where users need to only optimize the protocols for their specific application. In other words, the communication and data transmission between PC and SEABAS is entirely taken care of automatically by the SiTCP firmware. This significantly simplifies the development of the readout firmware for any further upgrades. The user FPGA is a Xilinx Virtex-4 FPGA (XC4VLX25) with 1296 kb maximum Block RAM capacity. We have designed the firmware such that it is mainly responsible for forwarding command bit streams to FE-I4 and receiving data outputs from FE-I4. Of all the command types of FE-I4, the firmware only manipulates the Level 1 Trigger Command (LV1) and Calibration Pulse Command (CAL). This is important as the timing of issuing the CAL and LV1 commands needs to be very accurate. Furthermore, this implementation is also meant to ensure maximum flexibility for FE-I4 configuration while saving processing time. The block diagram of the firmware components is shown in Fig. 2. First, in the Top_module the SiTCP Communicator block ensures proper communication between a PC, SiTCP FPGA and User FPGA at all times by taking care of the assignment of user interface signals of the SiTCP at correct timing. System clocks of various frequencies are generated in the Digital Clock Manager
FE-I4
LV1 Trigger, CAL Pulse
FIFOs
Signal_Sender
Signal_Reader
Reset Channels_Manager
Cmd_Decoder
DCM 3. Firmware implementation
SiTCP Communicator
The SiTCP firmware loaded in one of the FPGAs is a hardwarebased TCP/IP processor which is specifically designed and optimized for front-end devices Ref. [1]. It is basically functioning like a synchronous FIFO with a circuit size small enough to be implemented on a single FPGA. This makes the system inherently more flexible than the traditional backplane bus systems such as CAMAC, PCI, VME and ATCA. On top of the fast TCP protocol, a slow control mechanism over User Datagram Protocol (UDP) is also provided.
Single FE-I4 module card
USBpix FE-I4 adapter card
User FPGA SiTCP FPGA SEABAS PC Fig. 2. The block diagram of the firmware for reading out FE-I4.
64 pin connectors RJ45 PHY 100 BASE-T Ethernet
FE-I4
NIM I/O Flat ribbon cable
Configuration Relay
Power
KEL 100pin SEABAS-USBpix daughter card connector
Fig. 1. The simplified schematic of our new readout system (single chip setup).
Power (+5V, -5V)
Top_module
FE-I4 module card. The SEABAS DAQ board was originally developed by the Silicon On Insulator (SOI) collaboration at KEK [7]. Fig. 1 shows an overview of the readout system interface. For a single chip setup, the SEABAS DAQ board is connected to the SEABAS-USBpix daughter card. The SEABAS-USBpix daughter card is then connected to the USBpix FE-I4 adapter card via a KEL 100-pin connector. We have borrowed the USBpix FE-I4 adapter card made by the SiLab of University of Bonn [8] as a temporary solution while waiting for a dedicated adapter card to be made to suit our system. The USBpix FE-I4 adapter card and the single FE-I4 module card are connected by a flat ribbon cable. The flat ribbon cable acts both as a data and power supply line. The SEABAS DAQ board and FE-I4 are independently powered from external power supplies. Finally, an Ethernet cable provides the communication link between computer and the readout hardware chain. In this setup, the SEABAS-USBpix daughter card and the USBpix FE-I4 adapter card are used to route the signals essential for operating FE-I4. In particular, those signal lines include configuration command, reference clock, output data from FE-I4, forced reset signal and various others. All the signals sent from SEABAS to the adapter card are single ended. Remember that FE-I4's I/O signal is in the form of LVDS signal. Thus three I/O signals mentioned above, i.e. the configuration command, the reference clock and the output data from FE-I4 are converted to LVDS signals before being fed into the flat ribbon cable. This signal conversion is performed by the LVDS transceivers located on the adapter card. In addition to the LVDS transceiver, the adapter card also hosts the bias voltage regulators for FE-I4. The SEABAS board has two on-board FPGAs. One FPGA is loaded with the SiTCP firmware; the other affords user's configuration. At the moment we are using SEABAS version 1.1 which employs the 100BASE-T standard with a 100 Mbps line bandwidth. The Ethernet physical layer device (PHY) is linked to the SiTCP with the standard Gigabit Ethernet Media Independent Interface (GMII) specified by IEEE802.3. On top of that, NIM I/O ports could be used to provide external signals such as external trigger signals, external analog pulses and external digital pulses for certain FEI4 operations.
Job_Manager_module
238
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
(DCM). Meanwhile, the CMD_Decoder block identifies headers from the data streams that are sent from the control software and decides what to do next. Each header is associated with a specific task, for example, to activate other modules or to assign values to firmware registers by using a payload extracted from the data stream. On the other hand, the Signal_Sender block in the Job_Manager_module is responsible for forwarding or relaying the command bit streams originated from the software to FE-I4 at 40 MHz
FE-I4
SEABAS
SiTCP Controller
Slow & Fast Cmds
4. Software implementation
CMD Generator Decoder
Injection INJ DAQ
Calibration
Tuning
OPR CNFG
without further encoding. While the other commands are sent from the DAQ software, the LV1 and CAL commands are handled directly by this block. As an example, in a typical calibration or charge injection operation this block will send a CAL command followed by a LV1 command with the predefined trigger interval and the latency value. Once the CAL command is received, FE-I4's internal Pulse Generator circuit will generate digital or analog calibration pulses. The LV1 command will then trigger the acquisition of a new hit or event from FE-I4. After the proper initialization of FE-I4, the Signal_Reader block will continuously monitor the input port for any incoming data from FE-I4. Four asynchronous FIFOs are embedded in this block. The FIFOs will read the incoming 10-bit data words at 20 MHz while writing them out at 25 MHz. The 25 MHz readout clock is optimized for the SEABAS 100BASE-T Ethernet version in order to fully utilize the available bandwidth. In the case of multi-chip readout, the FIFO to be read out is automatically determined by the Channel_Manager block. At the moment, however, only one FIFO is used to read out a single FE-I4 by masking the other FIFOs. Lastly, the Reset block is responsible for resetting all registers within the Job_Manager_module and Top_module to their initial values. The reset signal is issued either after each data taking cycle or upon receiving a reset command from the control software.
Data Acquisition
Firmware Ctrl Cmd
Configuration
239
Analysis Tool DOUT
DAQ & Control Software Fig. 3. The simplified block diagram of the software implemented in our readout system.
The development of the DAQ software is mostly decoupled from the development of the FPGA firmware. This allows a much more efficient way of upgrading the system to suit the needs of future generation of FE readout IC. Fig. 3 illustrates the modular structure of the DAQ and the control software. We try to limit the interdependencies between each module to reduce system complexity. This in turn increases its robustness and maintainability. The software is designed to be compatible across multiple platforms and has so far been tested on Linux and MAC OSX machines. The control software has two main classes from which other classes are built upon. These two main classes are the Configuration class (CNFG) and Data Acquisition (DAQ) class. The Operation (OPR) and Injection (INJ) sub-classes are derived from the CNFG and DAQ top level classes, respectively. In addition, there are two stand-alone classes that are independent of the other classes; they are the SiTCP Controller and Data Output (DOUT) classes. As in the firmware, there is a SiTCP Controller class which will establish connection between a PC and the hardware chain. The CMD_Generator module in the CNFG class generates all the “Slow”
Fig. 4. The schematic diagram of the analog pixel of FE-I4, taken from Ref. [5].
240
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
commands and two of the “Fast” commands of FE-I4. Moreover, this module also generates firmware control commands. On the other hand, the Configuration module contains all the configuration routines which conduct an initialization of FE-I4 and intervene between various scanning operations. The calibration and tuning routines contained in the OPR sub-class are responsible to set a uniform parameter response for every pixel. The DAQ class has three major tasks. First is to initiate data taking, second is to handle data storage and third is decoding. The first two tasks are taken care of by the Injection and Data_Acquisition module. As SiTCP adopts only a minimum set of communication protocols, the data taking sequence uses only a handful of standard socket functions without special tuning. The decoder on the other hand is in charge of identifying specific data frames based on the FE-I4 protocol and extracting 24-bit record out of each 30-bit word of the 8b/10b encoded data stream. The primary output format is the ROOT file. By using analysis tools provided in the DOUT class, the user can easily generate various plots or histograms for data interpretation.
will register a hit. The hit record can then be examined in the output 2-D histogram. As can be seen in the hit map shown in Fig. 5, we were able to configure FE-I4 correctly to achieve any desired hit pattern. On top of that, the fact that the hit data matched the selected pixels proved that 8b/10b encoded data from FE-I4 can be decoded properly. To see if the charges can be injected correctly and to test various components of the pixel analog circuit (preamp, shaper, etc.) as well as to test one of the scanning routines, we have carried out the so-called Injected Charge Scan. In the Injected Charge Scan, different amount of charges are injected into the
Injected Charge Scan 1 χ2 / ndf 0.8
1.643 / 1
Prob
0.1999
Threshold 1391 ± 22.5 ENC 5. Test results
170.4 ± 22.1
FE-I4 contains self-calibration circuitries which can either generate digital strobe signals or inject charges. Referring to Fig. 4, the digital strobe signal can be sent via the “DigHit” port while the charges can be injected via two capacitors (CInj1 and CInj2). By exploiting this self-calibration capability we can show that our readout system can configure and read out FE-I4 correctly. We have performed several tests ranging from a basic digital injection test to a full-fledge register tuning. All the tests were performed on a single chip setup with the FE-I4 version A. More information about how to operate FE-I4 can be found in Ref. [5]. The Digital Injection Test is the simplest test for demonstrating that the configuration can be done properly. At the same time it can also test the decoder. In this test the digital strobe signal is sent directly into the pixel's digital readout circuitry to simulate a hit event. If the configuration is done properly, the selected pixels
Eff.
0.6
0.4
0.2
0 0
2000
4000 6000 # injected e-
Occupancy: Digital Injection 100
100
90
90
300
80
80
250
250 70
70
60 50
150
60
200 row
row
200
50 150
40 30
100
40 30
100
20 50
20 50
10
10
0 10 20 30 40 50 col
60 70 80
10000
Fig. 6. The Injected Charge Scan. We extracted the threshold of each pixel by injecting different amounts of charges into the pixel analog circuit multiple times and fit the distribution with the error function.
Occupancy: Digital Injection
300
8000
10 20 30 40 50 col
60 70 80
0
Fig. 5. Occupancy plot of Digital Injection Test. Different ways of pixel selection for testing can be achieved if the registers are written correctly.
J.J. Teoh et al. / Nuclear Instruments and Methods in Physics Research A 731 (2013) 237–241
Threshold Distribution (Before Threshold_Tune) h1 Entries 25506
1000
Mean
2078
RMS
419.4
#entries
800
600
400
200
0 1000
1500
2000
2500
3000
3500
4000
Threshold (e-) Threshold Distribution (After Threshold_Tune) 6000
h2 Entries 25506
5000
Mean
2000
RMS
59.54
241
then yields the pixel's threshold and the noise value. Fig. 6 shows the typical Injected Charge Scan distribution and the error function fit from which the threshold and the noise values were extracted. The expected “S-distribution” at certain threshold setting confirmed that charges can be injected correctly and the Injected Charge Scan scanning routine is working as designed. Finally by combining previously mentioned configuration and scanning routines, more complicated tasks such as Threshold Tuning and Time Over Threshold (TOT) Tuning can be performed. The purpose of the Threshold Tuning is to achieve a uniform threshold value across the whole pixel arrays. Similarly, the Time Over Threshold Tuning is aiming at getting a uniform Time Over Threshold response across all pixels for a fixed injected charge. These operations are performed by adjusting the local tuning registers in the analog circuit of each pixel. Taking Threshold Tuning as an example, it is performed by adjusting a 5-bit local threshold trimming register called TDAC (as shown in Fig. 4). Instead of a brute force linear search, we applied a binary search algorithm which can cover all possible 32 values while consuming less time. The average thresholds before and after the tuning are compared in Fig. 7. In this case the target threshold was set to 2000 electrons. After the tuning, the mean threshold clearly converged to the target and the spread was reduced from ∼420 electrons to ∼60 electrons. This result convinced us that our tuning algorithm has been correctly carried out. At the same time, we are convinced that the tuned threshold spread is less than 100 electrons, which meets the FE-I4's specification.
6. Conclusion
#entries
4000 By employing the SEABAS DAQ board, we have successfully developed a readout system for the new ATLAS pixel detector FEI4 readout IC. The basic software and firmware functionalities were tested. The results matched our expectations and satisfied FE-I4's specification. We will proceed with evaluating the performance of the readout system in terms of speed and extending the system to support multi-chip readout.
3000
2000
1000
References
0 1000
1500
2000
2500
3000
3500
4000
Threshold (e-) Fig. 7. The threshold distribution before (top) and after (bottom) the threshold tuning. After the threshold tuning, the spread in the threshold was reduced from around 420 electrons to 60 electrons.
pixel's analog circuit and the number of hits are recorded. The resultant hit distribution is fitted with an error function which
[1] T. Uchida, IEEE Transactions on Nuclear Science NS-55 (3) (2008) 1631. [2] P. Vankov, ATLAS Upgrade for the HL-LHC: meeting the challenges of a five-fold increase in collision rate, arXiv:1201.5469v1[physics.ins-det]. [3] M. Garcia-Sciveres, et al., Nuclear Instruments and Methods in Physics Research Section A 636 (2011) 155. [4] M. Barbero, et al., Nuclear Instruments and Methods in Physics Research Section A 604 (2009) 397. [5] FE-I4 Collaboration, The FE-I4A Integrated Circuit Guide, Version 11.6, 2011. [6] T. Uchida, M. Tanaka, Development of a TCP/IP processing hardware, in: IEEE Nuclear Science Symposium, NS33-6, 2006, pp. 1411–1414. [7] SOIPIX group homepage: 〈http://rd.kek.jp/project/soi/〉. [8] The USBpix test system homepage: 〈http://icwiki.physik.uni-bonn.de/twiki/bin/ view/Systems/UsbPix〉.