The European VLBI Network MkIV Data Processor

New Astronomy Reviews 43 (1999) 503–508 www.elsevier.nl / locate / newar

The European VLBI Network MkIV Data Processor J.L. Casse a a

Joint Institute for VLBI in Europe, Postbus 2, 7990 AA Dwingeloo, The Netherlands

Abstract This document describes in broad terms the EVN MkIV Data Processor at JIVE which was officially opened on 22 October 1998.  1999 Elsevier Science B.V. All rights reserved. PACS: 95.55.Br; 95.55.Jz; 95.55.Sh Keywords: Instrumentation: interferometers; Methods: data analysis

1. Introduction The EVN MkIV Data Processor allows the simultaneous correlation of data from up to 16 stations, i.e. 120 baselines. The Data Playback Units (DPU) are compatible with the MkIIIA, VLBA and MkIV data formats. Their maximum data playback rate is 1 Gbit / s / station with 2 headstacks. The correlator uses the X-F algorithm and is based on a new VLSI correlator chip. The architecture is station based, i.e. all station-based parameters are embedded in the data stream. The Data Processor is real-time compatible and has full space-VLBI compatibility. It has a recirculation option for improved spectral-line processing efficiency and a gating option for pulsar observations. Fig. 1 shows the overall block diagram of the EVN MkIV Data Processor at JIVE. Each DPU is equipped with 2 headstacks yielding 32 tracks each able to replay data at maximum 16 Mbits / s / track i.e. 8 MHz per track, yielding a total of 1024 Mbits / s per DPU. The data from the DPU passes on to the Station Unit (SU), which reconstructs (i.e. de-multiplexes, de-barrel-rolls, etc.) the data streams as required and passes the reconstructed data streams to the Data Distributor. Data transfer takes place via

high frequency link modules (Serial Links). A central clock in the Test Synchronisation & Pulsar gating Unit (TSPU) synchronises the data streams. The correlation process is controlled asynchronously via Ethernet by the Correlator Control Computer (CCC) which sends control messages to the real time processors in the SUs, TSPU, Data Distributor Unit (DDU) and Correlator Unit. The diagram also shows a ‘‘Tape Handling System’’ which is in fact a vertical storage conveyer system which is used to move the tapes between ground level and the basement where the data processor is located, and at the same time provides storage space for some 2000 tapes in their protection boxes. The block diagram includes the three work stations used to control the entire correlation process. The design and prototyping of the hardware and the software for the EVN MkIV Data Processor has been an international endeavour which has taken place under the auspices of the International Advanced Correlator Consortium (IACC). Within the IACC, the design tasks were equally divided between the European and US groups participating in the Consortium, the Joint Institute for VLBI in Europe (JIVE), the Netherlands Foundation for Research in Astronomy (NFRA) and the NASA

1387-6473 / 99 / $ – see front matter  1999 Elsevier Science B.V. All rights reserved. PII: S1387-6473( 99 )00042-1

504

J.L. Casse / New Astronomy Reviews 43 (1999) 503 – 508

Fig. 1. A block diagram of the EVN MkIV Data Processor.

Goddard Space Flight Center representing itself, the Haystack Observatory, the Smithsonian Center for Astrophysics and the University of New Mexico. The design of the Station Unit, the responsibility of JIVE, was contracted to Penny & Giles Data Systems (P&G, now Metrum Information Storage Ltd) in the UK. The Correlator hardware was primarily a task for the Haystack Observatory with a contribution from NFRA. The custom made correlator chip was contracted out to the NASA Space Engineering Research Center (SERC) for VLSI System Design at the University of New Mexico. The Station Unit Interface Module (SUIM) which links the Station Units to the correlator and the central clock system and also provides test and pulsar gating waveforms (TSPU) was the responsibility of JIVE in Dwingeloo with most of the work for the design and replication subcontracted to the Institute of Radio Astronomy (IRA) at Bologna. The Data Distributor which links the Station Units to the Correlator module is the responsibility of NFRA while the DPU were contracted to P&G. The high level control software was contracted to the Nuffield Radio Astronomy Laboratories (NRAL) at Jodrell Bank.

2. Data playback and station units The DPUs (shown in Fig. 2) feature five tape speeds: 80, 160, 320, 330, 135, 270 ips and as well

as a VLBA control instruction set. The DPU differs from the Honeywell 96 / MkIV system in that it does not use vacuum to provide the tape tension but spring-loaded tension arms. The guiding scheme is also different, although it is also based on tape-edge guiding. The DPUs have all been equipped with triple cap headstacks at both head positions. The SUs are responsible for reconstructing the raw data from the DPU (up to 64 tracks delivering data at a maximum rate of 16 Mbps). The SU must finally produce channelised data compatible with the format of the correlator units. Each SU fits in one crate that houses 17 boards. This compact arrangement has been made possible through the extensive use of Field Programmable Gate Arrays (FPGA) and local intelligence on most of the boards. The 64 output signals from the DPU can be flexibly routed to the Track Recovery Modules where the clock is first recovered from each of the received 64 tracks of data. The clock-recovered data is then decoded for each track by an FPGA which detects embedded sync words, validates CRC, and counts parity errors. The header data is analysed by an embedded microprocessor, and the data itself is placed in a large RAM area (524 3 9 kbit RAM per track). The track data can then be loaded synchronously into the Channel Recovery Module where it is converted from track data into channel data and the applied modulation removed. This process undoes multiplexing or barrel-rolling if applied during data acquisition. The outputs of the de-formatter module

J.L. Casse / New Astronomy Reviews 43 (1999) 503 – 508

505

Fig. 2. The JIVE Data Processor control room. The row of 16 DPUs and SUs can be seen in the background.

are then replicas of the sample streams from each recorded video-channel, and are at rates up to 32 Msamples / s each. For 2-bit sample data, the sign and magnitude data streams are treated independently. Phase calibration data is extracted in the Phase Calibration Module and read by the CCC. Each data-stream is then delayed, relative to the time written on the tape in the Delay Memory Module according to a polynomial model (optionally derived from either the CALC or GLORIA software packages) provided by the Correlator Control Computer. The Delay Memory Modules are controlled by a Delay Control Module. The parameters of the delay and phase model used by the Station Unit are inserted into the data stream at the beginning of each correlator frame for use by the correlator module. The last board in the SU is the Station Unit Interface Module. This module receives the system synchronisation signals, test frames and pulsar gate waveforms from the TSPU. It can inject test data into the input of the SU, perform pulsar gating of the channel data and calculate the level statistical counts for all channel streams. It transmits the SU output data to the Data Distributor via 2 multiplexed high speed data links (Serial Links). This construction has been adopted in order to secure coherence between all the data channels during their transport between units. In the Serial Links, the data from 8 channels, defined as sign, magnitude and validity bit lines, are multiplexed in an FPGA and fed to the ‘‘transmit

chip’’ together with a fourth line which provides the timing for the correlator with the Beginning Of Correlator Frame (BOCF) signal common to all units. The header contains delay and phase control parameters for the next data frame. These parameters are captured by the correlator cells and used to control the phase rotators and the vernier delays. The output from each serial link module consists of eight 1 or 2-bit data channels at rates up to 32 Msamples / s each. The data is synchronously recovered by the ‘‘receive chip’’ in the correlator. The synchronisation between the various parts of the hardware and the real time software is coordinated by a clock signal supplied by a central reference clock in the TSPU. The TSPU which includes the Real Time System is a separate crate linked to the Correlator Control Computer. It delivers the system synchronization signals at 18 MHz (clock) and 1 pps (SYSSEC); generates 8 independently programmed Pulsar Gate waveforms, reproduces test-patterns in the form of the MkIV formatted tracks (8 independently defined tracks may be reproduced simultaneously) and delivers the Test Frames, Pulsar Gating and Synchronisation signals to the SUs via serial links.

3. Correlator system The Data Distributor distributes the signals from the Station Units to the Correlator Modules. It

506

J.L. Casse / New Astronomy Reviews 43 (1999) 503 – 508

consists largely of cross-bar switches and memory elements. These provide the ‘‘recirculation mode’’ with a factor of up to l6 increase in spectral resolution for better support of spectral line processing with many lags, polarisation processing and multiple field of view processing. The Correlator (Bos, 1993) shown in Fig. 3 consists of 4 crates each populated with 8 correlator boards and a Real Time Processor board in the front and 2 input boards and a control board in the back. The input boards are populated with 8 multiplexed serial links of the receiver type. The control of the correlation process takes place via the on-board processor which communicates via the Ethernet with the control computer. Each correlator board (Goodman, 1993) contains 32 of the VLSI correlator chips designed at the Haystack observatory and implemented at the University of New Mexico. Each board has access to sixty four 2-bit data streams at a rate of 32 Msample / s. With the correlator chips connected in series, each board yields 16384 lags. It supports up to 120 channel baselines, processing at 32 Msamples / s / baseline. The 200 pin VLSI Correlator Chip (Aldrich & Whitney, 1993; Canaris & Aldrich, 1993; Bos et al., 1996) forms the heart of the correlator system. This

full-custom CMOS VLSI chip features 512 lags, which can be rearranged internally into 16 real or 8 complex independent correlator cells with respectively 32 or 16 lags. It operates at 32 MHz clock rate. It fully supports correlation with 2-bits / sample (4level) with an option for a valid bit for each sample (local validity). Each of the 8 complex correlator sections include a buffer that allows capture of station-based processing parameters embedded in the serial-data streams from the Station Units. The data captured in these buffers allows the necessary phase / delay parameters to be computed (in a DSP chip) and applied to the data. The 8 internal complex-correlator cells include a full 32-bit 3-level quadrature phase generator and phase rotator. They also include a vernier-bit-delay and, on each lag, a 24-bit latchable ripple counter. The flexibility offered by the hardware of the correlator module can be characterized by five parameters: the baseband factor B, the polarisation factor P, the recirculation factor R, the number of stations S and the number of spectral frequency points F. Table 1 shows a number of possible configurations allowed by the system (for 16 MHz basebands, global validity i.e. V 5 1, and no recirculation). The relation that holds for the configuration of the correlator is:

Fig. 3. Front view of the correlator unit.

J.L. Casse / New Astronomy Reviews 43 (1999) 503 – 508 Table 1 A number of possible configurations allowed by the MkIV correlator [ stations S

[ pol. channels P

[ basebands B

[ freq. channels F

8

4 1 1 4 1 1 4 1 1

32 64 1 14 28 1 8 32 1

32 64 4096 32 64 1024 32 32 1024

12

16

BPFS 2V ]]] 5 262144. R

(1)

4. Control of the correlation process There are several layers of software which contribute to the control of the correlation process. At the board level, in the Station Unit in particular, microcomputers are used to control the tasks which have been described earlier. On the subsystem level, for instance in the SU, TSPU or Correlator, Real Time Controllers take care of the next level of tasks. At the system level, the correlation process is controlled by the Correlator Control Computer. The software has been written in C 11 using the Object Oriented Methodology. Post correlation software will provide the tools for diagnosing the correlation process. The CCC controls the actions of the Data Processor and acts as the interface to the operators. A Unix workstation has been adopted in order to provide flexibility and for the support of the GUIs. Because Unix does not provide any real-time capability, all time-critical operations are carried out by the real-time processors in the correlator system (e.g., in the Station Units, the Data Distributor and the Correlator). The system needs the ability to run more than one job at a time to make maximum use of the Correlator’s hardware resources. The control of the correlating process is achieved by download-

507

ing a set of commands marked for execution at particular times to the real-time processors before the processing job starts. Once a job is running CCC’s involvement is then largely restricted to handling status messages and some data (phase calibration and sampler statistics). The handling of the correlated data is taken care of in a separate work station, the Data to Disc Distributor. The function of the JIVE Processor Control Software (Noble, 1994a; Noble, 1994b) is to provide the processor with the necessary instructions in order to perform the correlation of a number of tapes using data provided by the stations, data related to the observation and data related to the station hardware (from the data base). In addition, when processing an experiment, there is a number of other layers that handle individual tapes, individual telescopes, and finally the overall control of that job. Knitting the entire system together is the process model and messaging system. One particularly important message is the so-called Correlation Job Descriptor (CJD). This contains all of the information needed in order to process an experiment, and is built up from station logs, schedules and other related parameters. Structurally, this is a complex object. The CJD is passed to all of the software layers involved in processing an experiment, each one adding or extracting the information it needs. Beyond the running of individual jobs, there is the overall control of the Correlator, database handling and other sections of the Control Software. Post-correlation software is required for the distribution and archiving of the correlator output as well as the tools for data inspection. These are being developed under AIPS 11 . A conversion tool capable of transforming the raw correlator data, together with auxiliary data, directly into AIPS 11 (and also Classic AIPS) is currently available. This software must also take care of the archiving and distribution of the correlated data. This software will run on a separate platform called the Evaluation and Export Engine.

Acknowledgements The EVN MkIV Data Processor is the work of many individuals, in the teams located within Europe (JIVE and NFRA in Dwingeloo; NRAL Jodrell Bank

508

J.L. Casse / New Astronomy Reviews 43 (1999) 503 – 508

and the Institute of Radio Astronomy in Bologna) and the USA. The author would like to thank all members of the teams for their contributions which enabled us to run a live demonstration of fringes at the official ceremony. In addition, the author would like to thank Hans Hinteregger at the Haystack Observatory for his help in solving DPU problems and George Peck at the VLBA for helping us with the recording of numerous test tapes.

References Aldrich, W. & Whitney, A.R., 1993, MIT-Haystack Observatory, MkIV memo No. 226.

Bos, A., 1993, EVN Document No. 6. Bos, A., Aldrich, A., & Whitney, A.R., 1996, EVN Document No. 237. Canaris, J. & Aldrich, W., 1993, MkIV memo No. 225. Goodman, J., 1993, MIT-Haystack Observatory, MkIV Memo No. 191. Noble, R., 1994a, EVN Document No 11. Noble, R., 1994b, EVN Document No. 32.