- Email: [email protected]
Configuration of the virtual laboratory for fusion researches in Japan
Fusion Engineering and Design 85 (2010) 637–640
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Configuration of the virtual laboratory for fusion researches in Japan T. Yamamoto ∗ , Y. Nagayama, H. Nakanishi, S. Ishiguro, S. Takami, K. Tsuda, S. Okamura National Institute for Fusion Science, National Institutes of Natural Sciences, 322-6 Oroshi, Toki 509-5292, Japan
a r t i c l e
i n f o
Article history: Available online 27 April 2010 Keywords: SNET Nationwide collaboration Computer security Data transfer ITER
a b s t r a c t A virtual laboratory system for nuclear fusion researches in Japan known as SNET run by the National Institute for Fusion Science has been in development for the past seven years. Twenty-one remote sites have participated in SNET, which reached a speed of 1 Gbps in April 2009. The SNET is a closed network system based on L2 and L3VPN provided by SINET3, which is a national academic network operated by the National Institute of Informatics. SNET has been successfully supporting the remote participation of various sizes and types of experimental equipments and has also been supporting the remote use of a supercomputer. In this paper, we describe the configuration of SNET, which is overcoming the challenges that arise in virtual laboratories; we mainly explain the remote participation in the experiment. Remarks about the remote participation regarding the ITER activity, massive data transfer, and GRID are also discussed. A data transfer experiment between Japan and France was performed, with the average throughput reaching 880 Mbps on 1 Gbps of bandwidth. © 2010 Elsevier B.V. All rights reserved.
1. Introduction
2. Outline of SNET
Information networks have begun to enable the remote use of supercomputers and have made the remote participation of the experimental equipments a reality in recent years. In an ordinary experiment, the equipment and its control room are located at the same site; in the remote participation the location of the user who controls the device is sited far from the experimental equipment. Participants in a large project can be scattered throughout the country. Researchers at the remote sites cannot directly handle the controlled device and cannot communicate face-to-face because of cost constraints. Such research is thus not as smooth as local experiments because of delays. Information technology (IT) resolves those problems in the form of virtual laboratory. The growing use of the Internet has made our daily lives convenient, and its applications, such as e-mail clients and web browsers, are easy to use. However, suitable configurations and techniques are needed in the virtual laboratory. Necessary techniques include the high-speed transport of massive data over the long distances and solid computer security. In this paper, we describe the configuration of SNET, which is the virtual laboratory for fusion research in Japan [1,2]. SNET has been operated by the National Institute for Fusion Science (NIFS) for the past seven years.
2.1. Virtual laboratory
∗ Corresponding author at: National Institutes for Fusion Science, 322-6 Oroshicho, Toki-city, Gifu 509-5292, Japan. Tel.: +81 572 58 2553; fax: +81 572 58 2666. E-mail address: [email protected] (T. Yamamoto). 0920-3796/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.fusengdes.2010.03.051
The virtual laboratory is a system that resolves the requests of recent remote research activities by taking advantage of IT’s characteristics. The virtual laboratory includes facilities, equipments, PC clients, and TV conference systems. Requirements for such research activities are as follows: • The concentration of fusion experimental devices and supercomputers, owing to the high costs incurred by the improvements to enable high performance. • An increase in data from measurement devices with high spatial and time resolutions. The volume of raw data used by supercomputers in calculations is growing rapidly alongside performance improvements. • Larger projects, more collaborators. They may reside around the country or around the world. • Growth in risks surrounding computer security, such as computer viruses and information theft. The advantages of IT are as follows: • The bandwidth of the wide area network (WAN) can grow to over 10 Gbps by means of the Ethernet technology. • Virtual private network (VPN) technology on WAN has been established, which enables dedicated lines in the public network. VPN is a closed network with inherent high security.
638
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
Research collaboration is very active at NIFS. The number of staff at NIFS is about 250, and the number of collaborators around the world is over 1200. The Large Helical Device (LHD) at NIFS is the largest superconducting helical machine for plasma confinement experiments [3]. NIFS also has an attractive supercomputer for research into plasma science [4]. An academic nationwide network in Japan, the Science Information Network (SINET), was provided by the National Institute of Informatics (NII), and each site connected to SINET had a bandwidth of 100 Mbps. SuperSINET was started in January 2002 to connect leading research universities and institutes, with a bandwidth of 1 Gbps having a backbone of 10 Gbps. SuperSINET was able to connect the edge nodes using layer-3 (L3) VPN, Multi-Protocol Label Switching (MPLS-VPN). In July 2007, SINET3, which is an integrated version of SINET and SuperSINET, began operations [5]. The bandwidth of its backbone is up to 40 Gbps and the number of edge nodes of SINET3 has increased to more than 60 sites. SINET3 has recently begun to provide among other services, an layer-1 (L1)/layer-2 (L2) VPN service and a multicast service. 2.2. Progress of SNET Just as SuperSINET was begun, SNET activity was started with NIFS and three universities. The first category of SNET is “LHD experiment remote participation” [6]. Each university is connected to SNET with a bandwidth of 1 Gbps and the LHD network is now connected to SNET with a bandwidth of 5 Gbps based on L3VPN. Because SNET is a closed network, no firewall is installed between remote site networks and LHD network at NIFS. For example, Collaborators at Kyushu University control the ultrashort-pulse reflectometry system that is installed on the LHD, adjusts the position of the transmitter and receiver horn antennas from their laboratory room, collects the measurement data, and reconstructs the density profile [7]. Collaborators at Kyoto University also control the spectrometer system on the LHD and collects CCD data [8]. Their local agency at NIFS adjusts the device only at the start and end of the experimental period. In the financial year (FY) 2005, two categories, “remote use of supercomputer system” and “All Japan Spherical Tokamak (ST) research program,” were added to SNET. The former uses the Plasma Simulator at NIFS; the Plasma Simulator had been NEC SX8, and it was replaced with Hitachi SR16000 in March 2009. The remote site network is connected only to a gateway server for the Plasma Simulator through SNET. The researcher at the remote site first logs into the gateway, submits his/her job to the Plasma Simulator or transfers the data between the remote station and the gateway. “All Japan ST research program” is a key category in SNET. Qshu University Exp. with Steady-State Spherical Tokamak (QUEST), which began operating in June 2008 [9], is located at the Kyushu University. The data acquisition system (DAS) for QUEST is not
present onsite, and the LABCOM data acquisition system [10] for the LHD at NIFS collects data from the QUEST experiment through SNET. Remote sites that belong to this category can request the data from QUEST by accessing the LABCOM. In FY 2008, GAMMA-10 [11] and High Density Plasma Experiment-I (HYPER-I) experimental device [12] were added to the “All Japan ST research program” category. GAMMA-10 is a tandem mirror type of plasma experiment at Tsukuba University. The measurement data from GAMMA-10 are collected by the university’s DAS system, and the data are sent to LABCOM through SNET for distribution to other sites. HYPER-I at NIFS uses an the 80 kW microwave to excite the electron cyclotron waves in the plasma. Remote collaboration of HYPER-I is now in the test phase, and researchers at Kyushu University will soon be able to monitor the status of HYPER-I and discuss the results with collaborators at NIFS in the near future.
2.3. SNET activities and support SNET activities are summarized in Table 1. Year by year, the number of remote sites has increased, and more than 50 researchers have now joined SNET from 21 remote sites, nine universities, and one research institute (April 2009). Currently, SNET offers various research resources for researchers at the remote sites. The operation of SNET is supported by a SNET task team. The team connects the new remote sites to SNET with the cooperation of NII and network administrators of the remote sites. They register media access control (MAC) address of each researcher’s PC with the network device of the remote site. These details are described in Section 3.3. To prevent any problems, the team also monitors network conditions.
3. Configuration of SNET 3.1. Physical layer To add a new remote site to SNET, the network of the collaborators’ laboratory is connected to the edge node device of SINET3. The best method is to use a dedicated line for the connection. This is the most reliable and enables relatively easy troubleshooting. As the distance between the network center and the site of the laboratory is usually more than 100 m, a multi-mode optical fiber is required for 1000BASE-SX (max. 550 m), or a single-mode fiber is required for 1000BASE-LX (max. 5000 m). If the network cable is passed near high-power lines such as an experimental device room, the use of a metal line is prohibited because of electromagnetic induction. The most inexpensive method would be to loan a dedicated line from the university or an institute. However, if such a line does is not available, it is rather expensive to lay down optical fiber and/or metal cable in a campus.
Table 1 Summary of SNET activities. Category
Resource (location)
Type size or spec
Type of VPN
Number of remote sites
LHD experiment remote participation
LHD (NIFS)
Heliotron major/minor radius = 3.7/0.64 m Video
L3VPN L2VPN (Multicast)
10
Remote use of supercomputer system
Plasma Simulator (NIFS)
Supercomputer theoretical speed = 77 TFlops, main memory = 16 TB Spherical Tokamak major/minor radius = 0.64/0.36 m Tandem mirror axial length = 27 m, volume = 180 m3 liner plasma device diameter/axial length = 0.30/2.00 m
L3VPN
6
L2/L3VPN
3
L3VPN
1
L2VPN
1
QUEST (Kyushu Univ.) All Japan ST research program
GAMMA-10 (Tsukuba Univ.) Hyper-I (NIFS)
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
An alternative method of connection is to use a Virtual LAN (VLAN), which constructs a virtual dedicated line in the shared line. We can connect the laboratory’s network and the edge node device of SINET3 by VLAN tagging through the campus network. This method requires the network administrator of the remote site to configure the network device of the campus LAN. This method is economical because no new optical fiber or metal cable is needed. However, troubleshooting is rather complex, and the traffic of the campus network may suffer when SNET uses most of the bandwidth, or vice versa. In both methods, cooperation with the network administrator center at the remote site is very important in building the remote site. Although the VPN is basically a closed network, it is convenient to connect to the Internet for updating the virus definition files of antivirus software or updating the components of the operating system to fix security holes. Collaborators at remote sites prefer to use the FTP, e-mail, and web applications used within their own research. In the case of SNET, access from the Internet to SNET is prohibited, but all of PCs at remote sites are able to connect to the Internet through a firewall at NIFS. 3.2. Network layer SINET3 offers multiple VPN services such as L3 (IP), L2 (Ethernet), and L1VPN. SNET corrently uses L2 and L3VPN. SNET is mainly L3VPN because when SNET was first started, SuperSINET offered only an L3VPN service. L3VPN is convenient to use when the university or the research institute has more than one category, because one L3 switch at the remote site can create multiple VLAN. For example, at Nagoya University, one L3 switch has created three VLAN: two for “LHD experiment remote participation” and one for “remote use of supercomputer system.” Although an L2 switch can handle only one VLAN, the number of end-to-end hops is minimized; an L2VPN is suitable for transferring bulk data. An L2 switch is also more inexpensive than an L3 switch. After SuperSINET was replaced by SINET3, SNET introduced the L2VPN into “All Japan ST research program” and “Multicast trial” in October 2008. In the former case, L2VPN is expected to maintain the high speed of file transfers because it involves fewer hops than L3VPN. The SNET network between NIFS and Kyushu University has been replaced by L2VPN for QUEST. On the other hand, multicasting is a key technology for the virtual laboratory. With “LHD experiment remote participation,” the video projected on the center display in the control room of the LHD showing the summary graph of the preceding shot, is distributed to “e remote sites. Demand for the bandwidth needed to distribute the video may be dramatically reduced by multicasting. 3.3. Security layer The operation policy of SNET is different from that of the local campus network, and each category has its own operation policy. One of the common policies of SNET is that, for any PC intended to be connected to SNET at a remote site, the MAC address should be registered with an L2/L3 switch. This restriction aims to protect the computer network from unknown terminals and prohibit intrusion. Another common policy is to install antivirus software on PCs and perform Windows Update (Microsoft Windows OS) or Software Update (Apple Mac OS X) before connecting to SNET. These restrictions guard remote sites against computer viruses and bots. Connecting portable PCs to SNET is also prohibited, and USB memory should not be used with PCs on SNET, to prevent the spread of computer viruses. A PC at a remote site that is being used for “LHD experiment remote participation” is permitted to connect only to SNET and not to other networks such as the local campus network. On the other
639
hand, a PC at a remote site that is being used for “remote use of supercomputer system” is permitted to connect to its local campus network if the routing information is not exchanged on the PC. This difference arises from the method of connection to the NIFS network. The remote site’s network for “LHD experiment remote participation” is directly connected to the internal network of the LHD at NIFS, and the collaborator can directly use the resources. On the other hand, the remote site’s network for “remote use of supercomputer system” is connected only to the gateway, and the collaborator wanting to use the supercomputer is required to log in initially to the gateway and cannot directly connect to the supercomputer. In the case of “All Japan ST research program,” the remote site network is allowed to connect to the local campus network through a dedicated firewall. The data acquisition servers on SNET at the university should be synchronized with QUEST experiments. This firewall is administrated by the NIFS to ensure computer security. 4. Remarks about the ITER era ITER is a next-generation experimental tokamak reactor and is a key to realizing fusion. After negotiations in 2005, it was decided that the ITER reactor would be constructed in France and that the International Fusion Energy Research Center (IFERC) would be built in Japan within the framework of the ITER Broad Approach (BA) program. The experimental data produced by ITER and the video information at the control room would be sent to ITER parties [13]. If we assume that the amount of diagnostic data produced from one shot of ITER is 1 TB or more, the required bandwidth of the network between ITER and IFERC would be 100 Gbps; the data transfer time is more than 100 s if the effective throughput is 80 Gbps. Jitter is another important issue in attaining high-quality TV conferencing, particularly regarding voice communication. 4.1. LFN problem In the case of Japan, SNET is a good scheme to distribute the massive data obtained in the ITER experiment to relevant universities and institutes [13]. The collaborator at a remote site could request the data from ITER using the same method as with LHD or QUEST. When IFERC joins SNET, the required bandwidth for SNET may be 10 or 40 Gbps because researchers at the universities and research institutes would not request all of the data. However, this is still more than 10 times wider than the current bandwidth of SNET. This is one of the requirements that should be solved before ITER experiments begin. In the case of SNET, SINET3 provides a high-quality network in which there is very low jitter and no packet loss occurs, but the related quality of the campus network remains somewhat uncertain. The overall quality of the SNET network should be clarified to realize massive data transfer and high-quality communication. However, long fat-pipe network (LFN) problem will occur because of the distance between Japan and France [14]. The bandwidth of SNET is basically 1 Gbps, but the real speed of the file transport will not reach 1 Gbps without adequate adjustment, because TCP/IP protocol requires acknowledgement of the received data. A WAN covering a country is sufficiently wide to be affected by LFN problem. For example, file transfer is very slow (under 100 Mbps) between NIFS and Kyushu University, its round-trip time (RTT) being about 30 ms. To overcome this situation, various studies into TCP/IP mechanisms are being conducted. In the case of SNET, Fujitsu WANDIRECTOR A100 (the WAN accelerator), has been introduced, which can sustain a bandwidth of about 300 Mbps. We will also investigate how to maintain high throughput, by testing the several real networks.
640
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
To uncover this problem, a data transfer experiment between Japan and France was performed in June 2009. Each server was connected to a 1 Gbps line, and RTT between Japan (NIFS) and France (ITER IO) was about 320 ms. We were able to transfer more than 1 TB of data in 3 h from memory to memory using a technique that avoids packet loss [15]. The average throughput was 880 Mbps.
activity. The authors would like to thank Prof. K. Hiraki in University of Tokyo for supporting the data transfer experiment between Japan and France. An element of SNET is partly supported by Cyber Science Infrastructure (CSI) development project of NII, and NIFS (NIFS08USNX001 and NIFS08USNN002). References
4.2. GRID GRID is a fundamental technology that enables researchers to use distributed resources. Globus in the United States [16], EGEE in the European Union [17], and ITBL [18] and NAREGI [19] in Japan have been successfully developed and are being used for the sciences such as astronomy, high-energy physics, and fusion research. ITER and ITER-related research needs huge computational resources not only for plasma simulations but also for engineering analysis. Because SNET is based on a closed dedicated network, authentication and authority can be omitted when using remote resources. However, the concept of the DataGRID may be required for handling the massive data obtained from ITER. GRID may be useful when a remote site wishes to join SNET and/or ITER activity via the Internet. Applying GRID to SNET is a future topic of interest. 5. Conclusion SNET is a nationwide network for fusion researches based on VPN technology operated by NIFS. SNET has been successfully supporting various fusion research activities for several years. The experimental devices used are widely ranging from LHD to HYPERI; the type of experimental devices also vary from the helical system on LHD, the spherical tokamak on QUEST, to the tandem mirror on GAMMA-10, and the research methods involve both experiments and simulations. In the near future, ITER would present a large challenge for remote participations around the world. As a first step, 1 Gbps of a data transfer experiment was successfully performed between NIFS in Japan and ITER IO in France. SNET is a catalyst for fusion research activities in Japan and continues to contribute to it in the form of informational logistics. Acknowledgments The authors would like to thank SINET team of NII and the network administrators at the remote sites for maintaining SNET
[1] K. Tsuda, Y. Nagayama, T. Yamamoto, R. Horiuchi, S. Ishiguro, S. Takami, Virtual laboratory for fusion research in Japan, Fusion Eng. Des. 83 (2008) 471– 475. [2] SNET, http://snet.nifs.ac.jp/, February 2009. [3] O. Motojima, H. Yamada, A. Komori, N. Ohyabu, T. Mutoh, O. Kaneko, et al., Extended steady-state and high-beta regimes of net-current free heliotron plasmas in the Large Helical Device, Nucl. Fusion 47 (2007) S668–S676. [4] Plasma Simulator, http://www.ps.nifs.ac.jp/, April 2009. [5] S. Urushidani, J. Matsukata, Next-generation science information network for leading-edge applications, Fusion Eng. Des. 83 (2008) 498– 503. [6] Y. Nagayama, M. Emoto, H. Nakanishi, S. Sudo, S. Imazu, S. Inagaki, et al., Control, data acquisition, data analysis and remote participation in LHD, Fusion Eng. Des. 83 (2008) 170–175. [7] A. Mase, Y. Yokota, K. Uchida, Y. Kogi, N. Ito, T. Tokuzawa, et al., Remote experiment of ultrashort-pulse reflectometry for large helical device plasmas, Rev. Sci. Instrum. 77 (2006) 10E916. [8] A. Iwamae, M. Hayakawa, M. Atake, T. Fujimoto, M. Goto, S. Morita, Polarization resolved H alpha spectra from the large helical device: emission location, temperature, and inward flux of neutral hydrogen, Phys. Plasmas 12 (2005) 042501. [9] M. Hasegawa, K. Hanada, K. Sato, K. Nakamura, H. Zushi, M. Sakamoto, et al., Initial plasma production by townsend avalanche breakdown on QUEST Tokamak, Jpn. J. Appl. Phys. 47 (2008) 287–292. [10] H. Nakanishi, M. Ohsuna, M. Kojima, S. Imazu, M. Nonomura, K. Watanabe, et al., Adaptive data migration scheme with facilitator database and multi-tier distributed storage in LHD, Fusion Eng. Des. 83 (2008) 397–401. [11] M. Inutake, T. Cho, M. Ichimura, K. Ishii, A. Itakura, I. Katanuma, et al., Thermal barrier formation and plasma confinement in the axisymmetrized tandem mirror GAMMA 10, Phys. Rev. Let. 55 (1985) 939–942. [12] M.Y. Tanaka, M. Aramaki, K. Ogiwara, S. Etoh, S. Yoshimura, J. Varanjes, Vortex formation in a plasma interacting with neutral flow, frontiers in modern plasma physics, AIP Conference Proceedings 1061 (2008) 57–65. [13] Y. Nagayama, M. Emoto, Y. Kozaki, H. Nakanishi, S. Sudo, T. Yamamoto, et al., A proposal for the ITER remote participation system in Japan, Fusion Eng. Des. 85 (3–4) (2010) 535–539. [14] T. Yamamoto, Estimation of the advanced TCP/IP algorithms for long distance collaboration, Fusion Eng. Des. 83 (2008) 516–519. [15] K. Hiraki, K. Koizumi, J. Tamatsukuri, T. Yoshino, A. Kato, M. Inaba, Highly efficient data transmission facility through very long distance high-speed networks, in this issue, Fusion Eng. Design, submitted for publication. [16] Globus, http://www.globus.org/. [17] EGEE, http://www.eu-egee.org/. [18] ITBL, http://www.itbl.jp/. [19] NAREGI, http://www.naregi.org/.
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Configuration of the virtual laboratory for fusion researches in Japan T. Yamamoto ∗ , Y. Nagayama, H. Nakanishi, S. Ishiguro, S. Takami, K. Tsuda, S. Okamura National Institute for Fusion Science, National Institutes of Natural Sciences, 322-6 Oroshi, Toki 509-5292, Japan
a r t i c l e
i n f o
Article history: Available online 27 April 2010 Keywords: SNET Nationwide collaboration Computer security Data transfer ITER
a b s t r a c t A virtual laboratory system for nuclear fusion researches in Japan known as SNET run by the National Institute for Fusion Science has been in development for the past seven years. Twenty-one remote sites have participated in SNET, which reached a speed of 1 Gbps in April 2009. The SNET is a closed network system based on L2 and L3VPN provided by SINET3, which is a national academic network operated by the National Institute of Informatics. SNET has been successfully supporting the remote participation of various sizes and types of experimental equipments and has also been supporting the remote use of a supercomputer. In this paper, we describe the configuration of SNET, which is overcoming the challenges that arise in virtual laboratories; we mainly explain the remote participation in the experiment. Remarks about the remote participation regarding the ITER activity, massive data transfer, and GRID are also discussed. A data transfer experiment between Japan and France was performed, with the average throughput reaching 880 Mbps on 1 Gbps of bandwidth. © 2010 Elsevier B.V. All rights reserved.
1. Introduction
2. Outline of SNET
Information networks have begun to enable the remote use of supercomputers and have made the remote participation of the experimental equipments a reality in recent years. In an ordinary experiment, the equipment and its control room are located at the same site; in the remote participation the location of the user who controls the device is sited far from the experimental equipment. Participants in a large project can be scattered throughout the country. Researchers at the remote sites cannot directly handle the controlled device and cannot communicate face-to-face because of cost constraints. Such research is thus not as smooth as local experiments because of delays. Information technology (IT) resolves those problems in the form of virtual laboratory. The growing use of the Internet has made our daily lives convenient, and its applications, such as e-mail clients and web browsers, are easy to use. However, suitable configurations and techniques are needed in the virtual laboratory. Necessary techniques include the high-speed transport of massive data over the long distances and solid computer security. In this paper, we describe the configuration of SNET, which is the virtual laboratory for fusion research in Japan [1,2]. SNET has been operated by the National Institute for Fusion Science (NIFS) for the past seven years.
2.1. Virtual laboratory
∗ Corresponding author at: National Institutes for Fusion Science, 322-6 Oroshicho, Toki-city, Gifu 509-5292, Japan. Tel.: +81 572 58 2553; fax: +81 572 58 2666. E-mail address: [email protected] (T. Yamamoto). 0920-3796/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.fusengdes.2010.03.051
The virtual laboratory is a system that resolves the requests of recent remote research activities by taking advantage of IT’s characteristics. The virtual laboratory includes facilities, equipments, PC clients, and TV conference systems. Requirements for such research activities are as follows: • The concentration of fusion experimental devices and supercomputers, owing to the high costs incurred by the improvements to enable high performance. • An increase in data from measurement devices with high spatial and time resolutions. The volume of raw data used by supercomputers in calculations is growing rapidly alongside performance improvements. • Larger projects, more collaborators. They may reside around the country or around the world. • Growth in risks surrounding computer security, such as computer viruses and information theft. The advantages of IT are as follows: • The bandwidth of the wide area network (WAN) can grow to over 10 Gbps by means of the Ethernet technology. • Virtual private network (VPN) technology on WAN has been established, which enables dedicated lines in the public network. VPN is a closed network with inherent high security.
638
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
Research collaboration is very active at NIFS. The number of staff at NIFS is about 250, and the number of collaborators around the world is over 1200. The Large Helical Device (LHD) at NIFS is the largest superconducting helical machine for plasma confinement experiments [3]. NIFS also has an attractive supercomputer for research into plasma science [4]. An academic nationwide network in Japan, the Science Information Network (SINET), was provided by the National Institute of Informatics (NII), and each site connected to SINET had a bandwidth of 100 Mbps. SuperSINET was started in January 2002 to connect leading research universities and institutes, with a bandwidth of 1 Gbps having a backbone of 10 Gbps. SuperSINET was able to connect the edge nodes using layer-3 (L3) VPN, Multi-Protocol Label Switching (MPLS-VPN). In July 2007, SINET3, which is an integrated version of SINET and SuperSINET, began operations [5]. The bandwidth of its backbone is up to 40 Gbps and the number of edge nodes of SINET3 has increased to more than 60 sites. SINET3 has recently begun to provide among other services, an layer-1 (L1)/layer-2 (L2) VPN service and a multicast service. 2.2. Progress of SNET Just as SuperSINET was begun, SNET activity was started with NIFS and three universities. The first category of SNET is “LHD experiment remote participation” [6]. Each university is connected to SNET with a bandwidth of 1 Gbps and the LHD network is now connected to SNET with a bandwidth of 5 Gbps based on L3VPN. Because SNET is a closed network, no firewall is installed between remote site networks and LHD network at NIFS. For example, Collaborators at Kyushu University control the ultrashort-pulse reflectometry system that is installed on the LHD, adjusts the position of the transmitter and receiver horn antennas from their laboratory room, collects the measurement data, and reconstructs the density profile [7]. Collaborators at Kyoto University also control the spectrometer system on the LHD and collects CCD data [8]. Their local agency at NIFS adjusts the device only at the start and end of the experimental period. In the financial year (FY) 2005, two categories, “remote use of supercomputer system” and “All Japan Spherical Tokamak (ST) research program,” were added to SNET. The former uses the Plasma Simulator at NIFS; the Plasma Simulator had been NEC SX8, and it was replaced with Hitachi SR16000 in March 2009. The remote site network is connected only to a gateway server for the Plasma Simulator through SNET. The researcher at the remote site first logs into the gateway, submits his/her job to the Plasma Simulator or transfers the data between the remote station and the gateway. “All Japan ST research program” is a key category in SNET. Qshu University Exp. with Steady-State Spherical Tokamak (QUEST), which began operating in June 2008 [9], is located at the Kyushu University. The data acquisition system (DAS) for QUEST is not
present onsite, and the LABCOM data acquisition system [10] for the LHD at NIFS collects data from the QUEST experiment through SNET. Remote sites that belong to this category can request the data from QUEST by accessing the LABCOM. In FY 2008, GAMMA-10 [11] and High Density Plasma Experiment-I (HYPER-I) experimental device [12] were added to the “All Japan ST research program” category. GAMMA-10 is a tandem mirror type of plasma experiment at Tsukuba University. The measurement data from GAMMA-10 are collected by the university’s DAS system, and the data are sent to LABCOM through SNET for distribution to other sites. HYPER-I at NIFS uses an the 80 kW microwave to excite the electron cyclotron waves in the plasma. Remote collaboration of HYPER-I is now in the test phase, and researchers at Kyushu University will soon be able to monitor the status of HYPER-I and discuss the results with collaborators at NIFS in the near future.
2.3. SNET activities and support SNET activities are summarized in Table 1. Year by year, the number of remote sites has increased, and more than 50 researchers have now joined SNET from 21 remote sites, nine universities, and one research institute (April 2009). Currently, SNET offers various research resources for researchers at the remote sites. The operation of SNET is supported by a SNET task team. The team connects the new remote sites to SNET with the cooperation of NII and network administrators of the remote sites. They register media access control (MAC) address of each researcher’s PC with the network device of the remote site. These details are described in Section 3.3. To prevent any problems, the team also monitors network conditions.
3. Configuration of SNET 3.1. Physical layer To add a new remote site to SNET, the network of the collaborators’ laboratory is connected to the edge node device of SINET3. The best method is to use a dedicated line for the connection. This is the most reliable and enables relatively easy troubleshooting. As the distance between the network center and the site of the laboratory is usually more than 100 m, a multi-mode optical fiber is required for 1000BASE-SX (max. 550 m), or a single-mode fiber is required for 1000BASE-LX (max. 5000 m). If the network cable is passed near high-power lines such as an experimental device room, the use of a metal line is prohibited because of electromagnetic induction. The most inexpensive method would be to loan a dedicated line from the university or an institute. However, if such a line does is not available, it is rather expensive to lay down optical fiber and/or metal cable in a campus.
Table 1 Summary of SNET activities. Category
Resource (location)
Type size or spec
Type of VPN
Number of remote sites
LHD experiment remote participation
LHD (NIFS)
Heliotron major/minor radius = 3.7/0.64 m Video
L3VPN L2VPN (Multicast)
10
Remote use of supercomputer system
Plasma Simulator (NIFS)
Supercomputer theoretical speed = 77 TFlops, main memory = 16 TB Spherical Tokamak major/minor radius = 0.64/0.36 m Tandem mirror axial length = 27 m, volume = 180 m3 liner plasma device diameter/axial length = 0.30/2.00 m
L3VPN
6
L2/L3VPN
3
L3VPN
1
L2VPN
1
QUEST (Kyushu Univ.) All Japan ST research program
GAMMA-10 (Tsukuba Univ.) Hyper-I (NIFS)
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
An alternative method of connection is to use a Virtual LAN (VLAN), which constructs a virtual dedicated line in the shared line. We can connect the laboratory’s network and the edge node device of SINET3 by VLAN tagging through the campus network. This method requires the network administrator of the remote site to configure the network device of the campus LAN. This method is economical because no new optical fiber or metal cable is needed. However, troubleshooting is rather complex, and the traffic of the campus network may suffer when SNET uses most of the bandwidth, or vice versa. In both methods, cooperation with the network administrator center at the remote site is very important in building the remote site. Although the VPN is basically a closed network, it is convenient to connect to the Internet for updating the virus definition files of antivirus software or updating the components of the operating system to fix security holes. Collaborators at remote sites prefer to use the FTP, e-mail, and web applications used within their own research. In the case of SNET, access from the Internet to SNET is prohibited, but all of PCs at remote sites are able to connect to the Internet through a firewall at NIFS. 3.2. Network layer SINET3 offers multiple VPN services such as L3 (IP), L2 (Ethernet), and L1VPN. SNET corrently uses L2 and L3VPN. SNET is mainly L3VPN because when SNET was first started, SuperSINET offered only an L3VPN service. L3VPN is convenient to use when the university or the research institute has more than one category, because one L3 switch at the remote site can create multiple VLAN. For example, at Nagoya University, one L3 switch has created three VLAN: two for “LHD experiment remote participation” and one for “remote use of supercomputer system.” Although an L2 switch can handle only one VLAN, the number of end-to-end hops is minimized; an L2VPN is suitable for transferring bulk data. An L2 switch is also more inexpensive than an L3 switch. After SuperSINET was replaced by SINET3, SNET introduced the L2VPN into “All Japan ST research program” and “Multicast trial” in October 2008. In the former case, L2VPN is expected to maintain the high speed of file transfers because it involves fewer hops than L3VPN. The SNET network between NIFS and Kyushu University has been replaced by L2VPN for QUEST. On the other hand, multicasting is a key technology for the virtual laboratory. With “LHD experiment remote participation,” the video projected on the center display in the control room of the LHD showing the summary graph of the preceding shot, is distributed to “e remote sites. Demand for the bandwidth needed to distribute the video may be dramatically reduced by multicasting. 3.3. Security layer The operation policy of SNET is different from that of the local campus network, and each category has its own operation policy. One of the common policies of SNET is that, for any PC intended to be connected to SNET at a remote site, the MAC address should be registered with an L2/L3 switch. This restriction aims to protect the computer network from unknown terminals and prohibit intrusion. Another common policy is to install antivirus software on PCs and perform Windows Update (Microsoft Windows OS) or Software Update (Apple Mac OS X) before connecting to SNET. These restrictions guard remote sites against computer viruses and bots. Connecting portable PCs to SNET is also prohibited, and USB memory should not be used with PCs on SNET, to prevent the spread of computer viruses. A PC at a remote site that is being used for “LHD experiment remote participation” is permitted to connect only to SNET and not to other networks such as the local campus network. On the other
639
hand, a PC at a remote site that is being used for “remote use of supercomputer system” is permitted to connect to its local campus network if the routing information is not exchanged on the PC. This difference arises from the method of connection to the NIFS network. The remote site’s network for “LHD experiment remote participation” is directly connected to the internal network of the LHD at NIFS, and the collaborator can directly use the resources. On the other hand, the remote site’s network for “remote use of supercomputer system” is connected only to the gateway, and the collaborator wanting to use the supercomputer is required to log in initially to the gateway and cannot directly connect to the supercomputer. In the case of “All Japan ST research program,” the remote site network is allowed to connect to the local campus network through a dedicated firewall. The data acquisition servers on SNET at the university should be synchronized with QUEST experiments. This firewall is administrated by the NIFS to ensure computer security. 4. Remarks about the ITER era ITER is a next-generation experimental tokamak reactor and is a key to realizing fusion. After negotiations in 2005, it was decided that the ITER reactor would be constructed in France and that the International Fusion Energy Research Center (IFERC) would be built in Japan within the framework of the ITER Broad Approach (BA) program. The experimental data produced by ITER and the video information at the control room would be sent to ITER parties [13]. If we assume that the amount of diagnostic data produced from one shot of ITER is 1 TB or more, the required bandwidth of the network between ITER and IFERC would be 100 Gbps; the data transfer time is more than 100 s if the effective throughput is 80 Gbps. Jitter is another important issue in attaining high-quality TV conferencing, particularly regarding voice communication. 4.1. LFN problem In the case of Japan, SNET is a good scheme to distribute the massive data obtained in the ITER experiment to relevant universities and institutes [13]. The collaborator at a remote site could request the data from ITER using the same method as with LHD or QUEST. When IFERC joins SNET, the required bandwidth for SNET may be 10 or 40 Gbps because researchers at the universities and research institutes would not request all of the data. However, this is still more than 10 times wider than the current bandwidth of SNET. This is one of the requirements that should be solved before ITER experiments begin. In the case of SNET, SINET3 provides a high-quality network in which there is very low jitter and no packet loss occurs, but the related quality of the campus network remains somewhat uncertain. The overall quality of the SNET network should be clarified to realize massive data transfer and high-quality communication. However, long fat-pipe network (LFN) problem will occur because of the distance between Japan and France [14]. The bandwidth of SNET is basically 1 Gbps, but the real speed of the file transport will not reach 1 Gbps without adequate adjustment, because TCP/IP protocol requires acknowledgement of the received data. A WAN covering a country is sufficiently wide to be affected by LFN problem. For example, file transfer is very slow (under 100 Mbps) between NIFS and Kyushu University, its round-trip time (RTT) being about 30 ms. To overcome this situation, various studies into TCP/IP mechanisms are being conducted. In the case of SNET, Fujitsu WANDIRECTOR A100 (the WAN accelerator), has been introduced, which can sustain a bandwidth of about 300 Mbps. We will also investigate how to maintain high throughput, by testing the several real networks.
640
T. Yamamoto et al. / Fusion Engineering and Design 85 (2010) 637–640
To uncover this problem, a data transfer experiment between Japan and France was performed in June 2009. Each server was connected to a 1 Gbps line, and RTT between Japan (NIFS) and France (ITER IO) was about 320 ms. We were able to transfer more than 1 TB of data in 3 h from memory to memory using a technique that avoids packet loss [15]. The average throughput was 880 Mbps.
activity. The authors would like to thank Prof. K. Hiraki in University of Tokyo for supporting the data transfer experiment between Japan and France. An element of SNET is partly supported by Cyber Science Infrastructure (CSI) development project of NII, and NIFS (NIFS08USNX001 and NIFS08USNN002). References
4.2. GRID GRID is a fundamental technology that enables researchers to use distributed resources. Globus in the United States [16], EGEE in the European Union [17], and ITBL [18] and NAREGI [19] in Japan have been successfully developed and are being used for the sciences such as astronomy, high-energy physics, and fusion research. ITER and ITER-related research needs huge computational resources not only for plasma simulations but also for engineering analysis. Because SNET is based on a closed dedicated network, authentication and authority can be omitted when using remote resources. However, the concept of the DataGRID may be required for handling the massive data obtained from ITER. GRID may be useful when a remote site wishes to join SNET and/or ITER activity via the Internet. Applying GRID to SNET is a future topic of interest. 5. Conclusion SNET is a nationwide network for fusion researches based on VPN technology operated by NIFS. SNET has been successfully supporting various fusion research activities for several years. The experimental devices used are widely ranging from LHD to HYPERI; the type of experimental devices also vary from the helical system on LHD, the spherical tokamak on QUEST, to the tandem mirror on GAMMA-10, and the research methods involve both experiments and simulations. In the near future, ITER would present a large challenge for remote participations around the world. As a first step, 1 Gbps of a data transfer experiment was successfully performed between NIFS in Japan and ITER IO in France. SNET is a catalyst for fusion research activities in Japan and continues to contribute to it in the form of informational logistics. Acknowledgments The authors would like to thank SINET team of NII and the network administrators at the remote sites for maintaining SNET
[1] K. Tsuda, Y. Nagayama, T. Yamamoto, R. Horiuchi, S. Ishiguro, S. Takami, Virtual laboratory for fusion research in Japan, Fusion Eng. Des. 83 (2008) 471– 475. [2] SNET, http://snet.nifs.ac.jp/, February 2009. [3] O. Motojima, H. Yamada, A. Komori, N. Ohyabu, T. Mutoh, O. Kaneko, et al., Extended steady-state and high-beta regimes of net-current free heliotron plasmas in the Large Helical Device, Nucl. Fusion 47 (2007) S668–S676. [4] Plasma Simulator, http://www.ps.nifs.ac.jp/, April 2009. [5] S. Urushidani, J. Matsukata, Next-generation science information network for leading-edge applications, Fusion Eng. Des. 83 (2008) 498– 503. [6] Y. Nagayama, M. Emoto, H. Nakanishi, S. Sudo, S. Imazu, S. Inagaki, et al., Control, data acquisition, data analysis and remote participation in LHD, Fusion Eng. Des. 83 (2008) 170–175. [7] A. Mase, Y. Yokota, K. Uchida, Y. Kogi, N. Ito, T. Tokuzawa, et al., Remote experiment of ultrashort-pulse reflectometry for large helical device plasmas, Rev. Sci. Instrum. 77 (2006) 10E916. [8] A. Iwamae, M. Hayakawa, M. Atake, T. Fujimoto, M. Goto, S. Morita, Polarization resolved H alpha spectra from the large helical device: emission location, temperature, and inward flux of neutral hydrogen, Phys. Plasmas 12 (2005) 042501. [9] M. Hasegawa, K. Hanada, K. Sato, K. Nakamura, H. Zushi, M. Sakamoto, et al., Initial plasma production by townsend avalanche breakdown on QUEST Tokamak, Jpn. J. Appl. Phys. 47 (2008) 287–292. [10] H. Nakanishi, M. Ohsuna, M. Kojima, S. Imazu, M. Nonomura, K. Watanabe, et al., Adaptive data migration scheme with facilitator database and multi-tier distributed storage in LHD, Fusion Eng. Des. 83 (2008) 397–401. [11] M. Inutake, T. Cho, M. Ichimura, K. Ishii, A. Itakura, I. Katanuma, et al., Thermal barrier formation and plasma confinement in the axisymmetrized tandem mirror GAMMA 10, Phys. Rev. Let. 55 (1985) 939–942. [12] M.Y. Tanaka, M. Aramaki, K. Ogiwara, S. Etoh, S. Yoshimura, J. Varanjes, Vortex formation in a plasma interacting with neutral flow, frontiers in modern plasma physics, AIP Conference Proceedings 1061 (2008) 57–65. [13] Y. Nagayama, M. Emoto, Y. Kozaki, H. Nakanishi, S. Sudo, T. Yamamoto, et al., A proposal for the ITER remote participation system in Japan, Fusion Eng. Des. 85 (3–4) (2010) 535–539. [14] T. Yamamoto, Estimation of the advanced TCP/IP algorithms for long distance collaboration, Fusion Eng. Des. 83 (2008) 516–519. [15] K. Hiraki, K. Koizumi, J. Tamatsukuri, T. Yoshino, A. Kato, M. Inaba, Highly efficient data transmission facility through very long distance high-speed networks, in this issue, Fusion Eng. Design, submitted for publication. [16] Globus, http://www.globus.org/. [17] EGEE, http://www.eu-egee.org/. [18] ITBL, http://www.itbl.jp/. [19] NAREGI, http://www.naregi.org/.