- Email: [email protected]
Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Fusion Engineering and Design xxx (2016) xxx–xxx
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST Yao Huang a,∗ , B.J. Xiao a,b , Z.P. Luo a , Q.P. Yuan a , X.F. Pei a , X.N. Yue b a b
Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, China School of Nuclear Science & Technology, University of Science & Technology of China, China
h i g h l i g h t s • • • • •
We described parallel equilibrium reconstruction code P-EFIT running on GPU was integrated with EAST plasma control system. Compared with RT-EFIT used in EAST, P-EFIT has better spatial resolution and full algorithm of EFIT per iteration. With the data interface through RFM, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements for plasma control. Successful control using ISOFLUX/P-EFIT was established in the dedicated experiment during the EAST 2014 campaign. This work is a stepping-stone towards versatile ISOFLUX/P-EFIT control, such as real-time equilibrium reconstruction with more diagnostics.
a r t i c l e
i n f o
Article history: Received 19 June 2015 Received in revised form 23 January 2016 Accepted 15 February 2016 Available online xxx Keywords: Equilibrium reconstruction Plasma control EAST GPU parallel computation
a b s t r a c t Implementation of P-EFIT code for plasma control in EAST is described. P-EFIT is based on the EFIT framework, but built with the CUDATM architecture to take advantage of massively parallel Graphical Processing Unit (GPU) cores to significantly accelerate the computation. 65 × 65 grid size P-EFIT can complete one reconstruction iteration in 300 s, with one iteration strategy, it can satisfy the needs of real-time plasma shape control. Data interface between P-EFIT and PCS is realized and developed by transferring data through RFM. First application of P-EFIT to discharge control in EAST is described. © 2016 Elsevier B.V. All rights reserved.
1. Introduction Plasma equilibrium reconstruction is an important tool for tokamak data analysis. Based on experimental measurements, the plasma shape and current profile can be evaluated. The real-time plasma equilibrium reconstruction is routinely used for the plasma control. The essential requirement of optimum performance of plasma feedback control is to obtain the accurate and fast equilibrium reconstruction results. Various equilibrium reconstruction numerical codes had been developed and implemented on different tokamak. One of the most widely used codes is EFIT [1–3]. Unforturnatly, EFIT is only fit for offline calculation and between shot analysis. Its real-time version, called RT-EFIT [4] code is currently used in DIIID, EAST, KSTAR, NSTX and MAST [5–8] as a
∗ Corresponding author. E-mail address: [email protected] (Y. Huang).
routine real-time plasma equilibrium reconstruction for plasma feedback control. The RT-EFIT code makes many simplifications based on offline EFIT and the spatial resolution is relatively low. For example, in EAST the RT-EFIT scales the 2.4 m × 1.4 m rectangle plasma calculation area by 33 × 33 grids, where the offline EFIT typically scales this area by 129 × 129 grids. In order to satisfy the accurate and fast reconstruction of plasma equilibrium, a new numerical code was developed based on Graphic Processing Units (GPU), named as P-EFIT [9]. As a new high performance computing method, GPU has been applied in plasma control in recent years, a plasma control system which utilizes GPUs is designed to magnetically control the 3D perturbed equilibrium state of the plasma in HBT-EP tokamak [10]. P-EFIT took the advantage of massively parallel processing cores in GPU to accelerate the whole reconstruction process in offline EFIT. Comparison with RT-EFIT using multiple CPUs only, P-EFIT utilized the CPU and GPU with heterogeneous parallel computing provided by NVIDIA CUDA software environment.
http://dx.doi.org/10.1016/j.fusengdes.2016.02.048 0920-3796/© 2016 Elsevier B.V. All rights reserved.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
2
Results from EAST single static equilibrium benchmark tests indicate that P-EFIT could accurately reproduce the EFIT reconstruction algorithms at a fraction cost time of the computation. With a 65 × 65 grid size, P-EFIT can complete one reconstruction iteration within 300 s [9], which shows good capability for real-time plasma feedback control. By cutting the iteration number into one or two, P-EFIT can be used directly to control the plasma shape based on ISOFLUX/P-EFIT algorithm. Successful control was carried out in the dedicated experiment during the EAST 2014 campaign. The basic plasma equilibrium reconstruction algorithm and its implementation for parallel running on GPU are described in Section 2. System integration for P-EFIT and Plasma Control System (PCS) is presented in Section 3. Experimental application of P-EFIT in EAST plasma shape control is discussed in Section 4.
2. Introduction on parallel equilibrium reconstruction algorithms 2.1. Basic plasma equilibrium reconstruction algorithm in EFIT
0 FF ( ) 42 R
FF ( ) =
˛n
n=0 nF −1
n Nn
n − ıN F
n=0
(2)
n
n=0
where the normalized flux N = ( − 0 )/(1 − 0 ) varies between 0 and 1 with 0 the poloidal flux at the magnetic axis and 1 the poloidal magnetic flux at the plasma boundary. P and FF are set to be zero at the boundary when ı = 1 at ohmic or L-mode discharge, and set to be finite value at boundary when ı = 0 at H-mode discharge. For the number of coefficients nP and nF , it is essential that they should be appropriately chosen to correctly reflect the availability of the measured data [1,3]. The magnetic diagnostic data is consisted of magnetic flux from flux loops and magnetic field from magnetic probes outside the plasma and plasma current from a Rogowski loop. Then, the calculated value of magnetic diagnostic measurements are obtained from the Picard iterations: nC
m+1 (ri ) = GCi (ri , ren ) Ien
n=1
dR dZ GCi (ri , r )J (R , (m) , ˛n
(m+1)
+
(m+1)
, n
)
(3)
where Ci is the calculated value of ith measurement, GCi is the appropriate Green’s function corresponding to the ith measure(m+1) is the current in the external poloidal field coils at ment, Ien the m + 1 iteration, nc is the total number of the external poloidal field coils. During the iteration cycles, ˛n and n are readjusted continuously, according to the available measured data, by finding a linearized minimization 2
Here, is the poloidal magnetic flux per radian of the toroidal angle enclosed by a magnetic surface, F = 2RB /0 is the poloidal current function, B is the toroidal magnetic field, and ∗ = R2 ∇ × (∇ /R2 ). The equilibrium solution of poloidal flux and toroidal current density of plasma J are obtained on a rectangular grid which covers the entire area of the vacuum vessel. The plasma current is modeled as being distributed among the elements of grid
nP −1 n
˛n Nn − ıN p
n=0 nF −1
(m+1)
(1)
nP −1
P ( ) =
Ci
P-EFIT takes the algorithm of EFIT [1–3] as its basic framework. Its principle is to compute the poloidal flux distribution in the R, Z plane, then obtain the toroidal current density distribution in this plane, which should satisfy a least-square best fit to the diagnostic data under the model given by the Grad–Shafranov equation.
∗ = −0 RJ , J = RP ( ) +
points, with linear coefficients ˛n and n . As shown in Ref. [2], the pressure gradient and current flux are modeled as:
=
nM Mi − Ci i=1
i
2
(4)
where Mi , Ci and i denote the measured value, the calculated value and the error associated with the ith measurement, respectively. nM is the total number of measurements. The solution is calculated using a new value of normalized flux N to compute the new set values of the coefficients (˛n , n ). Least square fitting is used to
Fig. 1. The flow chart of P-EFIT: green parts are equilibrium calculation which are processed on GPU, and blue parts are data interface which are processed on CPU. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model FUSION-8565; No. of Pages 6
ARTICLE IN PRESS Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
3
method is used and the partial differential equation transformed into a block tri-diagonal equation. By eigenvalue decomposition, the block tri-diagonal equation is transformed into independent triangular system that could be solved in parallel on GPU [11]. Eq. (3) shows the relation between current sources (plasma current and PF coils current) and magnetic diagnostics. Based on the distributed grid points and PF coils, the Green’s function between these current sources and diagnostics can be calculated beforehand. To get the response matrix R, rearrangement of Green’s functions and their integration over the plasma volume weighted by the uncertainties i is needed. This part is mainly consisted of two large matrix response matrix multiplication. By dividing the matrix into small parts, the cost of matrix elements multiplications, additions and data accessing are reduced. Least square fitting solves the Eq. (5), the plasma current representation coefficients (˛n , n ) (m+1) is solved. EFIT solves this equaand external coil currents Ien tion by singular-value decomposition. P-EFIT makes use of parallel matrix multiplication, the initial over-determined equation system T
T
Fig. 2. The read/write method used in P-EFIT algorithm: thread 0 is for reading data from and writing data to RFM, thread 1 is for controlling parallel computing on GPU.
is transformed into full rank system as R R˛ = R M and can be solved directly. There are standard linear algebra routines in CUDA library, but the size of matrix in equilibrium reconstruction algorithms is too small to achieve good performance by directly using them. When developing modules in P-EFIT, we need to design customized parallel algorithm, arrange GPU cores into blocks and threads, consider how to take fully use of hundreds of GPU cores and high speed memory such as shared memory, minimize the communication and synchronization between different cores. For this reason, each parallel algorithms in P-EFIT are carefully designed through combining the needs of numerical algorithms and the GPU capacity. The detailed basis parallel algorithms can be found in [9]. In this paper, some timing results will be showed, all these results are tested on a Linux workstation with one Intel(R) Xeon(R) CPU E31230 @ 3.20GHz and one NVIDIA Tesla K20c GPU card.
solve this linearized minimization problem. Eqs. (3) and (4) can be combined to relate the unknown parameter ˛ directly to the vector
3. P-EFIT implementation in PCS
of diagnostic measurements M through response matrix R as: 3.1. Real-time algorithm R˛ = M
(5)
Here, the response matrix R is the rearrangement of Green’s functions and their integration over the plasma volume of Eq. (3), the (m+1) unknown parameter ˛ is consisted of Ien , ˛n and n in Eq. (3), Mis the measurement. The iteration process continues until the latest two successive error, indicated by the largest change in the grid points, is small enough (such as ε = 10−3 ).
(m) − (m+1) − ≤ε a b
(6)
Then, the iteration process finishes. The values of the (˛n , n ) obtained in the latest iteration are the correct coefficients of the plasma current density. Then the equilibrium parameters, such as pressure P, current flux function F, poloidal flux , poloidal beta ˇp , and the parameters of plasma shape and position are obtained. 2.2. Basis of parallel equilibrium reconstruction algorithms P-EFIT is based on the EFIT framework but takes advantage of massively parallel GPU cores to significantly accelerate the computation. The main time-consuming computing parts are poloidal flux refreshing (* Inversion), response matrix calculation and least square fitting. Eq. (1) shows the poloidal flux refreshing (* Inversion), which calculate the flux on grid points after getting the plasma current distribution. As same in EFIT, finite difference
Although P-EFIT could obtain a well-converged and accurate enough equilibrium result after 8–9 iterations within about 2.5 ms which more than ten times faster than offline EFIT, it does still not satisfy the requirement of real-time control in tokamak operation. For this reason, a strategy similar to RT-EFIT is adopted [4], whose basic premise is to use the equilibrium result of last time-slice as the initial input for next computation, and perform new equilibrium reconstruction iteration using the most recent diagnostic data. For each time slice, only one iteration is conducted. With this strategy, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements in real-time reconstruction for plasma discharge control [9]. After equilibrium calculation, control errors need to be computed. The algorithm of calculating control errors is also similar to RT-EFIT, all the current sources multiple Green’s function matrix prepared before hand to get the flux on control segments as: nC
Seg (i) = GSegi (ri , ren )Ien + n=1
dR dZ GSegi (ri , r )J (R , )
(7)
where Seg (i) is the flux value on control segments, GSegi (ri , ren ) is the appropriate Green’s function corresponding to the control segments, Ien is the current in the external poloidal field coils, J (R , ) is the toroidal plasma current density. The flow chart of P-EFIT is showed in Fig. 1.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
4
P−EFIT && RT−EFIT at 4.05s
a
P−EFIT RT−EFIT 0.8 Seg 3
Z (m)
0.4
0
−0.4
Seg 4
Seg 6
Seg 1
Seg 8 Seg 9
−0.8
2 R (m)
b 0
(Wb)
−0.05
(m)
0
(Wb)
IDERX2
0.05
IDESEG06
0
(Wb)
(m)
0
IDESEG08
0
2
4 Time(s)
6
IDESEG01
0.005
IDESEG09
0
−0.005
−0.005
−0.05
0
−0.01
0.005 IDEZX2
0.005
−0.005
−0.05 0.05
IDESEG03
0
−0.01
−0.005
0
P−EFIT
0.005
−0.005
0.005
−0.05
(m)
0
−0.005
IDEZX1
0.05
IDESEG04
(Wb)
(m)
0.005
RT−EFIT
(Wb)
IDERX1
0.05
2.5
(Wb)
1.5
−0.01 2
4 Time(s)
6
2
4 Time(s)
6
Fig. 3. (a) Configuration comparison between P-EFIT and RT-EFIT result of simulation test shot at 4.05 s and description of 6 control segments (b) control errors comparison between P-EFIT and RT-EFIT results, IDERX1, IDEZX1, IDERX2, IDEZX2 are X-points control errors; IDESEG04, IDESEG03, IDESEG06, IDESEG01, IDESEG08, IDESEG09 are control errors on 6 control segments shown in (a).
3.2. Data interface between PCS and P-EFIT’s GPU server Since P-EFIT is a independent system to PCS, a reflective memory network (RFM) between them was built to share the real-time data [12–15]. The device code in P-EFIT, running on GPU, can only access the data in GPU’s memory. Transferring data between GPU and RFM directly takes about 50 s, which is too much compared with 300 s costed by equilibrium calculation per iteration. As shown in Fig. 2, the data interface system in P-EFIT launches two threads, which are locked on CPU 0 and CPU 1 separately. One is responsible for reading data from and writing data to RFM, the other is in charge of sharing data between P-EFIT host code in CPU and device code in GPU. Between these two threads, a segment of memory on CPU is defined as exchange buffer to share the data themselves. The thread for exchanging data with RFM reads the diagnostic data from RFM then transfer these newest diagnostic data to exchange buffer, reads the control errors in exchange buffer then writes them to RFM in the promissory address. The thread for controlling parallel computing writes control errors to the exchange buffer and reads newest diagnostic data from exchange buffer. Two threads are
running in parallel on CPUs, the data in RFM and exchange buffer can maintain synchronized. In each P-EFIT iteration, 75 singleprecision numbers consisted of one slice-time and 74 magnetic diagnostic data from PCS, and about 25 single-precision numbers, which depends on the number of control segments, consisted of control errors on control segments and x-points from GPU, totally about 400 bytes data transfer through exchange buffer. Mutual exclusion lock in exchange buffer is needed to insure the exclusiveness of writing and reading procedure. By this way, the time consumption by exchanging data between GPU and RFM can be partly hidden, and the increased time cost by data transfer between GPU and RFM per iteration is reduced to less than 20 s. 4. Application to shape control in EAST 4.1. Benchmark test In order to verify the correctness of system integration of PEFIT and EAST PCS, a experimental environment simulation test was carried out. For EAST PCS, there is a running mode called hardware
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
5
0.005 0.005 IDERX1
−0.03
0
−0.005
IDEZX1
0.005
0
0.003 IDESEG06
(Wb)
−0.03
(m)
0.03 0
0
−0.006
(m)
IDEZX2
0
−0.003
−0.03 2
0.003 IDESEG08
(Wb)
(Wb)
0.003 0.03 0
IDESEG01
0
−0.003
−0.005
IDERX2
−0.03
IDESEG03 0
−0.005 (Wb)
(m)
0.03
IDESEG04
(Wb)
0
(Wb)
(m)
0.03
4 6 Time(s)
8
0
IDESEG09
−0.003 2
4 6 Time(s)
8
2
4 6 Time(s)
8
Fig. 4. Control errors of shot 51044: IDERX1, IDEZX1, IDERX2, IDEZX2 are X-points control errors; IDESEG04, IDESEG03, IDESEG06, IDESEG01, IDESEG08, IDESEG09 are control errors on control segments which are showed in Fig. 3(a).
P−EFIT && EFIT && RT−EFIT at 5.13s
1
−3
0.6
x 10
RX (cm)
P−EFIT EFIT RT−EFIT
164 EFIT P−EFIT RT−EFIT
162
160 2
2
3
4
3
4
5
6
7
5
6
7
Z (m)
0.2 1
ZX (cm)
Z (m)
−0.2
−78
−79
0
−80
−0.6
−1 2
Time (s) −2
−1 1.5
2 R (m)
2.5
2.3
2.301 R (m)
Fig. 5. Configuration comparison between P-EFIT, RT-EFIT and off-line EFIT results.
test, in which all the hardware instruments are used and diagnostic data are read from history experimental shot. In this test, P-EFIT reads experimental diagnostic data and sends back control errors in each iteration. At the same time, RT-EFIT was also running in PCS for comparison. The testing shows that the calculated control errors by P-EFIT and RT-EFIT are consistent with each other as shown in Fig. 3. P-EFIT provides control error results per 0.35 ms, about 0.3 ms for equilibrium calculation, 0.03 ms for control error calculation and 0.02 ms for exchanging data. 4.2. Valid discharge Successful plasma shape control by ISOFLUX algorithm based on P-EFIT was achieved in the dedicated experiment during the EAST 2014 campaign. After 4 shots dishrag testing, the valid plasma shape control based on P-EFIT reconstructed results was realized
Fig. 6. X-point location comparison between P-EFIT, RT-EFIT and off-line EFIT results.
successfully. In Fig. 4, control errors of shot 51044, which is a 8.68s lower single null divertor discharge, are shown. The comparison of the reconstructed plasma shape between PEFIT, RT-EFIT and offline EFIT at time slice 5.13 is shown in Fig. 5 both P-EFIT and RT-EFIT match well with off-line EFIT. The maximum boundary difference is only 1.5 mm at the out mid-plane. Fig. 6 shows X-point location comparison between P-EFIT, RT-EFIT and off-line EFIT results.
5. Conclusion In this paper, we described parallel equilibrium reconstruction code P-EFIT running on GPU was integrated with EAST plasma control system. Compared with RT-EFIT used in EAST, P-EFIT has better spatial resolution and full algorithm of EFIT per iteration. With the data interface through RFM, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements in real-time reconstruction for plasma discharge control. Successful control using
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model FUSION-8565; No. of Pages 6
ARTICLE IN PRESS Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
6
ISOFLUX/P-EFIT was established in the dedicated experiment during the EAST 2014 campaign. As the results showed, P-EFIT’s results match RT-EFIT’s and offline EFIT’s results and satisfy the requirement of real-time plasma configuration control. For equilibrium reconstruction only with external magnetic diagnostics, only macroparameter such as plasma boundary, ˇp , li and Ip can be determined [2], the advantage of P-EFIT can not significantly appear. With more diagnostics such as internal plasma current profile diagnostic and kinetic diagnostic, compared with RT-EFIT, P-EFIT has large potential to provide more detailed real-time equilibrium calculation which could support more sophisticated plasma control in the future. This work is a stepping-stone towards versatile ISOFLUX/P-EFIT control, such as the snowflake diverted shape control in next campaign and more precise real-time equilibrium reconstruction with more diagnostics. Acknowledgments This work is supported by the National Magnetic Confinement Fusion Research Program of China under Grant No. 2014GB103000, the National Natural Science Foundation of China under Grant No. 11575245 and National Natural Science Foundation of China for Youth under Grant No. 11205191. References
[3] L.L. Lao, et al., Equilibrium analysis of current profiles in tokamaks, Nucl. Fusion 30 (6) (1990) 1035. [4] J.R. Ferron, M.L. Walker, L.L. Lao, et al., Real time equilibrium reconstruction for tokamak discharge control, Nucl. Fusion 38 (1998) 1055. [5] J.G. Kwak, et al., Key features in the operation of KSTAR IEEE, Trans. Plasma Sci. 40 (2012) 697–704. [6] D.A. Gates, et al., Plasma shape control on the National Spherical Torus Experiment (NSTX) using real-time equilibrium reconstruction, Nucl. Fusion 46 (2006) 17–23. [7] L. Pangione, et al., New magnetic real time shape control for MAST, Fusion Eng. Des. 88 (2013) 1087–1090. [8] Wang Huazhong, Luo Jiarong, Huang Qinchao, Real time equilibrium reconstruction algorithm in EAST tokamak, Plasma Sci. Technol. 6 (4) (2004) 2390. [9] X.N. Yue, B.J. Xiao, Z.P. Luo, Y. Guo, Fast equilibrium reconstruction for tokamak discharge control based on GPU, Plasma Phys. Controlled Fusion 55 (2013) 085016. [10] N. Rath, et al., High-speed multi-input, multi-output control using GPU processing in the HBT-EP tokamak, Fusion Eng. Des. 87 (12) (2012) 1895–1899. [11] W.D. Hillis, G.L. Steele, Data parallel algorithms, Commun. ACM 29 (1986) 1170–1183. [12] B.J. Xiao, D.A. Humphreys, M.L. Walker, A. Hyatt, J.A. Leuer, D. Mueller, B.G. Penaflor, et al., EAST plasma control system, Fusion Eng. Des. 83 (2) (2008) 181–187. [13] B.J. Xiao, Q.P. Yuan, D.A. Humphreys, M.L. Walker, A.W. Hyatt, J.A. Leuer, G.L. Jackson, et al., Recent plasma control progress on EAST, Fusion Eng. Des. 87 (12) (2012) 1887–1890. [14] Q.P. Yuan, et al., Plasma current, position and shape feedback control on EAST, Nucl. Fusion 53 (4) (2013) 043009. [15] Q.P. Yuan, B.J. Xiao, B.G. Penaflor, D.A. Piglowski, L.Z. Liu, R.D. Johnson, M.L. Walker, D.A. Humphreys, New achievements in the EAST plasma control system, Fusion Eng. Des. 85 (3) (2010) 474–477.
[1] L.L. Lao, et al., Reconstruction of current profile parameters and plasma shapes in tokamaks, Nucl. Fusion 25 (11) (1985) 1611. [2] L.L. Lao, et al., MHD equilibrium reconstruction in the DIII-D tokamak, Fusion Sci. Technol. 48 (2) (2005) 968–977.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Fusion Engineering and Design xxx (2016) xxx–xxx
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST Yao Huang a,∗ , B.J. Xiao a,b , Z.P. Luo a , Q.P. Yuan a , X.F. Pei a , X.N. Yue b a b
Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, China School of Nuclear Science & Technology, University of Science & Technology of China, China
h i g h l i g h t s • • • • •
We described parallel equilibrium reconstruction code P-EFIT running on GPU was integrated with EAST plasma control system. Compared with RT-EFIT used in EAST, P-EFIT has better spatial resolution and full algorithm of EFIT per iteration. With the data interface through RFM, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements for plasma control. Successful control using ISOFLUX/P-EFIT was established in the dedicated experiment during the EAST 2014 campaign. This work is a stepping-stone towards versatile ISOFLUX/P-EFIT control, such as real-time equilibrium reconstruction with more diagnostics.
a r t i c l e
i n f o
Article history: Received 19 June 2015 Received in revised form 23 January 2016 Accepted 15 February 2016 Available online xxx Keywords: Equilibrium reconstruction Plasma control EAST GPU parallel computation
a b s t r a c t Implementation of P-EFIT code for plasma control in EAST is described. P-EFIT is based on the EFIT framework, but built with the CUDATM architecture to take advantage of massively parallel Graphical Processing Unit (GPU) cores to significantly accelerate the computation. 65 × 65 grid size P-EFIT can complete one reconstruction iteration in 300 s, with one iteration strategy, it can satisfy the needs of real-time plasma shape control. Data interface between P-EFIT and PCS is realized and developed by transferring data through RFM. First application of P-EFIT to discharge control in EAST is described. © 2016 Elsevier B.V. All rights reserved.
1. Introduction Plasma equilibrium reconstruction is an important tool for tokamak data analysis. Based on experimental measurements, the plasma shape and current profile can be evaluated. The real-time plasma equilibrium reconstruction is routinely used for the plasma control. The essential requirement of optimum performance of plasma feedback control is to obtain the accurate and fast equilibrium reconstruction results. Various equilibrium reconstruction numerical codes had been developed and implemented on different tokamak. One of the most widely used codes is EFIT [1–3]. Unforturnatly, EFIT is only fit for offline calculation and between shot analysis. Its real-time version, called RT-EFIT [4] code is currently used in DIIID, EAST, KSTAR, NSTX and MAST [5–8] as a
∗ Corresponding author. E-mail address: [email protected] (Y. Huang).
routine real-time plasma equilibrium reconstruction for plasma feedback control. The RT-EFIT code makes many simplifications based on offline EFIT and the spatial resolution is relatively low. For example, in EAST the RT-EFIT scales the 2.4 m × 1.4 m rectangle plasma calculation area by 33 × 33 grids, where the offline EFIT typically scales this area by 129 × 129 grids. In order to satisfy the accurate and fast reconstruction of plasma equilibrium, a new numerical code was developed based on Graphic Processing Units (GPU), named as P-EFIT [9]. As a new high performance computing method, GPU has been applied in plasma control in recent years, a plasma control system which utilizes GPUs is designed to magnetically control the 3D perturbed equilibrium state of the plasma in HBT-EP tokamak [10]. P-EFIT took the advantage of massively parallel processing cores in GPU to accelerate the whole reconstruction process in offline EFIT. Comparison with RT-EFIT using multiple CPUs only, P-EFIT utilized the CPU and GPU with heterogeneous parallel computing provided by NVIDIA CUDA software environment.
http://dx.doi.org/10.1016/j.fusengdes.2016.02.048 0920-3796/© 2016 Elsevier B.V. All rights reserved.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
2
Results from EAST single static equilibrium benchmark tests indicate that P-EFIT could accurately reproduce the EFIT reconstruction algorithms at a fraction cost time of the computation. With a 65 × 65 grid size, P-EFIT can complete one reconstruction iteration within 300 s [9], which shows good capability for real-time plasma feedback control. By cutting the iteration number into one or two, P-EFIT can be used directly to control the plasma shape based on ISOFLUX/P-EFIT algorithm. Successful control was carried out in the dedicated experiment during the EAST 2014 campaign. The basic plasma equilibrium reconstruction algorithm and its implementation for parallel running on GPU are described in Section 2. System integration for P-EFIT and Plasma Control System (PCS) is presented in Section 3. Experimental application of P-EFIT in EAST plasma shape control is discussed in Section 4.
2. Introduction on parallel equilibrium reconstruction algorithms 2.1. Basic plasma equilibrium reconstruction algorithm in EFIT
0 FF ( ) 42 R
FF ( ) =
˛n
n=0 nF −1
n Nn
n − ıN F
n=0
(2)
n
n=0
where the normalized flux N = ( − 0 )/(1 − 0 ) varies between 0 and 1 with 0 the poloidal flux at the magnetic axis and 1 the poloidal magnetic flux at the plasma boundary. P and FF are set to be zero at the boundary when ı = 1 at ohmic or L-mode discharge, and set to be finite value at boundary when ı = 0 at H-mode discharge. For the number of coefficients nP and nF , it is essential that they should be appropriately chosen to correctly reflect the availability of the measured data [1,3]. The magnetic diagnostic data is consisted of magnetic flux from flux loops and magnetic field from magnetic probes outside the plasma and plasma current from a Rogowski loop. Then, the calculated value of magnetic diagnostic measurements are obtained from the Picard iterations: nC
m+1 (ri ) = GCi (ri , ren ) Ien
n=1
dR dZ GCi (ri , r )J (R , (m) , ˛n
(m+1)
+
(m+1)
, n
)
(3)
where Ci is the calculated value of ith measurement, GCi is the appropriate Green’s function corresponding to the ith measure(m+1) is the current in the external poloidal field coils at ment, Ien the m + 1 iteration, nc is the total number of the external poloidal field coils. During the iteration cycles, ˛n and n are readjusted continuously, according to the available measured data, by finding a linearized minimization 2
Here, is the poloidal magnetic flux per radian of the toroidal angle enclosed by a magnetic surface, F = 2RB /0 is the poloidal current function, B is the toroidal magnetic field, and ∗ = R2 ∇ × (∇ /R2 ). The equilibrium solution of poloidal flux and toroidal current density of plasma J are obtained on a rectangular grid which covers the entire area of the vacuum vessel. The plasma current is modeled as being distributed among the elements of grid
nP −1 n
˛n Nn − ıN p
n=0 nF −1
(m+1)
(1)
nP −1
P ( ) =
Ci
P-EFIT takes the algorithm of EFIT [1–3] as its basic framework. Its principle is to compute the poloidal flux distribution in the R, Z plane, then obtain the toroidal current density distribution in this plane, which should satisfy a least-square best fit to the diagnostic data under the model given by the Grad–Shafranov equation.
∗ = −0 RJ , J = RP ( ) +
points, with linear coefficients ˛n and n . As shown in Ref. [2], the pressure gradient and current flux are modeled as:
=
nM Mi − Ci i=1
i
2
(4)
where Mi , Ci and i denote the measured value, the calculated value and the error associated with the ith measurement, respectively. nM is the total number of measurements. The solution is calculated using a new value of normalized flux N to compute the new set values of the coefficients (˛n , n ). Least square fitting is used to
Fig. 1. The flow chart of P-EFIT: green parts are equilibrium calculation which are processed on GPU, and blue parts are data interface which are processed on CPU. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model FUSION-8565; No. of Pages 6
ARTICLE IN PRESS Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
3
method is used and the partial differential equation transformed into a block tri-diagonal equation. By eigenvalue decomposition, the block tri-diagonal equation is transformed into independent triangular system that could be solved in parallel on GPU [11]. Eq. (3) shows the relation between current sources (plasma current and PF coils current) and magnetic diagnostics. Based on the distributed grid points and PF coils, the Green’s function between these current sources and diagnostics can be calculated beforehand. To get the response matrix R, rearrangement of Green’s functions and their integration over the plasma volume weighted by the uncertainties i is needed. This part is mainly consisted of two large matrix response matrix multiplication. By dividing the matrix into small parts, the cost of matrix elements multiplications, additions and data accessing are reduced. Least square fitting solves the Eq. (5), the plasma current representation coefficients (˛n , n ) (m+1) is solved. EFIT solves this equaand external coil currents Ien tion by singular-value decomposition. P-EFIT makes use of parallel matrix multiplication, the initial over-determined equation system T
T
Fig. 2. The read/write method used in P-EFIT algorithm: thread 0 is for reading data from and writing data to RFM, thread 1 is for controlling parallel computing on GPU.
is transformed into full rank system as R R˛ = R M and can be solved directly. There are standard linear algebra routines in CUDA library, but the size of matrix in equilibrium reconstruction algorithms is too small to achieve good performance by directly using them. When developing modules in P-EFIT, we need to design customized parallel algorithm, arrange GPU cores into blocks and threads, consider how to take fully use of hundreds of GPU cores and high speed memory such as shared memory, minimize the communication and synchronization between different cores. For this reason, each parallel algorithms in P-EFIT are carefully designed through combining the needs of numerical algorithms and the GPU capacity. The detailed basis parallel algorithms can be found in [9]. In this paper, some timing results will be showed, all these results are tested on a Linux workstation with one Intel(R) Xeon(R) CPU E31230 @ 3.20GHz and one NVIDIA Tesla K20c GPU card.
solve this linearized minimization problem. Eqs. (3) and (4) can be combined to relate the unknown parameter ˛ directly to the vector
3. P-EFIT implementation in PCS
of diagnostic measurements M through response matrix R as: 3.1. Real-time algorithm R˛ = M
(5)
Here, the response matrix R is the rearrangement of Green’s functions and their integration over the plasma volume of Eq. (3), the (m+1) unknown parameter ˛ is consisted of Ien , ˛n and n in Eq. (3), Mis the measurement. The iteration process continues until the latest two successive error, indicated by the largest change in the grid points, is small enough (such as ε = 10−3 ).
(m) − (m+1) − ≤ε a b
(6)
Then, the iteration process finishes. The values of the (˛n , n ) obtained in the latest iteration are the correct coefficients of the plasma current density. Then the equilibrium parameters, such as pressure P, current flux function F, poloidal flux , poloidal beta ˇp , and the parameters of plasma shape and position are obtained. 2.2. Basis of parallel equilibrium reconstruction algorithms P-EFIT is based on the EFIT framework but takes advantage of massively parallel GPU cores to significantly accelerate the computation. The main time-consuming computing parts are poloidal flux refreshing (* Inversion), response matrix calculation and least square fitting. Eq. (1) shows the poloidal flux refreshing (* Inversion), which calculate the flux on grid points after getting the plasma current distribution. As same in EFIT, finite difference
Although P-EFIT could obtain a well-converged and accurate enough equilibrium result after 8–9 iterations within about 2.5 ms which more than ten times faster than offline EFIT, it does still not satisfy the requirement of real-time control in tokamak operation. For this reason, a strategy similar to RT-EFIT is adopted [4], whose basic premise is to use the equilibrium result of last time-slice as the initial input for next computation, and perform new equilibrium reconstruction iteration using the most recent diagnostic data. For each time slice, only one iteration is conducted. With this strategy, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements in real-time reconstruction for plasma discharge control [9]. After equilibrium calculation, control errors need to be computed. The algorithm of calculating control errors is also similar to RT-EFIT, all the current sources multiple Green’s function matrix prepared before hand to get the flux on control segments as: nC
Seg (i) = GSegi (ri , ren )Ien + n=1
dR dZ GSegi (ri , r )J (R , )
(7)
where Seg (i) is the flux value on control segments, GSegi (ri , ren ) is the appropriate Green’s function corresponding to the control segments, Ien is the current in the external poloidal field coils, J (R , ) is the toroidal plasma current density. The flow chart of P-EFIT is showed in Fig. 1.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
4
P−EFIT && RT−EFIT at 4.05s
a
P−EFIT RT−EFIT 0.8 Seg 3
Z (m)
0.4
0
−0.4
Seg 4
Seg 6
Seg 1
Seg 8 Seg 9
−0.8
2 R (m)
b 0
(Wb)
−0.05
(m)
0
(Wb)
IDERX2
0.05
IDESEG06
0
(Wb)
(m)
0
IDESEG08
0
2
4 Time(s)
6
IDESEG01
0.005
IDESEG09
0
−0.005
−0.005
−0.05
0
−0.01
0.005 IDEZX2
0.005
−0.005
−0.05 0.05
IDESEG03
0
−0.01
−0.005
0
P−EFIT
0.005
−0.005
0.005
−0.05
(m)
0
−0.005
IDEZX1
0.05
IDESEG04
(Wb)
(m)
0.005
RT−EFIT
(Wb)
IDERX1
0.05
2.5
(Wb)
1.5
−0.01 2
4 Time(s)
6
2
4 Time(s)
6
Fig. 3. (a) Configuration comparison between P-EFIT and RT-EFIT result of simulation test shot at 4.05 s and description of 6 control segments (b) control errors comparison between P-EFIT and RT-EFIT results, IDERX1, IDEZX1, IDERX2, IDEZX2 are X-points control errors; IDESEG04, IDESEG03, IDESEG06, IDESEG01, IDESEG08, IDESEG09 are control errors on 6 control segments shown in (a).
3.2. Data interface between PCS and P-EFIT’s GPU server Since P-EFIT is a independent system to PCS, a reflective memory network (RFM) between them was built to share the real-time data [12–15]. The device code in P-EFIT, running on GPU, can only access the data in GPU’s memory. Transferring data between GPU and RFM directly takes about 50 s, which is too much compared with 300 s costed by equilibrium calculation per iteration. As shown in Fig. 2, the data interface system in P-EFIT launches two threads, which are locked on CPU 0 and CPU 1 separately. One is responsible for reading data from and writing data to RFM, the other is in charge of sharing data between P-EFIT host code in CPU and device code in GPU. Between these two threads, a segment of memory on CPU is defined as exchange buffer to share the data themselves. The thread for exchanging data with RFM reads the diagnostic data from RFM then transfer these newest diagnostic data to exchange buffer, reads the control errors in exchange buffer then writes them to RFM in the promissory address. The thread for controlling parallel computing writes control errors to the exchange buffer and reads newest diagnostic data from exchange buffer. Two threads are
running in parallel on CPUs, the data in RFM and exchange buffer can maintain synchronized. In each P-EFIT iteration, 75 singleprecision numbers consisted of one slice-time and 74 magnetic diagnostic data from PCS, and about 25 single-precision numbers, which depends on the number of control segments, consisted of control errors on control segments and x-points from GPU, totally about 400 bytes data transfer through exchange buffer. Mutual exclusion lock in exchange buffer is needed to insure the exclusiveness of writing and reading procedure. By this way, the time consumption by exchanging data between GPU and RFM can be partly hidden, and the increased time cost by data transfer between GPU and RFM per iteration is reduced to less than 20 s. 4. Application to shape control in EAST 4.1. Benchmark test In order to verify the correctness of system integration of PEFIT and EAST PCS, a experimental environment simulation test was carried out. For EAST PCS, there is a running mode called hardware
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model
ARTICLE IN PRESS
FUSION-8565; No. of Pages 6
Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
5
0.005 0.005 IDERX1
−0.03
0
−0.005
IDEZX1
0.005
0
0.003 IDESEG06
(Wb)
−0.03
(m)
0.03 0
0
−0.006
(m)
IDEZX2
0
−0.003
−0.03 2
0.003 IDESEG08
(Wb)
(Wb)
0.003 0.03 0
IDESEG01
0
−0.003
−0.005
IDERX2
−0.03
IDESEG03 0
−0.005 (Wb)
(m)
0.03
IDESEG04
(Wb)
0
(Wb)
(m)
0.03
4 6 Time(s)
8
0
IDESEG09
−0.003 2
4 6 Time(s)
8
2
4 6 Time(s)
8
Fig. 4. Control errors of shot 51044: IDERX1, IDEZX1, IDERX2, IDEZX2 are X-points control errors; IDESEG04, IDESEG03, IDESEG06, IDESEG01, IDESEG08, IDESEG09 are control errors on control segments which are showed in Fig. 3(a).
P−EFIT && EFIT && RT−EFIT at 5.13s
1
−3
0.6
x 10
RX (cm)
P−EFIT EFIT RT−EFIT
164 EFIT P−EFIT RT−EFIT
162
160 2
2
3
4
3
4
5
6
7
5
6
7
Z (m)
0.2 1
ZX (cm)
Z (m)
−0.2
−78
−79
0
−80
−0.6
−1 2
Time (s) −2
−1 1.5
2 R (m)
2.5
2.3
2.301 R (m)
Fig. 5. Configuration comparison between P-EFIT, RT-EFIT and off-line EFIT results.
test, in which all the hardware instruments are used and diagnostic data are read from history experimental shot. In this test, P-EFIT reads experimental diagnostic data and sends back control errors in each iteration. At the same time, RT-EFIT was also running in PCS for comparison. The testing shows that the calculated control errors by P-EFIT and RT-EFIT are consistent with each other as shown in Fig. 3. P-EFIT provides control error results per 0.35 ms, about 0.3 ms for equilibrium calculation, 0.03 ms for control error calculation and 0.02 ms for exchanging data. 4.2. Valid discharge Successful plasma shape control by ISOFLUX algorithm based on P-EFIT was achieved in the dedicated experiment during the EAST 2014 campaign. After 4 shots dishrag testing, the valid plasma shape control based on P-EFIT reconstructed results was realized
Fig. 6. X-point location comparison between P-EFIT, RT-EFIT and off-line EFIT results.
successfully. In Fig. 4, control errors of shot 51044, which is a 8.68s lower single null divertor discharge, are shown. The comparison of the reconstructed plasma shape between PEFIT, RT-EFIT and offline EFIT at time slice 5.13 is shown in Fig. 5 both P-EFIT and RT-EFIT match well with off-line EFIT. The maximum boundary difference is only 1.5 mm at the out mid-plane. Fig. 6 shows X-point location comparison between P-EFIT, RT-EFIT and off-line EFIT results.
5. Conclusion In this paper, we described parallel equilibrium reconstruction code P-EFIT running on GPU was integrated with EAST plasma control system. Compared with RT-EFIT used in EAST, P-EFIT has better spatial resolution and full algorithm of EFIT per iteration. With the data interface through RFM, 65 × 65 spatial grids P-EFIT can satisfy the accuracy and time feasibility requirements in real-time reconstruction for plasma discharge control. Successful control using
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048
G Model FUSION-8565; No. of Pages 6
ARTICLE IN PRESS Y. Huang et al. / Fusion Engineering and Design xxx (2016) xxx–xxx
6
ISOFLUX/P-EFIT was established in the dedicated experiment during the EAST 2014 campaign. As the results showed, P-EFIT’s results match RT-EFIT’s and offline EFIT’s results and satisfy the requirement of real-time plasma configuration control. For equilibrium reconstruction only with external magnetic diagnostics, only macroparameter such as plasma boundary, ˇp , li and Ip can be determined [2], the advantage of P-EFIT can not significantly appear. With more diagnostics such as internal plasma current profile diagnostic and kinetic diagnostic, compared with RT-EFIT, P-EFIT has large potential to provide more detailed real-time equilibrium calculation which could support more sophisticated plasma control in the future. This work is a stepping-stone towards versatile ISOFLUX/P-EFIT control, such as the snowflake diverted shape control in next campaign and more precise real-time equilibrium reconstruction with more diagnostics. Acknowledgments This work is supported by the National Magnetic Confinement Fusion Research Program of China under Grant No. 2014GB103000, the National Natural Science Foundation of China under Grant No. 11575245 and National Natural Science Foundation of China for Youth under Grant No. 11205191. References
[3] L.L. Lao, et al., Equilibrium analysis of current profiles in tokamaks, Nucl. Fusion 30 (6) (1990) 1035. [4] J.R. Ferron, M.L. Walker, L.L. Lao, et al., Real time equilibrium reconstruction for tokamak discharge control, Nucl. Fusion 38 (1998) 1055. [5] J.G. Kwak, et al., Key features in the operation of KSTAR IEEE, Trans. Plasma Sci. 40 (2012) 697–704. [6] D.A. Gates, et al., Plasma shape control on the National Spherical Torus Experiment (NSTX) using real-time equilibrium reconstruction, Nucl. Fusion 46 (2006) 17–23. [7] L. Pangione, et al., New magnetic real time shape control for MAST, Fusion Eng. Des. 88 (2013) 1087–1090. [8] Wang Huazhong, Luo Jiarong, Huang Qinchao, Real time equilibrium reconstruction algorithm in EAST tokamak, Plasma Sci. Technol. 6 (4) (2004) 2390. [9] X.N. Yue, B.J. Xiao, Z.P. Luo, Y. Guo, Fast equilibrium reconstruction for tokamak discharge control based on GPU, Plasma Phys. Controlled Fusion 55 (2013) 085016. [10] N. Rath, et al., High-speed multi-input, multi-output control using GPU processing in the HBT-EP tokamak, Fusion Eng. Des. 87 (12) (2012) 1895–1899. [11] W.D. Hillis, G.L. Steele, Data parallel algorithms, Commun. ACM 29 (1986) 1170–1183. [12] B.J. Xiao, D.A. Humphreys, M.L. Walker, A. Hyatt, J.A. Leuer, D. Mueller, B.G. Penaflor, et al., EAST plasma control system, Fusion Eng. Des. 83 (2) (2008) 181–187. [13] B.J. Xiao, Q.P. Yuan, D.A. Humphreys, M.L. Walker, A.W. Hyatt, J.A. Leuer, G.L. Jackson, et al., Recent plasma control progress on EAST, Fusion Eng. Des. 87 (12) (2012) 1887–1890. [14] Q.P. Yuan, et al., Plasma current, position and shape feedback control on EAST, Nucl. Fusion 53 (4) (2013) 043009. [15] Q.P. Yuan, B.J. Xiao, B.G. Penaflor, D.A. Piglowski, L.Z. Liu, R.D. Johnson, M.L. Walker, D.A. Humphreys, New achievements in the EAST plasma control system, Fusion Eng. Des. 85 (3) (2010) 474–477.
[1] L.L. Lao, et al., Reconstruction of current profile parameters and plasma shapes in tokamaks, Nucl. Fusion 25 (11) (1985) 1611. [2] L.L. Lao, et al., MHD equilibrium reconstruction in the DIII-D tokamak, Fusion Sci. Technol. 48 (2) (2005) 968–977.
Please cite this article in press as: Y. Huang, et al., Implementation of GPU parallel equilibrium reconstruction for plasma control in EAST, Fusion Eng. Des. (2016), http://dx.doi.org/10.1016/j.fusengdes.2016.02.048