- Email: [email protected]
Write Operations on Parallel File Systems
Available online at www.sciencedirect.com
ScienceDirect This Procedia space isComputer reserved for 108C the Procedia header, do not use it Science (2017) 2393–2397 This space is reserved for the Procedia header, do not use it This space is reserved for the Procedia header, do not use it
International Conference on Computational Science, ICCS 2017, 12-14 June 2017, Zurich, Switzerland
A A A
Statistical Analysis of the Performance Variability of Statistical Analysis of theon Performance Variability of Read/Write Operations Parallel File Systems Statistical Analysis of the Performance Variability of Read/Write Operations on Parallel File Systems Eduardo C. Inacio, Pedro A. Barbetta, and Mario A. Systems R. Dantas Read/Write Operations on Parallel File Eduardo C. Inacio, Pedro A. Barbetta, andopolis, Mario A. R. Dantas Federal University of Santa Catarina, Florian´ SC, Brazil Eduardo C. Inacio, Pedro A. Barbetta, and Mario A. R. Dantas [email protected], {pedro.barbetta, mario.dantas}@ufsc.br Federal University of Santa Catarina, Florian´ opolis, SC, Brazil [email protected], {pedro.barbetta, mario.dantas}@ufsc.br Federal University of Santa Catarina, Florian´ opolis, SC, Brazil [email protected], {pedro.barbetta, mario.dantas}@ufsc.br
Abstract This paper reports a statistical analysis about the performance variability of read and write Abstract operations parallel file systems. To properly for thevariability inherent system This paper on reports a statistical analysis about theaccount performance of read variability and write Abstract and to obtain statistically significant results, formal experimental design and methods operations parallel file systems. To properly for thevariability inherent system This paper on reports a statistical analysis about theaccount performance ofanalysis read variability and write were employed in this study. This research reveals that in the evaluated conditions six effects and to obtain formal experimental andsystem analysisvariability methods operations on statistically parallel file significant systems. results, To properly account for thedesign inherent dominate I/O statistically time, responding for 99.32% ofreveals the performance Further, some factors weretoemployed in this study. This research that in thevariability. evaluated conditions six effects and obtain significant results, formal experimental design and analysis methods traditionally explored in I/O optimization proposals presented no statistical evidence of signifdominate I/O time, responding for 99.32% the performance some were employed in this study. This researchofreveals that in thevariability. evaluated Further, conditions six factors effects icant effects in this study. Moreover, high-level effects were identified by the interpretation of traditionally in I/O optimization proposals presentedvariability. no statistical evidence of factors signifdominate I/Oexplored time, responding for 99.32% of the performance Further, some the set of statistically significant factors, providing a case for further research in the subject. icant effects in this study. Moreover, high-level effects were identified by theevidence interpretation of traditionally explored in I/O optimization proposals presented no statistical of signifthe seteffects of Authors. statistically significant factors, providing aexperimental case foridentified further research in the subject.of icant in this Moreover, high-level effects were bystatistical the interpretation Keywords: Parallel filestudy. systems, performance variability, design, analysis © 2017 The Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the International Conference on Computational Science the set of statistically significant factors, providing a case for further research in the subject. Keywords: Parallel file systems, performance variability, experimental design, statistical analysis Keywords: Parallel file systems, performance variability, experimental design, statistical analysis
1 Introduction 1 Introduction High Performance Computing (HPC) systems have a very complex file input/output (I/O) path 1 Introduction in terms of hardware and software [1]. There are compute nodes, in which parallel applications
High Performance Computing (HPC) systems have a very complex file input/output (I/O) path are executed usingComputing high-level I/O libraries and middlewares, thereinfile may be parallel I/O forwarding and in terms of hardware and software [1].systems There are compute nodes, which applications High Performance (HPC) have a very complex input/output (I/O) path data staging nodes, and a persistent storage subsystem. This storage subsystem is commonly are executed using high-level I/O libraries and middlewares, there may be I/O forwarding and in terms of hardware and software [1]. There are compute nodes, in which parallel applications composed of two layers: a parallel high throughput storage, usually implemented through a data staging nodes, and a persistent storage subsystem. This storage subsystem is commonly are executed using high-level I/O libraries and middlewares, there may be I/O forwarding and parallel file system (PFS), such as OrangeFS/PVFS2, Lustre and GPFS, and a slower longcomposed of two layers: a parallel high throughput storage, usually implemented through data staging nodes, and a persistent storage subsystem. This storage subsystem is commonlya term archival system. Customarily, file datasets in the archival system are and moved tothrough the longPFS parallel fileofsystem (PFS), as OrangeFS/PVFS2, Lustre usually and GPFS, a slower composed two layers: a such parallel high throughput storage, implemented a for simulation and visualization consumption. Hence, this research work concentrates in the term archival system. Customarily, file datasets in the archival system are and moved to the longPFS parallel file system (PFS), such as OrangeFS/PVFS2, Lustre and GPFS, a slower I/O performance theCustomarily, PFS. for simulation andof visualization consumption. research work the term archival system. file datasetsHence, in the this archival system areconcentrates moved to theinPFS Previous research works [2, 3, 7] investigated this problem from many perspectives, including I/Osimulation performance the PFS. for andof visualization consumption. Hence, this research work concentrates in the identifying significant performance factors, understanding thefrom waymany they perspectives, interact and affect the research works [2, 3, 7] investigated this problem including I/OPrevious performance of the PFS. I/O performance, and root causes for I/O performance variability. Although their contributions identifying performance factors, understanding thefrom waymany they perspectives, interact and affect the Previoussignificant research works [2, 3, 7] investigated this problem including are toand tackle the hugerfor number of potentialvariability. factors is Although athey challenging I/O undeniable, performance, root causes I/O performance theirtask. contributions identifying significant performance factors, understanding the way interact and affect the paper takes a step toward addressing issue variability. byfactors presenting a statistical analysis of the are This undeniable, toand tackle hugerfor number ofthis potential is Although a challenging I/O performance, rootthe causes I/O performance theirtask. contributions performance variability of read and write operations on a PFS. Formal experimental design paper takes a stepthe toward issue byfactors presenting a statisticaltask. analysis ofand the are This undeniable, to tackle hugeraddressing number ofthis potential is a challenging analysis methods were employed in this study to provide reproducible and statistically sound performance of read andaddressing write operations onby a PFS. Formal experimental design This papervariability takes a step toward this issue presenting a statistical analysis ofand the analysis methods were employed inwrite this study to provide reproducible and statistically sound performance variability of read and operations on a PFS. Formal experimental design and 1 analysis methods were employed in this study to provide reproducible and statistically sound 1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the International Conference on Computational Science 10.1016/j.procs.2017.05.026
1 1
2394
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
results about effects of nine factors and their interaction in the performance of I/O operations. Experiments were conducted using two computing clusters with dozens of nodes, considering six different workloads. The remainder of this paper is organized as follows. In the Section 2, a discussion about some related works is provided. The experimental method, environments, and tools, used in this study are detailed in Section 3 Results of the analysis of variance (ANOVA) are presented in Section 4. Section 5 concludes this paper with directions for future works.
2
Related Work
As one of the major bottlenecks in modern large-scale computer systems, the I/O performance has been the focus of many research works. Some researches focus on characterizing the workload of data-intensive scientific applications [5, 4] and the overall performance of parallel storage systems [8, 2]. Performance variability is an inherent property of HPC environments [6], and has been also investigated previously from the I/O perspective [3, 7]. The study reported in this paper has some similarities with previous research works and, nevertheless, many differences. More than characterizing a specific I/O workload or storage system, this paper looks for statistical evidences of which performance factors have significant impact in the I/O performance of a PFS. Performance metrics and load indexes were collected using simple monitoring tools, such as sysstat and dstat. As many previous research works, this study relies on an experimental research.
3
Environment Scenarios and Methods
The performance analysis was based on an experimental research carried out on two environments using the three clusters from the Grid’5000 testbed. The StRemi environment consists of 8 storage and 16 compute nodes, each with two CPUs AMD Opteron 6164 HE 1.7 GHz (12 cores), 48 GB RAM, and a 250 GB SATA disk; interconnected through a Gigabit Ethernet (GbE) network The Grimoire/Grisou environment consists of 8 storage (Grimoire) and 16 compute nodes (Grisou), each having two CPUs Intel Xeon E5-2630 v3 2.4 GHz (8 cores), 126 GB RAM, and four 10 GbE network interfaces. During experiments, all nodes were reserved to avoid interferences from concurrent applications. The same operating system image was deployed in all nodes, consisting of a CentOS 6.7 (kernel 2.6.32), with the OrangeFS 2.8.8, MPICH 3.2 and IOR 3.0.1. Despite other benchmarks, such as BT-IO and MADBench, generate a workload closer to real applications, IOR provides a more precise control of the investigated factors, a need for this type of study. In this study, six performance factors were considered, with two levels each: API (POSIX, MPI-IO), I/O strategy (file per process, shared file), request size (64, 256 MiB), access pattern (sequential, random), stripe size (64 KiB, 1 MiB), and stripe count (2, 8). As preliminary results demonstrated that a full factorial experiment would not be practical in terms of time, a fractional factorial experimental design of resolution 4 (26−2 IV ) was adopted. Each experiment of this design was replicated three times. The execution order was completely random to assure variables are independent and individually distributed. Six different workloads were evaluated for each experimental set. These workloads are described in terms of the number of tasks (#T), number of segments (#Seg) and block sizes (BlkSz) in Table 1. The values defined for workloads components were based on previous studies 2
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
Table 1: Workloads composition.
W1 W2
#T
#Seg
BlkSz
16 16
1 16
4 GiB 256 MiB
#T
#Seg
BlkSz
64 64
1 4
1 GiB 256 MiB
W3 W4
W5 W6
#T
#Seg
BlkSz
128 128
1 2
512 MiB 256 MiB
and workloads of application benchmarks [5, 1, 2]. Given the obvious effect of the amount of data read/written in the I/O time, all workloads move 64 GiB of file data. The response variable evaluated in this study is the time taken by the PFS to service all requests from all tasks of the parallel application (i.e., IOR), henceforth called the I/O time. This performance metric is computed from the time the first MPI task starts an read/write operation until the time the last request serviced is acknowledged by the last pending MPI task. At each experiment run, the I/O time is computed for a write and then a read operation. As a result, a total of 1, 152 experimental observations were collected in this study1 .
4
Experimental Results
This study employs a statistical analysis of effects, using the ANOVA F -test, to evaluate the significance of each factor and their two-way interactions in the I/O performance of a PFS. The p-value provided is used to accept or reject the null hypothesis H0 , which stands for no difference in the response variable given changes in the factor. Traditionally, a level of significance of 5% (α = 0.05) is adopted in hypothesis testing. Although this approach would provide statistically valid conclusions, we opted for using a different approach, considering intervals for the level of significance. We propose a classification system based on six p-value intervals: highly significant ([−∞, 0.00000[), very significant ([0.00000, 0.00101[), significant ([0.00101, 0.05000[), borderline significant ([0.05000, 0.06000[), barely nonsignificant ([0.06000, 0.10001[), and nonsignificant ([0.10001, ∞]). The benefits of this approach includes (i) a more semantic way to describe the statistical significance of an effect, (ii) a more systematic way to compare the significance of effects, and (iii) an easier way to identify and report effects in the borderline of acceptance/rejection region of the H0 .
4.1
Analysis of Variance
Applying the ANOVA F -test for a full effects model resulted in the following observations. Six out of nine main effects presented highly significance in the ANOVA. The request size, access pattern, and stripe size are the only main effects whose test indicated nonsignificance. This is an interesting result, given these factors are traditionally explored in I/O performance optimization approaches. Another interesting observation is the significant impact of not only main but also interaction effects involving the environment, operation, and workload factors. Not by coincidence, exceptions are interactions between these three factors and request size, access pattern and stripe size. Additionally, the interaction between the workload and operation showed nonsignificant effects. This can be interpreted as for either read or write operations, the impact in the I/O time independs from the workload, and vice versa.
1 Results dataset and scripts used in the analysis are available at https://drive.google.com/open?id= 0B-l8_VW_R9_IcnRNYWZrcTBJTkk
3
2395
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
1202.01
1200
87.14
1000
●
●
●
●
99.32
98.28
96.7
92.26
●
100 80
●
71.02
800
60
600
40
400
272.97
200
86.64
0
t
en
nm
o vir En
S
e trip
t
gy
un
co
I/O I/O
ate str
gy
ate str
xs
e trip
20
75.03
t
un
co
26.87
on
i rat
e Op
nx
tio
era
Op
17.5
t
en
nm
o vir En
0
Cumulative percentage
On interpreting interaction effects, it is necessary to consider confounded effects, such as effects of API x access pattern and I/O strategy x stripe count interactions. In order to evaluate which of the confounded interactions actually has a significant effect, more experiments are necessary. In the case of the API x access pattern and I/O strategy x stripe count, additional experiments showed that the later interaction is the one with significant effect. Figure 1 presents a Pareto chart with the ANOVA mean square for the most impacting factors and the cumulative percentage of the variability explained by them. These results
Mean square
2396
Figure 1: Pareto chart with the ANOVA mean square and the cumulative percentage of the variability explained by the factors. demonstrate that 99.32% of the data variability is explained by four main and two interaction effects: experimental environment, stripe count, I/O strategy, operation; and the interactions I/O strategy x stripe count and operation x environment. Among these, the experimental environment is the most impacting factor, answering for 71.02% of the variability of the model. This is a reasonable result, given that the Grimoire/Grisou environment has 3× more memory per node and a network with 10× more bandwidth. The stripe count comes in second place in terms of impact, followed by the I/O strategy and their interaction. Together, these main and interaction effects answer for 25.68% of the model variability. This impact can be attributed to load balancing effects. Both stripe count and I/O strategy are parameters that influence the data distribution process of the PFS. The operation and its interaction with the environment factor, together, explain 2.62% of the data variability. Results indicated that write operations were slower than read operations. This behavior may be related to caching effects and to the way that the IOR benchmark works. In the IOR, read operations are tested using files created and written during directly previous write testing. As write operations are cached in the page cache of the data servers, reads are most probably being serviced from the page cache as well. Effects from other factors and interactions, although statistically significant, are responsible for less than 1% of the total performance variability observed in these experiments. An interpretation for this result is that looking for the system as a whole, varying these factors within the considered levels would cause minor impact in the overall performance. It is an interesting observation that could guide future research on approaches for I/O performance optimization.
5
Conclusions and Future Research
This paper reports a statistical analysis of the effect of nine factors and their pairwise interactions in the performance of read and write operations on PFSs. A formal experimental design 4
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
combined with mathematically proved analysis methods were employed to properly account for variability and to obtain statistically significant and reproducible results. Experimental results indicated six effects are responsible for explaining 99.32% of the variability in experiments. The experimental environment was the most impacting factor, answering for 71.02% of the performance variability. In second place came the stripe count, followed by the I/O strategy, operation, and the interactions I/O strategy x stripe count and operation x environment. However, some factors traditionally explored in I/O optimization research works, namely the request size, access pattern, and stripe size, presented no statistical significant effects. From this result, we can conclude that varying those factors within the range of values evaluated affects the performance in a way that can not be differed from experimental errors. It is worth to enforce that this conclusion is statistically valid only for the range of values considered. For other values, a different set of significant factors can arise.
Acknowledgements We would like to thank the Brazilian Federal Agency CAPES for supporting this research. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).
References [1] Babak Behzad, Huong V. T. Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth Aydt, Quincey Koziol, and Marc Snir. Taming parallel I/O complexity with auto-tuning. In SC ’13 Proc. Int. Conf. on High Performance Computing, Networking, Storage and Analysis. ACM, 2013. [2] Eduardo C. Inacio, Mario A. R. Dantas, and Douglas D. J. de Macedo. Towards a performance characterization of a parallel file system over virtualized environments. In ISCC ’15 Proc. 20th IEEE Symp. on Computers and Communications, pages 595–600. IEEE, 2015. [3] Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten Schwan, and Matthew Wolf. Managing Variability in the IO Performance of Petascale Storage Systems. In SC ’10 Proc. 2010 ACM/IEEE Int. Conf. for High Performance Computing, Networking, Storage and Analysis, pages 1–12. IEEE, 2010. [4] Huong Luu, Marianne Winslett, William Gropp, Robert Ross, Philip Carns, Kevin Harms, Mr Prabhat, Suren Byna, and Yushu Yao. A Multiplatform Study of I/O Behavior on Petascale Supercomputers. In HPDC ’15 Proc. 24th Int. Symp. on High-Performance Parallel and Distributed Computing, pages 33–44. ACM, 2015. [5] Hongzhang Shan, Katie Antypas, and John Shalf. Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark. In SC ’08 Proc. 2008 ACM/IEEE Conf. on Supercomputing. IEEE, 2008. [6] David Skinner and William Kramer. Understanding the causes of performance variability in HPC workloads. In IWSC ’05 Proc. IEEE Int. Workload Characterization Symp., pages 137–149. IEEE, 2005. [7] Orcun Yildiz, Matthieu Dorier, Shadi Ibrahim, Rob Ross, and Gabriel Antoniu. On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems. In IPDPS ’16 Proc. IEEE Int. Parallel and Distributed Processing Symp., pages 750–759. IEEE, 2016. [8] Weikuan Yu, Jeffrey S. Vetter, and H. Sarp Oral. Performance characterization and optimization of parallel I/O on the Cray XT. In IPDPS ’08 Proc. IEEE Int. Symp. on Parallel and Distributed Processing, pages 1–11. IEEE, 2008.
5
2397
ScienceDirect This Procedia space isComputer reserved for 108C the Procedia header, do not use it Science (2017) 2393–2397 This space is reserved for the Procedia header, do not use it This space is reserved for the Procedia header, do not use it
International Conference on Computational Science, ICCS 2017, 12-14 June 2017, Zurich, Switzerland
A A A
Statistical Analysis of the Performance Variability of Statistical Analysis of theon Performance Variability of Read/Write Operations Parallel File Systems Statistical Analysis of the Performance Variability of Read/Write Operations on Parallel File Systems Eduardo C. Inacio, Pedro A. Barbetta, and Mario A. Systems R. Dantas Read/Write Operations on Parallel File Eduardo C. Inacio, Pedro A. Barbetta, andopolis, Mario A. R. Dantas Federal University of Santa Catarina, Florian´ SC, Brazil Eduardo C. Inacio, Pedro A. Barbetta, and Mario A. R. Dantas [email protected], {pedro.barbetta, mario.dantas}@ufsc.br Federal University of Santa Catarina, Florian´ opolis, SC, Brazil [email protected], {pedro.barbetta, mario.dantas}@ufsc.br Federal University of Santa Catarina, Florian´ opolis, SC, Brazil [email protected], {pedro.barbetta, mario.dantas}@ufsc.br
Abstract This paper reports a statistical analysis about the performance variability of read and write Abstract operations parallel file systems. To properly for thevariability inherent system This paper on reports a statistical analysis about theaccount performance of read variability and write Abstract and to obtain statistically significant results, formal experimental design and methods operations parallel file systems. To properly for thevariability inherent system This paper on reports a statistical analysis about theaccount performance ofanalysis read variability and write were employed in this study. This research reveals that in the evaluated conditions six effects and to obtain formal experimental andsystem analysisvariability methods operations on statistically parallel file significant systems. results, To properly account for thedesign inherent dominate I/O statistically time, responding for 99.32% ofreveals the performance Further, some factors weretoemployed in this study. This research that in thevariability. evaluated conditions six effects and obtain significant results, formal experimental design and analysis methods traditionally explored in I/O optimization proposals presented no statistical evidence of signifdominate I/O time, responding for 99.32% the performance some were employed in this study. This researchofreveals that in thevariability. evaluated Further, conditions six factors effects icant effects in this study. Moreover, high-level effects were identified by the interpretation of traditionally in I/O optimization proposals presentedvariability. no statistical evidence of factors signifdominate I/Oexplored time, responding for 99.32% of the performance Further, some the set of statistically significant factors, providing a case for further research in the subject. icant effects in this study. Moreover, high-level effects were identified by theevidence interpretation of traditionally explored in I/O optimization proposals presented no statistical of signifthe seteffects of Authors. statistically significant factors, providing aexperimental case foridentified further research in the subject.of icant in this Moreover, high-level effects were bystatistical the interpretation Keywords: Parallel filestudy. systems, performance variability, design, analysis © 2017 The Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the International Conference on Computational Science the set of statistically significant factors, providing a case for further research in the subject. Keywords: Parallel file systems, performance variability, experimental design, statistical analysis Keywords: Parallel file systems, performance variability, experimental design, statistical analysis
1 Introduction 1 Introduction High Performance Computing (HPC) systems have a very complex file input/output (I/O) path 1 Introduction in terms of hardware and software [1]. There are compute nodes, in which parallel applications
High Performance Computing (HPC) systems have a very complex file input/output (I/O) path are executed usingComputing high-level I/O libraries and middlewares, thereinfile may be parallel I/O forwarding and in terms of hardware and software [1].systems There are compute nodes, which applications High Performance (HPC) have a very complex input/output (I/O) path data staging nodes, and a persistent storage subsystem. This storage subsystem is commonly are executed using high-level I/O libraries and middlewares, there may be I/O forwarding and in terms of hardware and software [1]. There are compute nodes, in which parallel applications composed of two layers: a parallel high throughput storage, usually implemented through a data staging nodes, and a persistent storage subsystem. This storage subsystem is commonly are executed using high-level I/O libraries and middlewares, there may be I/O forwarding and parallel file system (PFS), such as OrangeFS/PVFS2, Lustre and GPFS, and a slower longcomposed of two layers: a parallel high throughput storage, usually implemented through data staging nodes, and a persistent storage subsystem. This storage subsystem is commonlya term archival system. Customarily, file datasets in the archival system are and moved tothrough the longPFS parallel fileofsystem (PFS), as OrangeFS/PVFS2, Lustre usually and GPFS, a slower composed two layers: a such parallel high throughput storage, implemented a for simulation and visualization consumption. Hence, this research work concentrates in the term archival system. Customarily, file datasets in the archival system are and moved to the longPFS parallel file system (PFS), such as OrangeFS/PVFS2, Lustre and GPFS, a slower I/O performance theCustomarily, PFS. for simulation andof visualization consumption. research work the term archival system. file datasetsHence, in the this archival system areconcentrates moved to theinPFS Previous research works [2, 3, 7] investigated this problem from many perspectives, including I/Osimulation performance the PFS. for andof visualization consumption. Hence, this research work concentrates in the identifying significant performance factors, understanding thefrom waymany they perspectives, interact and affect the research works [2, 3, 7] investigated this problem including I/OPrevious performance of the PFS. I/O performance, and root causes for I/O performance variability. Although their contributions identifying performance factors, understanding thefrom waymany they perspectives, interact and affect the Previoussignificant research works [2, 3, 7] investigated this problem including are toand tackle the hugerfor number of potentialvariability. factors is Although athey challenging I/O undeniable, performance, root causes I/O performance theirtask. contributions identifying significant performance factors, understanding the way interact and affect the paper takes a step toward addressing issue variability. byfactors presenting a statistical analysis of the are This undeniable, toand tackle hugerfor number ofthis potential is Although a challenging I/O performance, rootthe causes I/O performance theirtask. contributions performance variability of read and write operations on a PFS. Formal experimental design paper takes a stepthe toward issue byfactors presenting a statisticaltask. analysis ofand the are This undeniable, to tackle hugeraddressing number ofthis potential is a challenging analysis methods were employed in this study to provide reproducible and statistically sound performance of read andaddressing write operations onby a PFS. Formal experimental design This papervariability takes a step toward this issue presenting a statistical analysis ofand the analysis methods were employed inwrite this study to provide reproducible and statistically sound performance variability of read and operations on a PFS. Formal experimental design and 1 analysis methods were employed in this study to provide reproducible and statistically sound 1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the International Conference on Computational Science 10.1016/j.procs.2017.05.026
1 1
2394
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
results about effects of nine factors and their interaction in the performance of I/O operations. Experiments were conducted using two computing clusters with dozens of nodes, considering six different workloads. The remainder of this paper is organized as follows. In the Section 2, a discussion about some related works is provided. The experimental method, environments, and tools, used in this study are detailed in Section 3 Results of the analysis of variance (ANOVA) are presented in Section 4. Section 5 concludes this paper with directions for future works.
2
Related Work
As one of the major bottlenecks in modern large-scale computer systems, the I/O performance has been the focus of many research works. Some researches focus on characterizing the workload of data-intensive scientific applications [5, 4] and the overall performance of parallel storage systems [8, 2]. Performance variability is an inherent property of HPC environments [6], and has been also investigated previously from the I/O perspective [3, 7]. The study reported in this paper has some similarities with previous research works and, nevertheless, many differences. More than characterizing a specific I/O workload or storage system, this paper looks for statistical evidences of which performance factors have significant impact in the I/O performance of a PFS. Performance metrics and load indexes were collected using simple monitoring tools, such as sysstat and dstat. As many previous research works, this study relies on an experimental research.
3
Environment Scenarios and Methods
The performance analysis was based on an experimental research carried out on two environments using the three clusters from the Grid’5000 testbed. The StRemi environment consists of 8 storage and 16 compute nodes, each with two CPUs AMD Opteron 6164 HE 1.7 GHz (12 cores), 48 GB RAM, and a 250 GB SATA disk; interconnected through a Gigabit Ethernet (GbE) network The Grimoire/Grisou environment consists of 8 storage (Grimoire) and 16 compute nodes (Grisou), each having two CPUs Intel Xeon E5-2630 v3 2.4 GHz (8 cores), 126 GB RAM, and four 10 GbE network interfaces. During experiments, all nodes were reserved to avoid interferences from concurrent applications. The same operating system image was deployed in all nodes, consisting of a CentOS 6.7 (kernel 2.6.32), with the OrangeFS 2.8.8, MPICH 3.2 and IOR 3.0.1. Despite other benchmarks, such as BT-IO and MADBench, generate a workload closer to real applications, IOR provides a more precise control of the investigated factors, a need for this type of study. In this study, six performance factors were considered, with two levels each: API (POSIX, MPI-IO), I/O strategy (file per process, shared file), request size (64, 256 MiB), access pattern (sequential, random), stripe size (64 KiB, 1 MiB), and stripe count (2, 8). As preliminary results demonstrated that a full factorial experiment would not be practical in terms of time, a fractional factorial experimental design of resolution 4 (26−2 IV ) was adopted. Each experiment of this design was replicated three times. The execution order was completely random to assure variables are independent and individually distributed. Six different workloads were evaluated for each experimental set. These workloads are described in terms of the number of tasks (#T), number of segments (#Seg) and block sizes (BlkSz) in Table 1. The values defined for workloads components were based on previous studies 2
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
Table 1: Workloads composition.
W1 W2
#T
#Seg
BlkSz
16 16
1 16
4 GiB 256 MiB
#T
#Seg
BlkSz
64 64
1 4
1 GiB 256 MiB
W3 W4
W5 W6
#T
#Seg
BlkSz
128 128
1 2
512 MiB 256 MiB
and workloads of application benchmarks [5, 1, 2]. Given the obvious effect of the amount of data read/written in the I/O time, all workloads move 64 GiB of file data. The response variable evaluated in this study is the time taken by the PFS to service all requests from all tasks of the parallel application (i.e., IOR), henceforth called the I/O time. This performance metric is computed from the time the first MPI task starts an read/write operation until the time the last request serviced is acknowledged by the last pending MPI task. At each experiment run, the I/O time is computed for a write and then a read operation. As a result, a total of 1, 152 experimental observations were collected in this study1 .
4
Experimental Results
This study employs a statistical analysis of effects, using the ANOVA F -test, to evaluate the significance of each factor and their two-way interactions in the I/O performance of a PFS. The p-value provided is used to accept or reject the null hypothesis H0 , which stands for no difference in the response variable given changes in the factor. Traditionally, a level of significance of 5% (α = 0.05) is adopted in hypothesis testing. Although this approach would provide statistically valid conclusions, we opted for using a different approach, considering intervals for the level of significance. We propose a classification system based on six p-value intervals: highly significant ([−∞, 0.00000[), very significant ([0.00000, 0.00101[), significant ([0.00101, 0.05000[), borderline significant ([0.05000, 0.06000[), barely nonsignificant ([0.06000, 0.10001[), and nonsignificant ([0.10001, ∞]). The benefits of this approach includes (i) a more semantic way to describe the statistical significance of an effect, (ii) a more systematic way to compare the significance of effects, and (iii) an easier way to identify and report effects in the borderline of acceptance/rejection region of the H0 .
4.1
Analysis of Variance
Applying the ANOVA F -test for a full effects model resulted in the following observations. Six out of nine main effects presented highly significance in the ANOVA. The request size, access pattern, and stripe size are the only main effects whose test indicated nonsignificance. This is an interesting result, given these factors are traditionally explored in I/O performance optimization approaches. Another interesting observation is the significant impact of not only main but also interaction effects involving the environment, operation, and workload factors. Not by coincidence, exceptions are interactions between these three factors and request size, access pattern and stripe size. Additionally, the interaction between the workload and operation showed nonsignificant effects. This can be interpreted as for either read or write operations, the impact in the I/O time independs from the workload, and vice versa.
1 Results dataset and scripts used in the analysis are available at https://drive.google.com/open?id= 0B-l8_VW_R9_IcnRNYWZrcTBJTkk
3
2395
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
1202.01
1200
87.14
1000
●
●
●
●
99.32
98.28
96.7
92.26
●
100 80
●
71.02
800
60
600
40
400
272.97
200
86.64
0
t
en
nm
o vir En
S
e trip
t
gy
un
co
I/O I/O
ate str
gy
ate str
xs
e trip
20
75.03
t
un
co
26.87
on
i rat
e Op
nx
tio
era
Op
17.5
t
en
nm
o vir En
0
Cumulative percentage
On interpreting interaction effects, it is necessary to consider confounded effects, such as effects of API x access pattern and I/O strategy x stripe count interactions. In order to evaluate which of the confounded interactions actually has a significant effect, more experiments are necessary. In the case of the API x access pattern and I/O strategy x stripe count, additional experiments showed that the later interaction is the one with significant effect. Figure 1 presents a Pareto chart with the ANOVA mean square for the most impacting factors and the cumulative percentage of the variability explained by them. These results
Mean square
2396
Figure 1: Pareto chart with the ANOVA mean square and the cumulative percentage of the variability explained by the factors. demonstrate that 99.32% of the data variability is explained by four main and two interaction effects: experimental environment, stripe count, I/O strategy, operation; and the interactions I/O strategy x stripe count and operation x environment. Among these, the experimental environment is the most impacting factor, answering for 71.02% of the variability of the model. This is a reasonable result, given that the Grimoire/Grisou environment has 3× more memory per node and a network with 10× more bandwidth. The stripe count comes in second place in terms of impact, followed by the I/O strategy and their interaction. Together, these main and interaction effects answer for 25.68% of the model variability. This impact can be attributed to load balancing effects. Both stripe count and I/O strategy are parameters that influence the data distribution process of the PFS. The operation and its interaction with the environment factor, together, explain 2.62% of the data variability. Results indicated that write operations were slower than read operations. This behavior may be related to caching effects and to the way that the IOR benchmark works. In the IOR, read operations are tested using files created and written during directly previous write testing. As write operations are cached in the page cache of the data servers, reads are most probably being serviced from the page cache as well. Effects from other factors and interactions, although statistically significant, are responsible for less than 1% of the total performance variability observed in these experiments. An interpretation for this result is that looking for the system as a whole, varying these factors within the considered levels would cause minor impact in the overall performance. It is an interesting observation that could guide future research on approaches for I/O performance optimization.
5
Conclusions and Future Research
This paper reports a statistical analysis of the effect of nine factors and their pairwise interactions in the performance of read and write operations on PFSs. A formal experimental design 4
Statistical Analysis of the Performance Variability of R/W Computer Operations on PFS Inacio, Barbetta and Dantas Eduardo C. Inacio et al. / Procedia Science 108C (2017) 2393–2397
combined with mathematically proved analysis methods were employed to properly account for variability and to obtain statistically significant and reproducible results. Experimental results indicated six effects are responsible for explaining 99.32% of the variability in experiments. The experimental environment was the most impacting factor, answering for 71.02% of the performance variability. In second place came the stripe count, followed by the I/O strategy, operation, and the interactions I/O strategy x stripe count and operation x environment. However, some factors traditionally explored in I/O optimization research works, namely the request size, access pattern, and stripe size, presented no statistical significant effects. From this result, we can conclude that varying those factors within the range of values evaluated affects the performance in a way that can not be differed from experimental errors. It is worth to enforce that this conclusion is statistically valid only for the range of values considered. For other values, a different set of significant factors can arise.
Acknowledgements We would like to thank the Brazilian Federal Agency CAPES for supporting this research. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).
References [1] Babak Behzad, Huong V. T. Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth Aydt, Quincey Koziol, and Marc Snir. Taming parallel I/O complexity with auto-tuning. In SC ’13 Proc. Int. Conf. on High Performance Computing, Networking, Storage and Analysis. ACM, 2013. [2] Eduardo C. Inacio, Mario A. R. Dantas, and Douglas D. J. de Macedo. Towards a performance characterization of a parallel file system over virtualized environments. In ISCC ’15 Proc. 20th IEEE Symp. on Computers and Communications, pages 595–600. IEEE, 2015. [3] Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten Schwan, and Matthew Wolf. Managing Variability in the IO Performance of Petascale Storage Systems. In SC ’10 Proc. 2010 ACM/IEEE Int. Conf. for High Performance Computing, Networking, Storage and Analysis, pages 1–12. IEEE, 2010. [4] Huong Luu, Marianne Winslett, William Gropp, Robert Ross, Philip Carns, Kevin Harms, Mr Prabhat, Suren Byna, and Yushu Yao. A Multiplatform Study of I/O Behavior on Petascale Supercomputers. In HPDC ’15 Proc. 24th Int. Symp. on High-Performance Parallel and Distributed Computing, pages 33–44. ACM, 2015. [5] Hongzhang Shan, Katie Antypas, and John Shalf. Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark. In SC ’08 Proc. 2008 ACM/IEEE Conf. on Supercomputing. IEEE, 2008. [6] David Skinner and William Kramer. Understanding the causes of performance variability in HPC workloads. In IWSC ’05 Proc. IEEE Int. Workload Characterization Symp., pages 137–149. IEEE, 2005. [7] Orcun Yildiz, Matthieu Dorier, Shadi Ibrahim, Rob Ross, and Gabriel Antoniu. On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems. In IPDPS ’16 Proc. IEEE Int. Parallel and Distributed Processing Symp., pages 750–759. IEEE, 2016. [8] Weikuan Yu, Jeffrey S. Vetter, and H. Sarp Oral. Performance characterization and optimization of parallel I/O on the Cray XT. In IPDPS ’08 Proc. IEEE Int. Symp. on Parallel and Distributed Processing, pages 1–11. IEEE, 2008.
5
2397