- Email: [email protected]
A coherent environment of software improvement tools for CMS
ARTICLE IN PRESS
Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142 www.elsevier.com/locate/nima
A coherent environment of software improvement tools for CMS$ G. Eulisse, S. Muzaffar, I. Osborne, L. Taylor, L.A. Tuura Department of Physics, Northeastern University, Boston, MA 02115, USA Available online 29 July 2004
Abstract CMS has developed approximately one million lines of C++ code and uses many more from HEP, Grid and public domain projects. We describe a suite of tools which help to manage this complexity by measuring software dependencies, quality metrics, and CPU and memory performance. This coherent environment integrates and extends existing open-source tools where possible and provides new in-house components where a suitable solution does not already exist. This is a freely available environment with graphical user interface which can be run on any software without the need to recompile or instrument it. We have developed ignominy which performs software dependency analysis of source code, binary products and external software. CPU profiling is provided based on oprofile, with added features such as profile snapshots, distributed profiling and aggregate profiles for farm systems including server-side tools for collecting profile data. Finally, we have developed a low-overhead performance and memory profiling tool, MemProf, which can perform (gprof-style) hierarchical performance profiling, in a way that works with multiple threads and dynamically loaded libraries (unlike gprof). It also gathers exact memory allocation profiles including which code allocates most, in what sizes of chunks, for how long, where the memory is getting freed and where it is getting leaked. We describe this tool suite and how it has been used to enhance the quality of CMS software. r 2004 Elsevier B.V. All rights reserved. PACS: 07.05. t; 07.05.Bx Keywords: Software dependency analysis; Metrics; CPU and memory performance; Ignominy; CMS
1. Motivation $
This work was supported by the US National Science Foundation. Corresponding author. E-mail address: [email protected] (L. Taylor).
CMS has a number of software improvement tools collected into a coherent environment available to all our software developers. These exploit existing generally available tools as much as
0168-9002/$ - see front matter r 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.nima.2004.07.035
ARTICLE IN PRESS G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
possible but most have some difficulties coping with systems as large and complex as the LHC experiments have. Therefore, we developed some tools ourselves to supplement existing ones. These are collected into a project called IGNOMINY. The main component is the ignominy dependency scanner [1]. Ignominy was originally created in the IGUANA project [2, and references therein]. The tools are all not specific to CMS and are freely available. Contact [email protected] and/or visit http://iguana.cern.ch for more details. CMS tries to improve the quality of our software in four ways: by coding patterns that add assertions and other checks, by providing static analysis tools to give developers feedback on the software structure, with dynamic analysis tools to give meaningful quantitative numbers about the quality and performance of the software, and with test harness tools to validate program behavior.
2. Code quality improvement tools CMS has contributed an internal debugging body to the LCG SEAL foundation classes [3]. This allows developers to make assertions which produce detailed feedback when a failure occurs at run-time. One set of classes allows programs to produce detailed crash logs, including a stack trace and a list of all shared libraries loaded in memory. Other classes provide means to protect against and recover from low-memory conditions and other exceptional situations without losing details of what failed. 2.1. Static analysis tools Static software quality assurance tools provide a way to control the inter-component as well as the external software dependencies within projects plus to check and validate the configuration of the system [4]. Ignominy is our tool for the former while toolchecker does the latter task. 2.2. Dynamic analysis tools A great host of tools have been developed over time to analyze various aspects of programs at
139
run-time, especially for profiling and memory checking. Perhaps the largest disadvantage of most such tools is that they do not work on software as large and complex as what CMS has. Most older tools do not function at all with shared libraries, which are ubiquitous by now. Even fewer tools work correctly with multi-threaded programs. Most memory-checking tools are slow, or blow up memory usage, or are so inaccurate that they are unusable with programs as large and complex as LHC reconstruction systems. Nobody will try out or use a system that requires compiletime instrumentation, as there are millions of lines of code to recompile to get any useful and comprehensive statistics. We need tools that work robustly and accurately with multiple threads and large program sizes, without requiring any preinstrumentation. Recently two Linux tools have risen to the challenge and provide not only adequate functionality to developers, but excel in it: valgrind [5] and oprofile [6]. CMS has developed another tool, igprof, to supplement these, as well as a GUI for oprofile to make it easier to use, as described below.
3. Ignominy Ignominy is suite of perl and shell scripts plus configurations for a number of projects. It helps you understand the structure of software systems. Its primary component is a dependency scanner that distills information into graphical views and numerical metrics. Ignominy is designed to adapt to almost any reasonable structure and has been used to analyze several large projects, both in CMS and out-side: nearly all CMS offline software, ATLAS offline software, Geant4, ROOT and some LCG software. Ignominy determines dependencies between software packages in a large software system, where a package is an arbitrarily defined chunk of software. It can currently analyze Fortran, C, and C++ dependencies. Dependencies between packages are due to a variety of reasons but in general the more incoming and outgoing dependencies a package
ARTICLE IN PRESS 140
G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
has, the more difficult it is to develop, test, release, and support. Reduction of inter-package dependencies is a crucial goal for a large software system. Dependency analysis helps developers to choose sensible sets of loosely coupled packages with only required dependencies on other packages. 3.1. How ignominy works The working of ignominy is very straightforward. For example, in CMS a simple front end shell script (scram-ignominy) queries SCRAM [4], the CMS built tool, to collect all the information about the external software packages used to build and generate search/built rules, which are then used, along with any extra user defined search/ build rules, by the ignominy analyzer. The frontend is the only CMS-specific part and can be replaced by other front-ends adapted to other build systems. The Ignominy analyser then generates a dependencies database and an extensive log file, which are then used to generate diagrams. Diagrams can be customized by editing the ignominy configuration files, for example one can rename dependencies diagram nodes, set different colors for different nodes and drop nodes from diagrams.
4. igprof: the IGnominious PROFiler igprof is a tool which completes other opensource profiling and debugging facilities like oprofile and Valgrind. It provides performance and memory usage statistics. It is developed by the authors and allows a developer to profile program performance and memory allocation in different ways including the hunting for memory leaks. It is completely non-intrusive and is fast enough to profile the whole CMS software and it’s many associated external packages. The performance profiler works similar to Mozilla’s JProf [7] and many other profilers: a SIGPROF signal is emitted every so often and a custom signal handler keeps the statistics of where the sample happens. In the long run this results in the distribution of processing power among different functions in the executable and its shared libraries according to their use. The memory profiler part works by setting up custom hooks for memory allocation functions, keeping all the information about allocations and deallocations in a compact tree structure which in the end can be dumped to a file. The profiler requires no instrumentation nor source code changes as it is injected into executables using the LD_PRELOAD mechanism. It can also be activated by the program itself by dynamically loading the library.
3.2. What ignominy can tell you The results from ignominy analysis of your software system can tell you the system’s overall package structure, which would help determine the dependency hierarchy, independently manageable and releasable weakly coupled units and test/ release order of packages. The analysis results will tell you what you need to have before installing a package and which packages are using what. This information is useful among other things in determining the set of products that can easily be released from a project. The analysis results will also mark any cyclic dependencies. Study of the dependency diagrams and metrics allows one also to find false dependencies, and the additional tabular information allows a developer to quickly locate the origin of unexpected dependencies.
5. Toolchecker The toolchecker tool is a perl script to check the external software configured for a SCRAM project. It can be used by the release manager to validate the consistency of the configuration and by end users to verify that the project configuration actually works correctly on their particular computer. It compiles test programs and checks if the external software configured for a project is usable, and in particular whether there are any unresolved symbols. Toolchecker also checks the individual packages in the project to make sure they are complete. It also checks the compatibility and suitability of external packages and makes suggestions if something could be improved. It will
ARTICLE IN PRESS G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
also indicate which tools are in fact needed or not needed for the project. The toolchecker script first queries SCRAM to get the list of all external tools configured for the project, and then it queries the configuration of each tool. Using this information, the toolchecker executes tests, such as compiling test programs, for each tool.
6. Other tools 6.1. Oprofile: kernel level profiling Oprofile is an open-source community effort to take advantage of hardware profiling counters found in modern processors to do a system wide performance analysis. Modern processors, on Intel from Pentium onward, have the ability to count certain types of hardware events into special registers. Different processor types and generations support different countable events, but overall there are numerous types that help gain insight into the system performance. It is possible to set up the processor so that when one of these counters reaches a certain trigger level, a nonmaskable interrupt is issued. Oprofile provides a kernel module that traps these interrupts and keeps the statistics of where (in which program and symbol) they occurred. Over a long run period the distribution of the ‘‘hits’’ shows how the computer processing power is spent in different parts of the system: programs, the kernel itself and the kernel modules. A user-space daemon collects these statistics and dumps them to disk so that they can be analyzed by the users with oprofile tools. Oprofile is low-overhead and as such a good tool to monitor individual programs as well as overall system performance when executing large jobs, or even sustained runs of weeks of jobs-and a great host of other tasks. It requires no code instrumentation, yet is capable of generating annotated source files where every line is annotated with the time spent on that line, or whatever the unit the measurements correspond to.
141
6.2. IGUANA GUI to oprofile Oprofile is a very handy and powerful tool, however with strong focus on single user with root access on a single workstation. Moreover the text output can scare the casual developers such as the hundreds of physicists who write analysis code for CMS. IGUANA provides a GUI to present oprofile results in a more friendly way and to ease the collection of results from remote workstations or clusters. The GUI enables multiple users to access profiling facilities without interfering with each other. The GUI supports the possibility to snapshot the profiling status of a running application and begin to show results from that point on. This allows different users to not interfere with each other as they would with the opsession command. It can get hold of results from a remote workstation, such as a single problem job in batch farm. It supports the auto-detection of oprofile tools and of profiling counters currently in use and can save and restore the collected data. 6.3. Valgrind: memory access scrutiny Valgrind [5] is a tool for debugging and profiling IA32 Linux programs. The original, or main, tool is a utility for finding memory management bugs in extremely effective ways, saving developers days if not weeks of bug-hunting time. By now the suite includes several other tools for finding multithreading bugs and profiling CPU cache usage, and more. Valgrind is non-intrusive, one simply runs the program by prefixing ‘‘valgrind’’ to the command line. It only requires user privileges and does not need any special system access. While it does slow down the program, unlike most other memory debuggers, it actually works fast enough to be of practical use. For example, interactive GUI or 3D visualization applications are perfectly usable under valgrind-though often sluggish, as if running over a network connection. More importantly, it is accurate: it very rarely if ever reports false positives and virtually always finds real problems,
ARTICLE IN PRESS 142
G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
providing very precise traceback to the original problem. Valgrind works by ‘‘virtualizing’’ the processor: once the program starts, it never sees the real processor again. Valgrind then instruments the original machine code for various checks, for example to track memory accesses, and runs it on the processor. Whenever it detects an error, it gives concise but detailed information about where the problem occurred and who allocated or freed the memory in question. Valgrind is an invaluable tool that makes it quite easy to find some of the most difficult errors in programs. It works reliably and rarely fails on programs as large and complex as entire CMS software. It is quickly becoming a very important tool for CMS developers. Work is ongoing to run existing test program runs under valgrind to spot new memory access problems automatically. There is one class of memory access problems valgrind does not detect: when the accessed memory is valid, but not the one intended. Apart
from this issue, if valgrind does not find memory access problems in one’s program under fairly representative test cases that exercise enough of the program, one can be relatively confident there are no serious bugs in it.
References [1] L. Tuura, Ignominy: tool for analysing software dependencies and for reducing complexity in large software systems, ACAT Proceedings, Moscow, June, 2002. [2] G. Alverson, G. Eulisse, S. Muzaffar, I. Osborne, L. Taylor, L. Tuura, Nucl. Instr. and Meth. A, (2004) these proceedings. [3] Mato, et al., SEAL: common core libraries and services for LHC applications, CHEP’03 Proceedings, CHEP’03, La Jolla, March, 2003. [4] S. Ashby, I. Osborne, J.P. Wellisch, C. Williams, Code organization and configuration management, Proceedings of CHEP 2001, Beijing, September, 2001. [5] ohttp://valgrind.kde.org4. [6] ohttp://oprofile.sf.net4. [7] ohttp://mozilla.org/performance/jprof.html4.
Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142 www.elsevier.com/locate/nima
A coherent environment of software improvement tools for CMS$ G. Eulisse, S. Muzaffar, I. Osborne, L. Taylor, L.A. Tuura Department of Physics, Northeastern University, Boston, MA 02115, USA Available online 29 July 2004
Abstract CMS has developed approximately one million lines of C++ code and uses many more from HEP, Grid and public domain projects. We describe a suite of tools which help to manage this complexity by measuring software dependencies, quality metrics, and CPU and memory performance. This coherent environment integrates and extends existing open-source tools where possible and provides new in-house components where a suitable solution does not already exist. This is a freely available environment with graphical user interface which can be run on any software without the need to recompile or instrument it. We have developed ignominy which performs software dependency analysis of source code, binary products and external software. CPU profiling is provided based on oprofile, with added features such as profile snapshots, distributed profiling and aggregate profiles for farm systems including server-side tools for collecting profile data. Finally, we have developed a low-overhead performance and memory profiling tool, MemProf, which can perform (gprof-style) hierarchical performance profiling, in a way that works with multiple threads and dynamically loaded libraries (unlike gprof). It also gathers exact memory allocation profiles including which code allocates most, in what sizes of chunks, for how long, where the memory is getting freed and where it is getting leaked. We describe this tool suite and how it has been used to enhance the quality of CMS software. r 2004 Elsevier B.V. All rights reserved. PACS: 07.05. t; 07.05.Bx Keywords: Software dependency analysis; Metrics; CPU and memory performance; Ignominy; CMS
1. Motivation $
This work was supported by the US National Science Foundation. Corresponding author. E-mail address: [email protected] (L. Taylor).
CMS has a number of software improvement tools collected into a coherent environment available to all our software developers. These exploit existing generally available tools as much as
0168-9002/$ - see front matter r 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.nima.2004.07.035
ARTICLE IN PRESS G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
possible but most have some difficulties coping with systems as large and complex as the LHC experiments have. Therefore, we developed some tools ourselves to supplement existing ones. These are collected into a project called IGNOMINY. The main component is the ignominy dependency scanner [1]. Ignominy was originally created in the IGUANA project [2, and references therein]. The tools are all not specific to CMS and are freely available. Contact [email protected] and/or visit http://iguana.cern.ch for more details. CMS tries to improve the quality of our software in four ways: by coding patterns that add assertions and other checks, by providing static analysis tools to give developers feedback on the software structure, with dynamic analysis tools to give meaningful quantitative numbers about the quality and performance of the software, and with test harness tools to validate program behavior.
2. Code quality improvement tools CMS has contributed an internal debugging body to the LCG SEAL foundation classes [3]. This allows developers to make assertions which produce detailed feedback when a failure occurs at run-time. One set of classes allows programs to produce detailed crash logs, including a stack trace and a list of all shared libraries loaded in memory. Other classes provide means to protect against and recover from low-memory conditions and other exceptional situations without losing details of what failed. 2.1. Static analysis tools Static software quality assurance tools provide a way to control the inter-component as well as the external software dependencies within projects plus to check and validate the configuration of the system [4]. Ignominy is our tool for the former while toolchecker does the latter task. 2.2. Dynamic analysis tools A great host of tools have been developed over time to analyze various aspects of programs at
139
run-time, especially for profiling and memory checking. Perhaps the largest disadvantage of most such tools is that they do not work on software as large and complex as what CMS has. Most older tools do not function at all with shared libraries, which are ubiquitous by now. Even fewer tools work correctly with multi-threaded programs. Most memory-checking tools are slow, or blow up memory usage, or are so inaccurate that they are unusable with programs as large and complex as LHC reconstruction systems. Nobody will try out or use a system that requires compiletime instrumentation, as there are millions of lines of code to recompile to get any useful and comprehensive statistics. We need tools that work robustly and accurately with multiple threads and large program sizes, without requiring any preinstrumentation. Recently two Linux tools have risen to the challenge and provide not only adequate functionality to developers, but excel in it: valgrind [5] and oprofile [6]. CMS has developed another tool, igprof, to supplement these, as well as a GUI for oprofile to make it easier to use, as described below.
3. Ignominy Ignominy is suite of perl and shell scripts plus configurations for a number of projects. It helps you understand the structure of software systems. Its primary component is a dependency scanner that distills information into graphical views and numerical metrics. Ignominy is designed to adapt to almost any reasonable structure and has been used to analyze several large projects, both in CMS and out-side: nearly all CMS offline software, ATLAS offline software, Geant4, ROOT and some LCG software. Ignominy determines dependencies between software packages in a large software system, where a package is an arbitrarily defined chunk of software. It can currently analyze Fortran, C, and C++ dependencies. Dependencies between packages are due to a variety of reasons but in general the more incoming and outgoing dependencies a package
ARTICLE IN PRESS 140
G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
has, the more difficult it is to develop, test, release, and support. Reduction of inter-package dependencies is a crucial goal for a large software system. Dependency analysis helps developers to choose sensible sets of loosely coupled packages with only required dependencies on other packages. 3.1. How ignominy works The working of ignominy is very straightforward. For example, in CMS a simple front end shell script (scram-ignominy) queries SCRAM [4], the CMS built tool, to collect all the information about the external software packages used to build and generate search/built rules, which are then used, along with any extra user defined search/ build rules, by the ignominy analyzer. The frontend is the only CMS-specific part and can be replaced by other front-ends adapted to other build systems. The Ignominy analyser then generates a dependencies database and an extensive log file, which are then used to generate diagrams. Diagrams can be customized by editing the ignominy configuration files, for example one can rename dependencies diagram nodes, set different colors for different nodes and drop nodes from diagrams.
4. igprof: the IGnominious PROFiler igprof is a tool which completes other opensource profiling and debugging facilities like oprofile and Valgrind. It provides performance and memory usage statistics. It is developed by the authors and allows a developer to profile program performance and memory allocation in different ways including the hunting for memory leaks. It is completely non-intrusive and is fast enough to profile the whole CMS software and it’s many associated external packages. The performance profiler works similar to Mozilla’s JProf [7] and many other profilers: a SIGPROF signal is emitted every so often and a custom signal handler keeps the statistics of where the sample happens. In the long run this results in the distribution of processing power among different functions in the executable and its shared libraries according to their use. The memory profiler part works by setting up custom hooks for memory allocation functions, keeping all the information about allocations and deallocations in a compact tree structure which in the end can be dumped to a file. The profiler requires no instrumentation nor source code changes as it is injected into executables using the LD_PRELOAD mechanism. It can also be activated by the program itself by dynamically loading the library.
3.2. What ignominy can tell you The results from ignominy analysis of your software system can tell you the system’s overall package structure, which would help determine the dependency hierarchy, independently manageable and releasable weakly coupled units and test/ release order of packages. The analysis results will tell you what you need to have before installing a package and which packages are using what. This information is useful among other things in determining the set of products that can easily be released from a project. The analysis results will also mark any cyclic dependencies. Study of the dependency diagrams and metrics allows one also to find false dependencies, and the additional tabular information allows a developer to quickly locate the origin of unexpected dependencies.
5. Toolchecker The toolchecker tool is a perl script to check the external software configured for a SCRAM project. It can be used by the release manager to validate the consistency of the configuration and by end users to verify that the project configuration actually works correctly on their particular computer. It compiles test programs and checks if the external software configured for a project is usable, and in particular whether there are any unresolved symbols. Toolchecker also checks the individual packages in the project to make sure they are complete. It also checks the compatibility and suitability of external packages and makes suggestions if something could be improved. It will
ARTICLE IN PRESS G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
also indicate which tools are in fact needed or not needed for the project. The toolchecker script first queries SCRAM to get the list of all external tools configured for the project, and then it queries the configuration of each tool. Using this information, the toolchecker executes tests, such as compiling test programs, for each tool.
6. Other tools 6.1. Oprofile: kernel level profiling Oprofile is an open-source community effort to take advantage of hardware profiling counters found in modern processors to do a system wide performance analysis. Modern processors, on Intel from Pentium onward, have the ability to count certain types of hardware events into special registers. Different processor types and generations support different countable events, but overall there are numerous types that help gain insight into the system performance. It is possible to set up the processor so that when one of these counters reaches a certain trigger level, a nonmaskable interrupt is issued. Oprofile provides a kernel module that traps these interrupts and keeps the statistics of where (in which program and symbol) they occurred. Over a long run period the distribution of the ‘‘hits’’ shows how the computer processing power is spent in different parts of the system: programs, the kernel itself and the kernel modules. A user-space daemon collects these statistics and dumps them to disk so that they can be analyzed by the users with oprofile tools. Oprofile is low-overhead and as such a good tool to monitor individual programs as well as overall system performance when executing large jobs, or even sustained runs of weeks of jobs-and a great host of other tasks. It requires no code instrumentation, yet is capable of generating annotated source files where every line is annotated with the time spent on that line, or whatever the unit the measurements correspond to.
141
6.2. IGUANA GUI to oprofile Oprofile is a very handy and powerful tool, however with strong focus on single user with root access on a single workstation. Moreover the text output can scare the casual developers such as the hundreds of physicists who write analysis code for CMS. IGUANA provides a GUI to present oprofile results in a more friendly way and to ease the collection of results from remote workstations or clusters. The GUI enables multiple users to access profiling facilities without interfering with each other. The GUI supports the possibility to snapshot the profiling status of a running application and begin to show results from that point on. This allows different users to not interfere with each other as they would with the opsession command. It can get hold of results from a remote workstation, such as a single problem job in batch farm. It supports the auto-detection of oprofile tools and of profiling counters currently in use and can save and restore the collected data. 6.3. Valgrind: memory access scrutiny Valgrind [5] is a tool for debugging and profiling IA32 Linux programs. The original, or main, tool is a utility for finding memory management bugs in extremely effective ways, saving developers days if not weeks of bug-hunting time. By now the suite includes several other tools for finding multithreading bugs and profiling CPU cache usage, and more. Valgrind is non-intrusive, one simply runs the program by prefixing ‘‘valgrind’’ to the command line. It only requires user privileges and does not need any special system access. While it does slow down the program, unlike most other memory debuggers, it actually works fast enough to be of practical use. For example, interactive GUI or 3D visualization applications are perfectly usable under valgrind-though often sluggish, as if running over a network connection. More importantly, it is accurate: it very rarely if ever reports false positives and virtually always finds real problems,
ARTICLE IN PRESS 142
G. Eulisse et al. / Nuclear Instruments and Methods in Physics Research A 534 (2004) 138–142
providing very precise traceback to the original problem. Valgrind works by ‘‘virtualizing’’ the processor: once the program starts, it never sees the real processor again. Valgrind then instruments the original machine code for various checks, for example to track memory accesses, and runs it on the processor. Whenever it detects an error, it gives concise but detailed information about where the problem occurred and who allocated or freed the memory in question. Valgrind is an invaluable tool that makes it quite easy to find some of the most difficult errors in programs. It works reliably and rarely fails on programs as large and complex as entire CMS software. It is quickly becoming a very important tool for CMS developers. Work is ongoing to run existing test program runs under valgrind to spot new memory access problems automatically. There is one class of memory access problems valgrind does not detect: when the accessed memory is valid, but not the one intended. Apart
from this issue, if valgrind does not find memory access problems in one’s program under fairly representative test cases that exercise enough of the program, one can be relatively confident there are no serious bugs in it.
References [1] L. Tuura, Ignominy: tool for analysing software dependencies and for reducing complexity in large software systems, ACAT Proceedings, Moscow, June, 2002. [2] G. Alverson, G. Eulisse, S. Muzaffar, I. Osborne, L. Taylor, L. Tuura, Nucl. Instr. and Meth. A, (2004) these proceedings. [3] Mato, et al., SEAL: common core libraries and services for LHC applications, CHEP’03 Proceedings, CHEP’03, La Jolla, March, 2003. [4] S. Ashby, I. Osborne, J.P. Wellisch, C. Williams, Code organization and configuration management, Proceedings of CHEP 2001, Beijing, September, 2001. [5] ohttp://valgrind.kde.org4. [6] ohttp://oprofile.sf.net4. [7] ohttp://mozilla.org/performance/jprof.html4.