- Email: [email protected]
Application of Negative Selection Algorithm (NSA) for test data generation of path testing
Accepted Manuscript Title: Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing Author: Shayma Mustafa Mohi-Aldeen Radziah Mohamad Safaai Deris PII: DOI: Reference:
S1568-4946(16)30503-8 http://dx.doi.org/doi:10.1016/j.asoc.2016.09.044 ASOC 3843
To appear in:
Applied Soft Computing
Received date: Revised date: Accepted date:
26-12-2015 23-8-2016 24-9-2016
Please cite this article as: Shayma Mustafa Mohi-Aldeen, Radziah Mohamad, Safaai Deris, Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing, Applied Soft Computing Journal http://dx.doi.org/10.1016/j.asoc.2016.09.044 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing
1- Shayma Mustafa Mohi-Aldeen1,3,*, 2- Radziah Mohamad1, 3- Safaai Deris2 1
Faculty of Computing, Universiti Teknologi Malaysia, 81310 UTM Skudai, Johor, Malaysia
2
Faculty Teknologi Creatif Dan Warisan, Universiti Malaysia Kelantan, 16100 Kelantan, Malaysia
3
College of Computer Science and Mathematics, University of Mosul, 41002, Mosul, Iraq
E-mail: [email protected], [email protected], and [email protected] *Corresponding author
Article title: Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing Reference: ASOC3843 Journal title: Applied Soft Computing Journal Corresponding author: shayma mustafa Mohi-Aldeen
1
Graphical abstract
Input the Source code of the program (Program Under Test)
Construct the control flow graph CFG from the source code of the program
Parameters set and maximum number of iterations
Apply NSA in generating the Paths of the programs and generating the test data
Comparison of techniques depending on evaluation metrics
1
Satisfy the Stopping Condition
3
2
4
5
6
Control Flow Graph CFG Program Paths Set Test Data Set
- Coverage % - Execution Time - No. of Generations
Abstract Path testing is one of the areas covered in structural testing. In this process, it is a key challenge to search for a set of test data in the whole search space to satisfy path coverage. Thus, finding an efficient method for generating test data automatically is a key issue in software testing. This paper proposed a method based on Negative Selection Algorithm (NSA) for generating test data to satisfy the path coverage criterion. The results show that NSA could reduce the number of test data generated and improve the coverage percentage, as well as enhance the efficiency of the test data generation process. To evaluate the performance of the method, results from the proposed method were compared with random testing and a previous work that used Genetic Algorithm and Ant Colony Optimization. The results demonstrate that NSA outperforms other methods in reducing the number of test data that cover all program paths even the difficult ones. Keywords: Software Testing; Path Testing; Automatic Test Data Generation; Negative Selection Algorithm. 1. Introduction Software testing is an essential process which is used for improving quality and increasing the reliability of the software. Software testing is a difficult, labor-intensive, and time-consuming process, accounting for more than half of the total cost of software development [1]. With the rapid growth of software being developed, generating a huge number of test data in software testing is needed. The main task of software testing is to generate a number of test data that satisfied the required adequacy criterion. However, path coverage is an extremely complicated, laborious and time-consuming process resulting to automatic generation of meaningful test data that can reduce testing time and improve the process of generation to satisfy a given test adequacy criterion [2]. The subject of automatically generated test data has gained the attention of researchers. However, comprehensive testing of the program’s input is unfeasible for any reasonable size of the program [3]. The process of generating test data is an undecidable problem and could be non-deterministic (NP-hard) or existing solutions are infeasible. The extremely non-linear construction of the program represents a problem for search algorithms to find efficient and optimal test data from a nonlinear, complicated and intermittent input in the searching space [3]. 2
The structural testing coverage criteria which have been proposed is used to determine which test data can be exercised or covered. The source code of the program could be divided into control flow coverage criteria, which is focused on testing the flow of the program and data flow coverage criteria, which focused on the flow of the data within the program [1, 2]. There are several criteria of the control flow coverage such as: statement coverage (execute every statement in the program at least once), branch coverage (every logical branch in the program is executed with both consequences at least once), condition coverage (every condition in the decision of the program is executed), multiple condition coverage (exercising all True and False combinations of conditions in compound decisions at least once), and path coverage (every distinct path in the program is executed at least once), which has been arranged from the weakest to the strongest type of coverage [1]. Path coverage is the strongest coverage criterion of test adequacy criteria in white box testing, i.e. every path in a program must be exercised at least once [4, 5]. Searching for test data to traverse each path is a very difficult process, since there are too many paths in the program and it may be infinite in the case of loops [5], or infeasible in case of nested branches i.e. difficult for finding the set of test data which can cover these paths. As a result, many researchers proposed different techniques for searching test data that could achieve high path coverage [4, 6]. Information is needed to guide and decide where is the best test data lie in the search space. This process involves searching for test data from a set of input data that causes a certain path in a program to be executed. Such process can also solve optimization problems. Many studies have been made on how to generate test data automatically in the past few decades and many researchers have proposed several techniques from time to time [7, 8]. The paper is prepared as follows: section 2 presents the works related to this study; section 3 explains the basic concept of the Negative Selection Algorithm; section 4 describes the proposed algorithm; section 5 presents the experimental evaluations; while conclusions are shown in section 6.
2. Related Works The generation of test data which satisfied the coverage criteria is an essential problem in software testing. Many researches have been made to solve this difficult problem. Several 3
methods have been used from time to time to generate test data automatically based on structural testing for different coverage criteria in order to increase the coverage percentage of program testing, decrease the number of test data, as well as reduce the time execution of the testing process [9]. Commonly, automatic test data generation techniques could be classified into: random test data generation, symbolic test data generation, dynamic test data generation, and searchbased test data generation techniques [1]. In random technique, the test data have been selected arbitrarily from an input domain. Although it is simple and could generate a lot of test data quickly, it generates a great number of redundant data [10]. In the early days of automation of software testing, majority of the test data generators used symbolic or dynamic technique. Symbolic test data generators are static, and assign symbolic values to variables instead of actual values in the program. Dynamic test data generators, on the other hand, require actual execution of the program with some selected inputs. If there are some desirable test requirements that have not been accomplished, the data which have been collected during the execution process could still be used to decide which data is nearest to satisfy the test requirements. This feedback helps in modifying the test inputs incrementally until all requirements are satisfied. However, these techniques were incompetent, time consuming, and got stuck in local optima of the searching space of the possible input data domain. Problems also happen when loops, array or pointers occur in Software Under Test (SUT) [5, 10]. Search-based test data generation is part of a very wide area of research in searchbased software engineering. It includes exploring for data in the input scope of the testing program to fulfil certain test data coverage criterion [6, 7]. Search-based test data generation techniques like Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), Simulated Annealing (SA), Artificial Bee Colony (ABC), and Memetic Algorithm (MA) are the most widely used methods in generating test data automatically, and meta-heuristic optimizing technique is used to find the best solution for automating the generation of test data by directing the search toward the most likely areas of the domain [8]. Automating the test data generation is still a popular topic of research. Although several methods have been recently proposed for automating the process of generating test data to fulfil the adequacy criteria, the application of these methods to find the optimal solution, minimum number of test data, maximum coverage of the required criterion, 4
reasonable completion time, and avoiding redundancies is still limited. Despite the advantages of these methods, there is a need to improve the efficiency and effectiveness of the test generation process, since these techniques may yield a great amount of test data or redundant test data that wastes both time and cost. In addition, there may also be cases of low probability execution path which are not covered. There may also be a problem with infeasible paths in the program. 2.1 Existing Literature on Search-Based Test Data Generation Techniques This section will present the existing literature on search-based test data generation techniques. Many search-based techniques have been used in generating test data automatically with the main goal to maximize the selected coverage with minimum test suite size and minimum time consumed. Local search-based methods such as Simulated Annealing (SA) proposed by Mansour and Salame [11] are simpler and offer better runtime than random testing and are more efficient for simple problems; however, they can become trapped in local optima and are very slow. Global search methods like GA, PSO, ACO, and ABC solved the problems of local search methods by finding more global optimal solutions, but these methods are more complex and difficult to implement. GA has been applied in Srivastava and Kim [12], Nirpal and Kale [13], Suresh and Rath [14], Rao et al. [15], Xibo and Na [16], Sun and Jiang [17], Singh [18], Gupta and Gupta [19], Ahmed and Hermadi [20], Shimin and Zhangang [10], AlZabidi et al. [21], Athar and Gupta [22], Ghiduk [5], and Hermadi et al. [4]. The results show that GA is better than random testing in terms of coverage and time consuming as well as in reducing the number of test data, but the efficiency and effectiveness of this method depend on its fitness function. PSO was applied in Li and Zhang [23], Peng [24], and Latiu et al. [25]. It has a fast convergence time and needs short computation time, which can effectively improve the slow convergence problems of the GA. However, it suffers from prematurity, affecting its speed and direction. For these reasons, Zhang et al. [26] and Li et al. [27] proposed a hybrid algorithm of GA with PSO to tackle these problems. Although the hybrid algorithm improved the number of generations, it increases the time needed for testing. In addition, the value of fitness is affected by the efficiency of the algorithm and in many problems, it gives the same coverage percentage as in GA and PSO. 5
Although the ACO has a better success rate compared to GA, it has several weaknesses namely increasing the size of the test suite and repeating the test data within the same test suite without any improvement in test coverage criteria. It also needs to update its pheromone value that negatively impacts its efficiency, and it requires a lot of time as shown in Srivastava et al. [28] and Dahiya et al. [29]. ABC has been applied by Mala and Mohan [30], Mala et al. [31], Jeya Mala et al. [32], Kulkarni et al. [33], and Lam et al. [3]. In generating test data, the results show that ABC is superior to GA and ACO in reducing the number of generations and time, but it is lacking in terms of coverage percentage. In addition, the colony size has an effect on the method’s performance. MA which combines local search and global search has been applied by Zhang and Wang [34]. MA is more efficient than pure local or global search, but it fails to cover new branch and requires more computational cost. Although the evolutionary algorithm could direct the search and generate number of test data to guarantee the maximum percentage of path coverage, it can still be improved in terms of its limitation in generating the optimal test data set [9, 35]. 3. Overview of Negative Selection Algorithm (NSA)’’ Negative Selection Algorithm (NSA)’is one of the most important methods in an Artificial Immune System’(AIS), which is a branch of computational intelligence models. AIS was inspired by the biological behaviour of Natural Immune System’(NIS), which is a compound biological network using fast and active techniques to defend the body versus a specified foreign body called antigens [36]. AIS is one of the different kinds of algorithms inspired by biologic systems, like evolutionary algorithms, swarm intelligence and neural networks, which have attracted a lot of researchers’ attention. Its objective is to develop immune-based techniques for solving complicated computation [36]. NSA was introduced by Forrest (1994) and it has been applied in fields such computer security, pattern recognition, anomaly detection and fault detection. The main goal of NSA is to distinguish between self and non-self samples while only the self samples are available. In particular, the goal is covering the non-self space with a specific number of detectors [37]. The basic idea of NSA is generating a number of detectors in the search space and then applying these detectors for classifying new data as self or non-self [38]. NSA consists of two stages: generation stage (also called training stage) and detection stage (also called testing 6
stage). Firstly, in the generation stage, a random process is used to generate the detectors and the process is monitored. The matched candidates are discarded and the residues are stored as detectors. The generation stage is terminated when a sufficient number of detectors (detector sets) are generated that could be determined by certain stopping criteria [39]. While in the detection stage, the detector sets which have been generated in the first stage are used to check whether the input samples correspond to self or non-self samples. If the new sample matched any detector in the detectors set, then it is classified as a non-self which indicates anomaly in most applications [40, 41]. Figure 1 describes the main concept of NSA. Applying Negative Selection Algorithm (NSA) in Test Data Generation Path testing is the main strategy in structural testing. A basic way to solve path testing is finding the specified input data in the search space that is likely to cover a path in the testing program. Many works have been done to generate test data automatically to achieve path coverage. This research proposed a new test data generator by using the NSA to generate a set of test data which satisfied the path coverage criterion and reduce the size of the test suite of the program under test by removing redundant test data. NSA is one of the most successful methods of AIS and it has been used in many applications, but its utilization in the field of test data generation is still limited. The algorithm consists of two stages, generation stage and detection stage, as explained in the previous section. In this research, we focus on the generation stage, which is responsible for generating the detector sets. In this study, it is used to generate the set of test data or test suite. At the beginning of this stage, a candidate test data is generated randomly using a process to find duplicate data within the set of test data, and if the new test data match any test data in the set, it is eliminated. This process prevents the test data from being redundant. Otherwise, the matching rule is determined by calculating the Hamming distance between the two test data and the minimum distance obtained will be compared with a threshold value. If the distance is less than the threshold value, then this test data is removed, otherwise it is added to the set of test data This process helps to cover as much of the search area as possible, and here it could cover more paths with limited number of test data for the SUT. This algorithm not only generates a specific number of test data for each path to guarantee test adequacy, but it could also reduce the redundancy of test data. By comparing the results with random test data generation technique, it is noted that this method is a much better technique. Figure 2 shows the overview of the method.
7
The path test data generation can be explained as follows: P represents the program under test, U represents the program paths, S is the input search space of P, the generator determines x S , where x is an input test data which caused the path u U to be traversed through the execution, and u is one of program paths U. Application of NSA in generating the set of test data represented by D begins with randomly generating a candidate population of test data that will then mature during the iterative process such that di D and x S , di (x, u), where di is a test data within the set D which covered the path u, f aff di , x is the matching rule that is the Hamming distance, which is used to compute the distance between the test data input with all the test data in the set, the minimum distance, τ, is checked, which represents the threshold value of affinity in order to eliminate and remove redundant detectors in test data. While the matching threshold (affinity matching distance) represents the allowable variation between the self samples and detectors, it is important and affects the NSA performance. With large threshold, more detectors are necessary to cover a larger region of the non self space which are not neighbours to the self space. On the other hand, if the threshold is too small, it may not be possible to generate a reasonable sized detector set from the available self. The selection of a suitable value is desirable for optimal performance of the algorithm. The determination of the threshold value is done by trial and error and the value which gives the best performance is selected. Thus, experimentally, the best threshold value used in the proposed method was set to 0.5. The proposed algorithm selects the test data with maximum distance which allows the algorithm to cover the whole search space, improving its ability to find test data and cover as many paths as possible. The iteration is continued until the stopping criterion has been met. This criterion is fulfilled when the maximum number of detectors has been met or all paths of the program have been covered. The steps of implementing NSA in test data generation algorithm and its detailed flow chart are shown in Figures 3 and 4 respectively.
4. Empirical Evaluation 8
4.1 Empirical Setup To evaluate the performance of the proposed method, a comparison has been made between programs that are popular from the literature. These programs have been widely used as test studies by researchers in the search-based software testing (SBST) field. The programs are written in JAVA and C++, and they contain a selection of IF, IF-ELSE statements with sequence, and nested complex structures like nested IF statements or deeply nested predicates. These programs also contain conditions with equality conditions or relational operators, i.e. (=, !=, <, >, <=, >=) and compound conditions of AND and OR logical operators, as well as existing arithmetic operators, i.e. (+, -, /, *) which made these programs appropriate in testing several techniques of test data generation. A complex data structure with different data types, i.e. integers, floats, characters and string have also been included in these programs [4, 21]. Table 1 illustrates these programs. For each program, the table lists a short description of the program, number of arguments in each program, lines of code (LOC), number of decision nodes, number of paths, and source. The common parameter of the method is the maximum number of detectors which was set to 5000. The experiment was run on 64-bits, 2.10 GHz Intel® Core™ i7 computer running Microsoft Windows 7 with 8 GB memory. The method was executed in the Delphi 5 platform.
4.2 Results and Discussion In this work, NSA is used for generating the test data automatically. The results show NSA’s ability in generating test data which can minimize the number of generations needed to cover the programs’ paths, reducing testing time. To investigate the effect of the proposed algorithm, we compared the results of this method with those obtained from random test data generation method. Generally, in NSA, the number of generated detectors in each generation or each run time (population size) has a large effect on the efficiency. The efficiency is represented by the number of generations needed to cover all paths of the program while the effectiveness is represented by the number of generated test data which needs to cover all paths of the program. The increase in population size will increase the effectiveness of the algorithm, but an increase in the number of the generations will affect the required testing time. 9
If the population size is increased, the coverage percentage of the paths will be increased. However, at the same time, the number of test data will increase and this is not desirable. When the population size is decreased, the number of generations will be increased, which means more time needed, so we need to choose a sufficient number of detectors. The algorithm has been performed and ran several times with different population sizes (different number of detectors in each generation) for each stopping time from 50 to 5000 generated detectors, and the results have been recorded (number of test data that cover the difficult path). In each run, the number of test data generated, number of generations, and execution time have been recorded, and the results showed that the best solution for the algorithm is when the population size is 500 and the number of generation is 5 we got more than 600 times to cover the equilateral path. The results showed that when the population number exceeds 500 or the generation exceeds 5, the relations move in linear fashion, and will decrease with the effectiveness which is represented by the number of test data needed to cover all paths and linearly increase the number of generations which affects the efficiency of the method. Figure 5 displays the relationship between the number of test data needed to cover the Equilateral path according to population size and number of generations. This is the most difficult path in the famous triangle classification test program (TriTyp). The experiments have been done on well-known benchmark programs, and in this section we will illustrate the results for each testing program. Triangle type classifier (Trityp), which is the most popular program in software testing, received three input numbers and is used to decide the type of triangle which is represented by: not a triangle, isosceles, equilateral or scalene. There are 12 branches which include two AND decisions and two OR decision conditions. It has a condition of equality and four nesting branches which increases the complexity of searching for the suitable test data. If each of the three input variables is 16 bits long, the program search space will be massive, and the tester will need to execute 2 16* 216* 216 = 248 test cases which is impossible [13]. This program seems to be a simple testing program, but in fact, it is not. By considering the size of the search problem space, there are only a few input test data that satisfy the equilateral triangle which is the most difficult path to satisfy by random testing, because the path of an equilateral triangle is traversed only if the three input values are equal and positive. For example, there are 215-1 equilateral types of triangle. Thus the probability of selecting three inputs randomly that traverse the equilateral 10
path is 215-1 / 248 which is approximately equal to 1 / 233. This makes it a typical program to evaluate several methods in software testing [43]. The source code and control flow graph (CFG) of the program are shown in Figure 6. As shown, the most difficult path to be covered in the program is the equilateral triangle. An example of how to represent the test data of the triangle classification program and how to calculate Hamming distance is presented in Figure 7, where a, b and c represents the value of three input numbers which are used to decide the type of triangle: whether it is not a triangle, isosceles, equilateral or scalene. Two test data have been presented and converted into binary form, and their Hamming distance is calculated. The effectiveness of the program that is represented by the number of test data needed to cover each path of the program is shown in Figure 8. The figure involves 100 detectors and a maximum of 10 generations. From the figure, we can see that the proposed method overcame random testing in generating data for equilateral path, achieving 100% of path coverage while random testing covered only 75%. The number of test data and execution time for all paths tested also decreased at least 90% as shown in Figure 9. Figure 10 presents the number of test data and paths covered using NSA and random testing for each benchmarking programs. As shown in the figures, NSA has more chance in generating test data for all paths even for difficult program paths. The proposed method decreases the number of test data by more than 75% and also reduces the time required to cover all paths of the programs by more than 85% as shown in Figure 11. To evaluate the efficiency and the effectiveness of the proposed method, we compared the proposed method’s results with Alzabidi’s [21] study which have the same type of coverage and used the same five well-known benchmark programs. In addition, in random testing, the comparison was based on the average values of two performance metrics which are average number of test data and average number of generations needed for total coverage. The results have been presented in Table 2. The table shows that the generated test data using the proposed method covers all programs paths with minimum number of iteration and reduced the average number of test
data produced. In the P-lie program, GA could not achieve full path coverage (denoted as 11
NA) since it failed to generate the test data which satisfy the path of the points that lie in the origin because the input (0,0) was not generated within the maximum number of generations. The table also shows the path coverage percentage of the programs compared with random testing and from the results, NSA has highest coverage percentage.
Figures 12 and 13 described the number of test data needed to cover all paths of the programs and the percentage of path coverage in each program by using random testing and NSA respectively. It is clear from Figure 12 that the number of generated test data in NSA is lesser than random testing technique, but the proposed method could not cover all paths covered by the NumDays program because of the complex structure of this program, while, Alltrue program needs more than 90000 detector to cover all paths. Figure 13 illustrates the percentage of path coverage for all programs used in the experiment. For further evaluation, we compared the triangle classification program results with previous works. Mao et al. [44] used ACO based test data generation and the results were compared with SA, GA and PSO. The findings indicate that ACO outperformed SA and GA, while it is comparable to PSO. Other comparisons were made with Singh [18] and Alzabidy [21] which used GA to generate test data for path coverage. Table 3 shows the comparison between our method with previous works. From the table of comparison, the results show that NSA needs less than the average number of test data since it has filtered and checked for duplicate test data, and lesser number of generations. However, it is also shown that ACO outperforms NSA in execution time. In order to increase the validity of the results, we implemented a statistical analysis, Ttest, of the results to compare the two methods. Table 4 shows the results for all programs with 10 runs for each program. The values in Table 4 were obtained by computing the number of test data which have been generated by the programs from 10 runs of each testing program. The table shows the mean, median, standard deviation (Std-Dev), minimum and maximum number of test data that were generated by using both algorithms. As shown in the table, the proposed method achieved better results and better efficiency compared with random testing. The small StdDev values in NSA indicates that the algorithm could achieve better coverage of the program 12
paths with less number of test data and less number of generations. This decreases the testing time because some paths are simpler to cover using NSA. The proposed NSA method could perform better in covering programs paths even when dealing with difficult paths that include Nested-If statements, loops, different data types, and even string compared to other methods. This method also produce good results in reducing the number of test data generated, the only issue shortcoming of this method is it could not cover the whole paths of the NumDays program (it only covers 84% of the program’s paths). 5. Conclusions This paper used NSA in generating the test data automatically to achieve path coverage criterion. List of benchmark programs have been used to validate the efficiency and effectiveness of the method in generating and reducing the test data. The experimental results showed that NSA is more efficient and more effective than random generation because of the method’s ability to move the search space to the optimal search range with fewer number of generations. The results also showed that NSA has the capability to reduce the number of test data when compared with ACO and GA used in previous works. For future works, the proposed test data generation using NSA algorithm could be enhanced by using adequate fitness function or hybridization with other optimization algorithms to overcome its lack in terms of full coverage.
13
References 1. Singh, Y., Software Testing, Cambridge University Press, 2011. 2. Manaseer, S., Manasir, W., Alshraideh, M., Hashish, N. A., & Adwan, O., Automatic Test Data Generation for Java Card Applications Using Genetic Algorithm, Journal of Software Engineering and Applications. 8(12), ( 2015) 603. 3. Lam, S. S. B., et al., Automated generation of independent paths and test suite optimization using artificial bee colony, Procedia Engineering. 30 (2012) 191-200. 4. Hermadi, I., et al., Dynamic stopping criteria for search-based test data generation for path testing, Information and Software Technology, 56(4), (2014) 395-407. 5. Ghiduk, A. S., Automatic generation of basis test paths using variable length genetic algorithm, Information Processing Letters, 114(6), (2014) 304-316. 6. Khan, R., & Amjad, M., Automatic Generation of Test Cases for Data Flow Test Paths Using K-Means Clustering and Generic Algorithm, International Journal of Applied Engineering Research, 11(1), (2016), 473-478. 7. McMinn, P., Search_based software test data generation: a survey, Software testing, Verification and reliability, 14 (2), (2004) 105-156. 8. Anand, S., et al., An orchestrated survey of methodologies for automated software test case generation, Journal of Systems and Software 86 (8), (2013) 1978-2001. 9. Mohi-aldeen, S. M., Deris, S., and Mohamad, R., Systematic mapping study in automatic test case generation. In New Trends in Software Methodologies, Tools and Techniques, Frontiers in Artificial Intelligence and Applications. IOS Press Ebooks., (2014) 703–720. 10. Shimin, L. and W. Zhangang, Genetic Algorithm and its Application in the path-oriented test data automatic generation, Procedia Engineering 15 (2011) 1186-1190. 11. Mansour, N. and M. Salame, Data generation for path testing, Software Quality Journal, 12(2), (2004) 121-136. 12. Srivastava, P. R. and T.-h. Kim, Application of genetic algorithm in software testing, International Journal of Software Engineering and Its Applications, 3 (4), (2009) 87-96. 13. Nirpal, P. B. and K. Kale, Using genetic algorithm for automated efficient software test case generation for path testing, International Journal of Advanced Networking and Applications, 2(6), (2011) 911-915. 14. Suresh, Y. and S. K. Rath, A genetic algorithm based approach for test data generation in basis path testing, International Journal of Soft Computing and Software Engineering, 3 (3), (2013) 326-323. 14
15. Rao, K. K., et al., Optimizing the software testing efficiency by using a genetic algorithm: a design methodology, ACM SIGSOFT Software Engineering Notes, 38 (3), (2013) 1-5. 16. Xibo, W. and S. Na, Automatic test data generation for path testing using genetic algorithms, Third International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), IEEE, (2011) 596-599. 17. Sun, J. H. and S. J. Jiang, An approach to automatic generating test data for multi-path coverage by genetic algorithm, Sixth International Conference on Natural Computation (ICNC), IEEE, (2010) 1533-1536. 18. Singh, B. K., Automatic efficient test data generation based on genetic algorithm for path testing, International Journal of Research in Engineering & Applied Sciences 2 (2), (2012) 1460-1472. 19. Gupta, M. and G. Gupta, Effective Test Data Generation using Genetic Algorithms, Journal of Engineering, Computers & Applied Sciences 1 (2), (2012) 17-21. 20. Ahmed, M. A. and I. Hermadi (2008). "GA-based multiple paths test data generator." Computers & Operations Research 35 (10): 3107-3124. 21. Al-zabidi, M. Kumar, D., Shaligram, D., Study of Genetic Algorithm for Automatic Software Test Data Generation, GALAXY International Interdisciplinary Research Journal, 1 ( 2), (2013), 65-74. 22. Athar, M. and Gupta, A., Automatic Generation of Test Suits by Applying Genetic Algorithm, IJCA Proceedings on International Conference on Advances in Computer Application ICACA, (2013) 1-6. 23. Li, A. and Y. Zhang, Automatic generating all-path test data of a program based on PSO, WRI World Congress on Software Engineering WCSE'09, IEEE, (2009) 189-193. 24. Peng, N., A PSO test case generation algorithm with enhanced exploration ability, Journal of Computational Information Systems 8 (14), (2012) 5785-5793. 25. Latiu, G. I., et al., Automatic Test Data Generation for Software Path Testing using Evolutionary Algorithms, Third International Conference on Emerging Intelligent Data and Web Technologies (EIDWT), IEEE, (2012) 1-8. 26. Zhang, S., et al., Automatic path test data generation based on GA-PSO. International Conference on Intelligent Computing and Intelligent Systems (ICIS), IEEE, (2010) 142146.
15
27. Li, K., et al., Breeding Software Test Data with Genetic-Particle Swarm Mixed Algorithm, Journal of computers, 5 (2), (2010) 258-265. 28. Srivastava, P. R., et al., Generation of test data using meta heuristic approach, TENCON 2008-2008 IEEE Region 10 Conference, IEEE, (2008) 1-6. 29. Dahiya, S. S., et al., Application of artificial bee colony algorithm to software testing, 21st Australian Software Engineering Conference (ASWEC), IEEE, (2010). 30. Mala, D. J. and V. Mohan, ABC Tester-Artificial bee colony based software test suite optimization approach, International Journal of Software Engineering, 2 (2), (2009) 1543. 31. Mala, D. J., et al., A non-pheromone based intelligent swarm optimization technique in software test suite optimization, International Conference on Intelligent Agent & MultiAgent Systems IAMA 2009, IEEE, (2009). 32. Jeya Mala, D., et al., Automated software test optimisation framework-an artificial bee colony optimisation-based approach, IET Software, 4 (5), (2010) 334-348. 33. Kulkarni, N. J., et al., Test case optimization using artificial bee colony algorithm, Advances in Computing and Communications, Springer, (2011) 570-579. 34. Zhang, B. and C. Wang, Automatic generation of test data for path testing by adaptive genetic simulated annealing algorithm, International Conference on Computer Science and Automation Engineering (CSAE 2011) IEEE, (2011) 38-42. 35. McMinn, P., Search-based software testing: Past, present and future, Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, (2011). 36. Dasgupta, D., Advances in artificial immune systems, Computational intelligence magazine, IEEE, 1 (4), (2006) 40-49. 37. Dasgupta, D. and F. Nino, Immunological computation: theory and applications, CRC Press, 2008. 38. Timmis, J., et al., Theoretical advances in artificial immune systems, Theoretical Computer Science, 403 (1), (2008) 11-32. 39. Ayara, M., et al., Negative selection: How to generate detectors, Proceedings of the 1st International Conference on Artificial Immune Systems (ICARIS), Canterbury, UK, (2002). 40. Idris, I. and A. Selamat, Improved email spam detection model with negative selection algorithm and particle swarm optimization, Applied soft computing, 22 (2014) 11-27. 16
41. Gong, M., et al., An efficient negative selection algorithm with further training for anomaly detection, Knowledge-Based Systems, 30 (2012) 185-191. 42. Jiang, S., Shi, J., Zhang, Y., & Han, H., Automatic test data generation based on reduced adaptive particle swarm optimization algorithm.Neurocomputing, 158 (2015), 109-116. 43. Rahnamoun,
R..,
Distributed
Black-Box
Software
Testing
Using
Negative
Selection, International Journal of Smart Electrical Engineering 2 (2013), 151-157. 44. Mao, Ch., Xiao, L., Yu, X., Chen, J., Adapting ant colony optimization to generate test data for software structural testing, Swarm and Evolutionary Computation 20 (2015) 23– 36. 45. Alshraideh, M., Bottaci, L., & Mahafzah, B. A., Using program data-state scarcity to guide automatic test data generation, Software Quality Journal, 18(1), (2010), 109-144.
17
Begin
Generate Random Detector Candidates
Begin
Input New sample Yes
Match Self? mples? Match Any Detector?
No Accept as New Detector
No
Yes Non-Self
Enough Detectors?
No End
Yes End (b) Generation Stage
(a) Detection Stage
Figure (1) Main concepts of Negative Selection Algorithm’(NSA)’[36]
18
Self
Figure (2) Overview of the proposed method
19
Input: 1- The program under test P and its input variable list X= (x1, x2, …, xn) where x S ; 2- The control flow graph CFG of P; 3- The parameters of the algorithm, Number of detectors max, τ; Output: 1- Set of test data D=(d1, d2, …, dn) which satisfied the path coverage; 2- The set of paths generated U=(u1, u2, …, un); Begin Step 1: Generate initial test data set randomly (Candidate population); Step 2: Test if the initial population reach to full coverage of path U Go to end Step 3: Generate a new test data x, where x S ; Step 4: Calculate the similarity of x with every test data di in D, di D which is represent the hamming distance and could be calculating from Eq. 1; N
f aff (di , x) (di x)
..
(1)
i 1
Step 5: Check the distance f aff di , x ; Step 6: if f aff di , x < τ then Remove the new test data x; Step 7: else add x to D; Step 8: end if Step 9: Repeat steps 3 to 8 until detecror number > max or D reach to full coverage of paths U; End
Figure (3) The NSA test data generation algorithm
20
Start
Input source code of SUT CFG Construction Generate Random Detector Candidate (test data)
Calculate Hamming distance with all other detectors from the equation h( x, y )
n
(X Y ) i 1
i
i
Get the minimum distance from among other detectors
Distance < τ?
Yes
Eliminate the new detector
No Add the new Detector to the Set of Detectors
No
Enough Detectors or full coverage?
Yes
- Coverage Percentage - Execution Time - No. of Generations - Set of Test Data
End
Figure (4) Flowchart of the proposed algorithm 21
22
(a)
(b)
Figure (5) Relationship between the number of test data that covers the equilateral path with (a) number of generations; (b) population size
23
Figure (6) The source code and control flow graph of Trityp
24
Figure (7) Example of test data and calculation of Trityp’s Hamming distance
25
Figure (8) Number of test data to cover Trityp program paths
Figure (9) Number of test data and testing time to cover all paths of Trityp program
26
(a)
(b)
(c)
(d)
Figure (10) Number of test data to cover the program paths for 10 generations in (a) Mid program; (b) Roots of Equation program; (c) Student Scale program; and (d) P-lie program
27
(a)
(b)
Figure (11) (a) Number of test data and (b) Testing time to cover all paths of the programs
Figure (12) Average number of test data to cover all paths of the programs
28
Figure (13) Path coverage percentage of the programs
29
Trityp
Mid QuadEq
STD-Scale P-lie MinMax
L-Search BubSort NumDays CalDay AllTrue32
Triangle Type: aims to receive three numbers as input and deciding the type of triangle they represent: equilateral, isosceles, scalene, or no triangle. Find the middle value between three values. Find the root of a quadratic equation and specify whether they are real or imaginary. Output the scale of a student based on the marks in three subjects. Find if x and y point lie on xaxis, y-axis or origin. Find the max and min value in array of integers and floats numbers. Searching a key in array of integer, floats, characters and string. Sort an array of elements in an increasing order. Calculating number of days between two dates. Calculates what day of the week for a specific date. Checks the Boolean values of an array if are all true.
30
Source
of Number Paths
Number of Decision Nodes
Description
No. of Loops
Abbreviation
Number Arguments
Benchmark
Lines of Code
of
Table (1) Description of the benchmark programs
3
18
0
6
7
[4, 5, 21, 44]
3
20
0
5
6
[5, 21]
3
28
0
6
7
[5, 32]
3
46
0
5
6
[21]
2
21
0
3
4
[21]
1
16
1
3
13
[4, 5, 21]
2
27
3
5
32
[4, 21]
1
20
2
1
15
[4, 42]
6
233
3
43
86
[5, 42, 44]
3
119
0
24
25
[1, 44]
1
7
1
32
232
[45]
Table (2) Experimental results of all programs compared with Alzabidi [21]
Program Name
Average Number of Generations needed for total coverage GA Random NSA
Average Number of Test Average Path data needed for total Coverage % coverage Random
NSA
GA [21]
Random
NSA
4945 23 4848 3816 2534 497.2 698.6 157.7 625.8 387 8658 NA NA
229 6 691 362 215 113.6 290.5 77.5 242.8 176.8 5338 NA 93966
11091 17787 7199 19070 14619 -
87 95
100 100
50 78 65 75 73 90 50 69 73 30 2
50 95 67 77 83 95 78 75 92 84 54
[21]
Trityp Mid P-Lie QuadEq STD- Scale MinMax L-Search L-SearchChar L-SearchString BubSort CalDay NumDays AllTrue32
50 1 49 39 26 5 7 2 7 4 87 NA NA
3 1 7 4 3 2 3 1 3 2 54 NA 940
10.3 1 NA 1 1 -
31
Table (3) Comparison with previous works Number of Generations Method
needed
for
total
coverage
Average
Average Number
Time (ms)
of Test data
NSA
3
31
229
ACO [44]
6
6.22
---
GA [18]
27
4000
24421
GA [21]
10
---
11091
32
Table (4) Statistical comparison of the results Program Trityp
Mid
P-lie
QuadEq
STD-Scale MinMax L-Search
L-SearchChar L-SearchString
BubSort
CalDay
AllTrue32
Method Runs Mean
Median Std-Dev
Min
Max
Random 10
4945
3187.5
3246.63
2520
10987
NSA
229
230.5
39.07
153
287
Random 10
23
23.5
7.69
13
35
NSA
6
5
3.8
3
15
Random 10
4848
4712.5 2037.12
1800
9048
NSA
691
675.5
590
845
Random 10
3816
3524.5 2509.91
1028
8023
NSA
362
383.5
91.06
225
460
Random 10
2534
2085.5 2406.89
1019
9122
NSA
215
213
74.24
118
293
Random 10
497.2
512
82.2905
321
618
NSA 10 Random 10
113.6
95
78
16
244
698.6
319
247.2
205
985
NSA
10
290.5
260
116.9
209
585
Random 10
157.7
137.5
48.2
101
235
NSA
77.5
70
32.9
43
147
Random 10
625.8
500
386.6
236
1290
NSA
242.8
221.5
110
107
410
Random 10
387
377
119.599
149
598
NSA
10
176.8
178.5
34.2403
132
215
Random 10
8658
7787.5 4720.3
3106
18168
NSA
5338
4436.5 5499.1
1930
19511
Random 10
NA
NA
NA
NA
NSA
93966
95600 5356.2
85630
99172
10
10
10
10
10
10
10
10
10
33
86.9
NA
S1568-4946(16)30503-8 http://dx.doi.org/doi:10.1016/j.asoc.2016.09.044 ASOC 3843
To appear in:
Applied Soft Computing
Received date: Revised date: Accepted date:
26-12-2015 23-8-2016 24-9-2016
Please cite this article as: Shayma Mustafa Mohi-Aldeen, Radziah Mohamad, Safaai Deris, Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing, Applied Soft Computing Journal http://dx.doi.org/10.1016/j.asoc.2016.09.044 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing
1- Shayma Mustafa Mohi-Aldeen1,3,*, 2- Radziah Mohamad1, 3- Safaai Deris2 1
Faculty of Computing, Universiti Teknologi Malaysia, 81310 UTM Skudai, Johor, Malaysia
2
Faculty Teknologi Creatif Dan Warisan, Universiti Malaysia Kelantan, 16100 Kelantan, Malaysia
3
College of Computer Science and Mathematics, University of Mosul, 41002, Mosul, Iraq
E-mail: [email protected], [email protected], and [email protected] *Corresponding author
Article title: Application of Negative Selection Algorithm (NSA) for Test Data Generation of Path Testing Reference: ASOC3843 Journal title: Applied Soft Computing Journal Corresponding author: shayma mustafa Mohi-Aldeen
1
Graphical abstract
Input the Source code of the program (Program Under Test)
Construct the control flow graph CFG from the source code of the program
Parameters set and maximum number of iterations
Apply NSA in generating the Paths of the programs and generating the test data
Comparison of techniques depending on evaluation metrics
1
Satisfy the Stopping Condition
3
2
4
5
6
Control Flow Graph CFG Program Paths Set Test Data Set
- Coverage % - Execution Time - No. of Generations
Abstract Path testing is one of the areas covered in structural testing. In this process, it is a key challenge to search for a set of test data in the whole search space to satisfy path coverage. Thus, finding an efficient method for generating test data automatically is a key issue in software testing. This paper proposed a method based on Negative Selection Algorithm (NSA) for generating test data to satisfy the path coverage criterion. The results show that NSA could reduce the number of test data generated and improve the coverage percentage, as well as enhance the efficiency of the test data generation process. To evaluate the performance of the method, results from the proposed method were compared with random testing and a previous work that used Genetic Algorithm and Ant Colony Optimization. The results demonstrate that NSA outperforms other methods in reducing the number of test data that cover all program paths even the difficult ones. Keywords: Software Testing; Path Testing; Automatic Test Data Generation; Negative Selection Algorithm. 1. Introduction Software testing is an essential process which is used for improving quality and increasing the reliability of the software. Software testing is a difficult, labor-intensive, and time-consuming process, accounting for more than half of the total cost of software development [1]. With the rapid growth of software being developed, generating a huge number of test data in software testing is needed. The main task of software testing is to generate a number of test data that satisfied the required adequacy criterion. However, path coverage is an extremely complicated, laborious and time-consuming process resulting to automatic generation of meaningful test data that can reduce testing time and improve the process of generation to satisfy a given test adequacy criterion [2]. The subject of automatically generated test data has gained the attention of researchers. However, comprehensive testing of the program’s input is unfeasible for any reasonable size of the program [3]. The process of generating test data is an undecidable problem and could be non-deterministic (NP-hard) or existing solutions are infeasible. The extremely non-linear construction of the program represents a problem for search algorithms to find efficient and optimal test data from a nonlinear, complicated and intermittent input in the searching space [3]. 2
The structural testing coverage criteria which have been proposed is used to determine which test data can be exercised or covered. The source code of the program could be divided into control flow coverage criteria, which is focused on testing the flow of the program and data flow coverage criteria, which focused on the flow of the data within the program [1, 2]. There are several criteria of the control flow coverage such as: statement coverage (execute every statement in the program at least once), branch coverage (every logical branch in the program is executed with both consequences at least once), condition coverage (every condition in the decision of the program is executed), multiple condition coverage (exercising all True and False combinations of conditions in compound decisions at least once), and path coverage (every distinct path in the program is executed at least once), which has been arranged from the weakest to the strongest type of coverage [1]. Path coverage is the strongest coverage criterion of test adequacy criteria in white box testing, i.e. every path in a program must be exercised at least once [4, 5]. Searching for test data to traverse each path is a very difficult process, since there are too many paths in the program and it may be infinite in the case of loops [5], or infeasible in case of nested branches i.e. difficult for finding the set of test data which can cover these paths. As a result, many researchers proposed different techniques for searching test data that could achieve high path coverage [4, 6]. Information is needed to guide and decide where is the best test data lie in the search space. This process involves searching for test data from a set of input data that causes a certain path in a program to be executed. Such process can also solve optimization problems. Many studies have been made on how to generate test data automatically in the past few decades and many researchers have proposed several techniques from time to time [7, 8]. The paper is prepared as follows: section 2 presents the works related to this study; section 3 explains the basic concept of the Negative Selection Algorithm; section 4 describes the proposed algorithm; section 5 presents the experimental evaluations; while conclusions are shown in section 6.
2. Related Works The generation of test data which satisfied the coverage criteria is an essential problem in software testing. Many researches have been made to solve this difficult problem. Several 3
methods have been used from time to time to generate test data automatically based on structural testing for different coverage criteria in order to increase the coverage percentage of program testing, decrease the number of test data, as well as reduce the time execution of the testing process [9]. Commonly, automatic test data generation techniques could be classified into: random test data generation, symbolic test data generation, dynamic test data generation, and searchbased test data generation techniques [1]. In random technique, the test data have been selected arbitrarily from an input domain. Although it is simple and could generate a lot of test data quickly, it generates a great number of redundant data [10]. In the early days of automation of software testing, majority of the test data generators used symbolic or dynamic technique. Symbolic test data generators are static, and assign symbolic values to variables instead of actual values in the program. Dynamic test data generators, on the other hand, require actual execution of the program with some selected inputs. If there are some desirable test requirements that have not been accomplished, the data which have been collected during the execution process could still be used to decide which data is nearest to satisfy the test requirements. This feedback helps in modifying the test inputs incrementally until all requirements are satisfied. However, these techniques were incompetent, time consuming, and got stuck in local optima of the searching space of the possible input data domain. Problems also happen when loops, array or pointers occur in Software Under Test (SUT) [5, 10]. Search-based test data generation is part of a very wide area of research in searchbased software engineering. It includes exploring for data in the input scope of the testing program to fulfil certain test data coverage criterion [6, 7]. Search-based test data generation techniques like Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), Simulated Annealing (SA), Artificial Bee Colony (ABC), and Memetic Algorithm (MA) are the most widely used methods in generating test data automatically, and meta-heuristic optimizing technique is used to find the best solution for automating the generation of test data by directing the search toward the most likely areas of the domain [8]. Automating the test data generation is still a popular topic of research. Although several methods have been recently proposed for automating the process of generating test data to fulfil the adequacy criteria, the application of these methods to find the optimal solution, minimum number of test data, maximum coverage of the required criterion, 4
reasonable completion time, and avoiding redundancies is still limited. Despite the advantages of these methods, there is a need to improve the efficiency and effectiveness of the test generation process, since these techniques may yield a great amount of test data or redundant test data that wastes both time and cost. In addition, there may also be cases of low probability execution path which are not covered. There may also be a problem with infeasible paths in the program. 2.1 Existing Literature on Search-Based Test Data Generation Techniques This section will present the existing literature on search-based test data generation techniques. Many search-based techniques have been used in generating test data automatically with the main goal to maximize the selected coverage with minimum test suite size and minimum time consumed. Local search-based methods such as Simulated Annealing (SA) proposed by Mansour and Salame [11] are simpler and offer better runtime than random testing and are more efficient for simple problems; however, they can become trapped in local optima and are very slow. Global search methods like GA, PSO, ACO, and ABC solved the problems of local search methods by finding more global optimal solutions, but these methods are more complex and difficult to implement. GA has been applied in Srivastava and Kim [12], Nirpal and Kale [13], Suresh and Rath [14], Rao et al. [15], Xibo and Na [16], Sun and Jiang [17], Singh [18], Gupta and Gupta [19], Ahmed and Hermadi [20], Shimin and Zhangang [10], AlZabidi et al. [21], Athar and Gupta [22], Ghiduk [5], and Hermadi et al. [4]. The results show that GA is better than random testing in terms of coverage and time consuming as well as in reducing the number of test data, but the efficiency and effectiveness of this method depend on its fitness function. PSO was applied in Li and Zhang [23], Peng [24], and Latiu et al. [25]. It has a fast convergence time and needs short computation time, which can effectively improve the slow convergence problems of the GA. However, it suffers from prematurity, affecting its speed and direction. For these reasons, Zhang et al. [26] and Li et al. [27] proposed a hybrid algorithm of GA with PSO to tackle these problems. Although the hybrid algorithm improved the number of generations, it increases the time needed for testing. In addition, the value of fitness is affected by the efficiency of the algorithm and in many problems, it gives the same coverage percentage as in GA and PSO. 5
Although the ACO has a better success rate compared to GA, it has several weaknesses namely increasing the size of the test suite and repeating the test data within the same test suite without any improvement in test coverage criteria. It also needs to update its pheromone value that negatively impacts its efficiency, and it requires a lot of time as shown in Srivastava et al. [28] and Dahiya et al. [29]. ABC has been applied by Mala and Mohan [30], Mala et al. [31], Jeya Mala et al. [32], Kulkarni et al. [33], and Lam et al. [3]. In generating test data, the results show that ABC is superior to GA and ACO in reducing the number of generations and time, but it is lacking in terms of coverage percentage. In addition, the colony size has an effect on the method’s performance. MA which combines local search and global search has been applied by Zhang and Wang [34]. MA is more efficient than pure local or global search, but it fails to cover new branch and requires more computational cost. Although the evolutionary algorithm could direct the search and generate number of test data to guarantee the maximum percentage of path coverage, it can still be improved in terms of its limitation in generating the optimal test data set [9, 35]. 3. Overview of Negative Selection Algorithm (NSA)’’ Negative Selection Algorithm (NSA)’is one of the most important methods in an Artificial Immune System’(AIS), which is a branch of computational intelligence models. AIS was inspired by the biological behaviour of Natural Immune System’(NIS), which is a compound biological network using fast and active techniques to defend the body versus a specified foreign body called antigens [36]. AIS is one of the different kinds of algorithms inspired by biologic systems, like evolutionary algorithms, swarm intelligence and neural networks, which have attracted a lot of researchers’ attention. Its objective is to develop immune-based techniques for solving complicated computation [36]. NSA was introduced by Forrest (1994) and it has been applied in fields such computer security, pattern recognition, anomaly detection and fault detection. The main goal of NSA is to distinguish between self and non-self samples while only the self samples are available. In particular, the goal is covering the non-self space with a specific number of detectors [37]. The basic idea of NSA is generating a number of detectors in the search space and then applying these detectors for classifying new data as self or non-self [38]. NSA consists of two stages: generation stage (also called training stage) and detection stage (also called testing 6
stage). Firstly, in the generation stage, a random process is used to generate the detectors and the process is monitored. The matched candidates are discarded and the residues are stored as detectors. The generation stage is terminated when a sufficient number of detectors (detector sets) are generated that could be determined by certain stopping criteria [39]. While in the detection stage, the detector sets which have been generated in the first stage are used to check whether the input samples correspond to self or non-self samples. If the new sample matched any detector in the detectors set, then it is classified as a non-self which indicates anomaly in most applications [40, 41]. Figure 1 describes the main concept of NSA. Applying Negative Selection Algorithm (NSA) in Test Data Generation Path testing is the main strategy in structural testing. A basic way to solve path testing is finding the specified input data in the search space that is likely to cover a path in the testing program. Many works have been done to generate test data automatically to achieve path coverage. This research proposed a new test data generator by using the NSA to generate a set of test data which satisfied the path coverage criterion and reduce the size of the test suite of the program under test by removing redundant test data. NSA is one of the most successful methods of AIS and it has been used in many applications, but its utilization in the field of test data generation is still limited. The algorithm consists of two stages, generation stage and detection stage, as explained in the previous section. In this research, we focus on the generation stage, which is responsible for generating the detector sets. In this study, it is used to generate the set of test data or test suite. At the beginning of this stage, a candidate test data is generated randomly using a process to find duplicate data within the set of test data, and if the new test data match any test data in the set, it is eliminated. This process prevents the test data from being redundant. Otherwise, the matching rule is determined by calculating the Hamming distance between the two test data and the minimum distance obtained will be compared with a threshold value. If the distance is less than the threshold value, then this test data is removed, otherwise it is added to the set of test data This process helps to cover as much of the search area as possible, and here it could cover more paths with limited number of test data for the SUT. This algorithm not only generates a specific number of test data for each path to guarantee test adequacy, but it could also reduce the redundancy of test data. By comparing the results with random test data generation technique, it is noted that this method is a much better technique. Figure 2 shows the overview of the method.
7
The path test data generation can be explained as follows: P represents the program under test, U represents the program paths, S is the input search space of P, the generator determines x S , where x is an input test data which caused the path u U to be traversed through the execution, and u is one of program paths U. Application of NSA in generating the set of test data represented by D begins with randomly generating a candidate population of test data that will then mature during the iterative process such that di D and x S , di (x, u), where di is a test data within the set D which covered the path u, f aff di , x is the matching rule that is the Hamming distance, which is used to compute the distance between the test data input with all the test data in the set, the minimum distance, τ, is checked, which represents the threshold value of affinity in order to eliminate and remove redundant detectors in test data. While the matching threshold (affinity matching distance) represents the allowable variation between the self samples and detectors, it is important and affects the NSA performance. With large threshold, more detectors are necessary to cover a larger region of the non self space which are not neighbours to the self space. On the other hand, if the threshold is too small, it may not be possible to generate a reasonable sized detector set from the available self. The selection of a suitable value is desirable for optimal performance of the algorithm. The determination of the threshold value is done by trial and error and the value which gives the best performance is selected. Thus, experimentally, the best threshold value used in the proposed method was set to 0.5. The proposed algorithm selects the test data with maximum distance which allows the algorithm to cover the whole search space, improving its ability to find test data and cover as many paths as possible. The iteration is continued until the stopping criterion has been met. This criterion is fulfilled when the maximum number of detectors has been met or all paths of the program have been covered. The steps of implementing NSA in test data generation algorithm and its detailed flow chart are shown in Figures 3 and 4 respectively.
4. Empirical Evaluation 8
4.1 Empirical Setup To evaluate the performance of the proposed method, a comparison has been made between programs that are popular from the literature. These programs have been widely used as test studies by researchers in the search-based software testing (SBST) field. The programs are written in JAVA and C++, and they contain a selection of IF, IF-ELSE statements with sequence, and nested complex structures like nested IF statements or deeply nested predicates. These programs also contain conditions with equality conditions or relational operators, i.e. (=, !=, <, >, <=, >=) and compound conditions of AND and OR logical operators, as well as existing arithmetic operators, i.e. (+, -, /, *) which made these programs appropriate in testing several techniques of test data generation. A complex data structure with different data types, i.e. integers, floats, characters and string have also been included in these programs [4, 21]. Table 1 illustrates these programs. For each program, the table lists a short description of the program, number of arguments in each program, lines of code (LOC), number of decision nodes, number of paths, and source. The common parameter of the method is the maximum number of detectors which was set to 5000. The experiment was run on 64-bits, 2.10 GHz Intel® Core™ i7 computer running Microsoft Windows 7 with 8 GB memory. The method was executed in the Delphi 5 platform.
4.2 Results and Discussion In this work, NSA is used for generating the test data automatically. The results show NSA’s ability in generating test data which can minimize the number of generations needed to cover the programs’ paths, reducing testing time. To investigate the effect of the proposed algorithm, we compared the results of this method with those obtained from random test data generation method. Generally, in NSA, the number of generated detectors in each generation or each run time (population size) has a large effect on the efficiency. The efficiency is represented by the number of generations needed to cover all paths of the program while the effectiveness is represented by the number of generated test data which needs to cover all paths of the program. The increase in population size will increase the effectiveness of the algorithm, but an increase in the number of the generations will affect the required testing time. 9
If the population size is increased, the coverage percentage of the paths will be increased. However, at the same time, the number of test data will increase and this is not desirable. When the population size is decreased, the number of generations will be increased, which means more time needed, so we need to choose a sufficient number of detectors. The algorithm has been performed and ran several times with different population sizes (different number of detectors in each generation) for each stopping time from 50 to 5000 generated detectors, and the results have been recorded (number of test data that cover the difficult path). In each run, the number of test data generated, number of generations, and execution time have been recorded, and the results showed that the best solution for the algorithm is when the population size is 500 and the number of generation is 5 we got more than 600 times to cover the equilateral path. The results showed that when the population number exceeds 500 or the generation exceeds 5, the relations move in linear fashion, and will decrease with the effectiveness which is represented by the number of test data needed to cover all paths and linearly increase the number of generations which affects the efficiency of the method. Figure 5 displays the relationship between the number of test data needed to cover the Equilateral path according to population size and number of generations. This is the most difficult path in the famous triangle classification test program (TriTyp). The experiments have been done on well-known benchmark programs, and in this section we will illustrate the results for each testing program. Triangle type classifier (Trityp), which is the most popular program in software testing, received three input numbers and is used to decide the type of triangle which is represented by: not a triangle, isosceles, equilateral or scalene. There are 12 branches which include two AND decisions and two OR decision conditions. It has a condition of equality and four nesting branches which increases the complexity of searching for the suitable test data. If each of the three input variables is 16 bits long, the program search space will be massive, and the tester will need to execute 2 16* 216* 216 = 248 test cases which is impossible [13]. This program seems to be a simple testing program, but in fact, it is not. By considering the size of the search problem space, there are only a few input test data that satisfy the equilateral triangle which is the most difficult path to satisfy by random testing, because the path of an equilateral triangle is traversed only if the three input values are equal and positive. For example, there are 215-1 equilateral types of triangle. Thus the probability of selecting three inputs randomly that traverse the equilateral 10
path is 215-1 / 248 which is approximately equal to 1 / 233. This makes it a typical program to evaluate several methods in software testing [43]. The source code and control flow graph (CFG) of the program are shown in Figure 6. As shown, the most difficult path to be covered in the program is the equilateral triangle. An example of how to represent the test data of the triangle classification program and how to calculate Hamming distance is presented in Figure 7, where a, b and c represents the value of three input numbers which are used to decide the type of triangle: whether it is not a triangle, isosceles, equilateral or scalene. Two test data have been presented and converted into binary form, and their Hamming distance is calculated. The effectiveness of the program that is represented by the number of test data needed to cover each path of the program is shown in Figure 8. The figure involves 100 detectors and a maximum of 10 generations. From the figure, we can see that the proposed method overcame random testing in generating data for equilateral path, achieving 100% of path coverage while random testing covered only 75%. The number of test data and execution time for all paths tested also decreased at least 90% as shown in Figure 9. Figure 10 presents the number of test data and paths covered using NSA and random testing for each benchmarking programs. As shown in the figures, NSA has more chance in generating test data for all paths even for difficult program paths. The proposed method decreases the number of test data by more than 75% and also reduces the time required to cover all paths of the programs by more than 85% as shown in Figure 11. To evaluate the efficiency and the effectiveness of the proposed method, we compared the proposed method’s results with Alzabidi’s [21] study which have the same type of coverage and used the same five well-known benchmark programs. In addition, in random testing, the comparison was based on the average values of two performance metrics which are average number of test data and average number of generations needed for total coverage. The results have been presented in Table 2. The table shows that the generated test data using the proposed method covers all programs paths with minimum number of iteration and reduced the average number of test
data produced. In the P-lie program, GA could not achieve full path coverage (denoted as 11
NA) since it failed to generate the test data which satisfy the path of the points that lie in the origin because the input (0,0) was not generated within the maximum number of generations. The table also shows the path coverage percentage of the programs compared with random testing and from the results, NSA has highest coverage percentage.
Figures 12 and 13 described the number of test data needed to cover all paths of the programs and the percentage of path coverage in each program by using random testing and NSA respectively. It is clear from Figure 12 that the number of generated test data in NSA is lesser than random testing technique, but the proposed method could not cover all paths covered by the NumDays program because of the complex structure of this program, while, Alltrue program needs more than 90000 detector to cover all paths. Figure 13 illustrates the percentage of path coverage for all programs used in the experiment. For further evaluation, we compared the triangle classification program results with previous works. Mao et al. [44] used ACO based test data generation and the results were compared with SA, GA and PSO. The findings indicate that ACO outperformed SA and GA, while it is comparable to PSO. Other comparisons were made with Singh [18] and Alzabidy [21] which used GA to generate test data for path coverage. Table 3 shows the comparison between our method with previous works. From the table of comparison, the results show that NSA needs less than the average number of test data since it has filtered and checked for duplicate test data, and lesser number of generations. However, it is also shown that ACO outperforms NSA in execution time. In order to increase the validity of the results, we implemented a statistical analysis, Ttest, of the results to compare the two methods. Table 4 shows the results for all programs with 10 runs for each program. The values in Table 4 were obtained by computing the number of test data which have been generated by the programs from 10 runs of each testing program. The table shows the mean, median, standard deviation (Std-Dev), minimum and maximum number of test data that were generated by using both algorithms. As shown in the table, the proposed method achieved better results and better efficiency compared with random testing. The small StdDev values in NSA indicates that the algorithm could achieve better coverage of the program 12
paths with less number of test data and less number of generations. This decreases the testing time because some paths are simpler to cover using NSA. The proposed NSA method could perform better in covering programs paths even when dealing with difficult paths that include Nested-If statements, loops, different data types, and even string compared to other methods. This method also produce good results in reducing the number of test data generated, the only issue shortcoming of this method is it could not cover the whole paths of the NumDays program (it only covers 84% of the program’s paths). 5. Conclusions This paper used NSA in generating the test data automatically to achieve path coverage criterion. List of benchmark programs have been used to validate the efficiency and effectiveness of the method in generating and reducing the test data. The experimental results showed that NSA is more efficient and more effective than random generation because of the method’s ability to move the search space to the optimal search range with fewer number of generations. The results also showed that NSA has the capability to reduce the number of test data when compared with ACO and GA used in previous works. For future works, the proposed test data generation using NSA algorithm could be enhanced by using adequate fitness function or hybridization with other optimization algorithms to overcome its lack in terms of full coverage.
13
References 1. Singh, Y., Software Testing, Cambridge University Press, 2011. 2. Manaseer, S., Manasir, W., Alshraideh, M., Hashish, N. A., & Adwan, O., Automatic Test Data Generation for Java Card Applications Using Genetic Algorithm, Journal of Software Engineering and Applications. 8(12), ( 2015) 603. 3. Lam, S. S. B., et al., Automated generation of independent paths and test suite optimization using artificial bee colony, Procedia Engineering. 30 (2012) 191-200. 4. Hermadi, I., et al., Dynamic stopping criteria for search-based test data generation for path testing, Information and Software Technology, 56(4), (2014) 395-407. 5. Ghiduk, A. S., Automatic generation of basis test paths using variable length genetic algorithm, Information Processing Letters, 114(6), (2014) 304-316. 6. Khan, R., & Amjad, M., Automatic Generation of Test Cases for Data Flow Test Paths Using K-Means Clustering and Generic Algorithm, International Journal of Applied Engineering Research, 11(1), (2016), 473-478. 7. McMinn, P., Search_based software test data generation: a survey, Software testing, Verification and reliability, 14 (2), (2004) 105-156. 8. Anand, S., et al., An orchestrated survey of methodologies for automated software test case generation, Journal of Systems and Software 86 (8), (2013) 1978-2001. 9. Mohi-aldeen, S. M., Deris, S., and Mohamad, R., Systematic mapping study in automatic test case generation. In New Trends in Software Methodologies, Tools and Techniques, Frontiers in Artificial Intelligence and Applications. IOS Press Ebooks., (2014) 703–720. 10. Shimin, L. and W. Zhangang, Genetic Algorithm and its Application in the path-oriented test data automatic generation, Procedia Engineering 15 (2011) 1186-1190. 11. Mansour, N. and M. Salame, Data generation for path testing, Software Quality Journal, 12(2), (2004) 121-136. 12. Srivastava, P. R. and T.-h. Kim, Application of genetic algorithm in software testing, International Journal of Software Engineering and Its Applications, 3 (4), (2009) 87-96. 13. Nirpal, P. B. and K. Kale, Using genetic algorithm for automated efficient software test case generation for path testing, International Journal of Advanced Networking and Applications, 2(6), (2011) 911-915. 14. Suresh, Y. and S. K. Rath, A genetic algorithm based approach for test data generation in basis path testing, International Journal of Soft Computing and Software Engineering, 3 (3), (2013) 326-323. 14
15. Rao, K. K., et al., Optimizing the software testing efficiency by using a genetic algorithm: a design methodology, ACM SIGSOFT Software Engineering Notes, 38 (3), (2013) 1-5. 16. Xibo, W. and S. Na, Automatic test data generation for path testing using genetic algorithms, Third International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), IEEE, (2011) 596-599. 17. Sun, J. H. and S. J. Jiang, An approach to automatic generating test data for multi-path coverage by genetic algorithm, Sixth International Conference on Natural Computation (ICNC), IEEE, (2010) 1533-1536. 18. Singh, B. K., Automatic efficient test data generation based on genetic algorithm for path testing, International Journal of Research in Engineering & Applied Sciences 2 (2), (2012) 1460-1472. 19. Gupta, M. and G. Gupta, Effective Test Data Generation using Genetic Algorithms, Journal of Engineering, Computers & Applied Sciences 1 (2), (2012) 17-21. 20. Ahmed, M. A. and I. Hermadi (2008). "GA-based multiple paths test data generator." Computers & Operations Research 35 (10): 3107-3124. 21. Al-zabidi, M. Kumar, D., Shaligram, D., Study of Genetic Algorithm for Automatic Software Test Data Generation, GALAXY International Interdisciplinary Research Journal, 1 ( 2), (2013), 65-74. 22. Athar, M. and Gupta, A., Automatic Generation of Test Suits by Applying Genetic Algorithm, IJCA Proceedings on International Conference on Advances in Computer Application ICACA, (2013) 1-6. 23. Li, A. and Y. Zhang, Automatic generating all-path test data of a program based on PSO, WRI World Congress on Software Engineering WCSE'09, IEEE, (2009) 189-193. 24. Peng, N., A PSO test case generation algorithm with enhanced exploration ability, Journal of Computational Information Systems 8 (14), (2012) 5785-5793. 25. Latiu, G. I., et al., Automatic Test Data Generation for Software Path Testing using Evolutionary Algorithms, Third International Conference on Emerging Intelligent Data and Web Technologies (EIDWT), IEEE, (2012) 1-8. 26. Zhang, S., et al., Automatic path test data generation based on GA-PSO. International Conference on Intelligent Computing and Intelligent Systems (ICIS), IEEE, (2010) 142146.
15
27. Li, K., et al., Breeding Software Test Data with Genetic-Particle Swarm Mixed Algorithm, Journal of computers, 5 (2), (2010) 258-265. 28. Srivastava, P. R., et al., Generation of test data using meta heuristic approach, TENCON 2008-2008 IEEE Region 10 Conference, IEEE, (2008) 1-6. 29. Dahiya, S. S., et al., Application of artificial bee colony algorithm to software testing, 21st Australian Software Engineering Conference (ASWEC), IEEE, (2010). 30. Mala, D. J. and V. Mohan, ABC Tester-Artificial bee colony based software test suite optimization approach, International Journal of Software Engineering, 2 (2), (2009) 1543. 31. Mala, D. J., et al., A non-pheromone based intelligent swarm optimization technique in software test suite optimization, International Conference on Intelligent Agent & MultiAgent Systems IAMA 2009, IEEE, (2009). 32. Jeya Mala, D., et al., Automated software test optimisation framework-an artificial bee colony optimisation-based approach, IET Software, 4 (5), (2010) 334-348. 33. Kulkarni, N. J., et al., Test case optimization using artificial bee colony algorithm, Advances in Computing and Communications, Springer, (2011) 570-579. 34. Zhang, B. and C. Wang, Automatic generation of test data for path testing by adaptive genetic simulated annealing algorithm, International Conference on Computer Science and Automation Engineering (CSAE 2011) IEEE, (2011) 38-42. 35. McMinn, P., Search-based software testing: Past, present and future, Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, (2011). 36. Dasgupta, D., Advances in artificial immune systems, Computational intelligence magazine, IEEE, 1 (4), (2006) 40-49. 37. Dasgupta, D. and F. Nino, Immunological computation: theory and applications, CRC Press, 2008. 38. Timmis, J., et al., Theoretical advances in artificial immune systems, Theoretical Computer Science, 403 (1), (2008) 11-32. 39. Ayara, M., et al., Negative selection: How to generate detectors, Proceedings of the 1st International Conference on Artificial Immune Systems (ICARIS), Canterbury, UK, (2002). 40. Idris, I. and A. Selamat, Improved email spam detection model with negative selection algorithm and particle swarm optimization, Applied soft computing, 22 (2014) 11-27. 16
41. Gong, M., et al., An efficient negative selection algorithm with further training for anomaly detection, Knowledge-Based Systems, 30 (2012) 185-191. 42. Jiang, S., Shi, J., Zhang, Y., & Han, H., Automatic test data generation based on reduced adaptive particle swarm optimization algorithm.Neurocomputing, 158 (2015), 109-116. 43. Rahnamoun,
R..,
Distributed
Black-Box
Software
Testing
Using
Negative
Selection, International Journal of Smart Electrical Engineering 2 (2013), 151-157. 44. Mao, Ch., Xiao, L., Yu, X., Chen, J., Adapting ant colony optimization to generate test data for software structural testing, Swarm and Evolutionary Computation 20 (2015) 23– 36. 45. Alshraideh, M., Bottaci, L., & Mahafzah, B. A., Using program data-state scarcity to guide automatic test data generation, Software Quality Journal, 18(1), (2010), 109-144.
17
Begin
Generate Random Detector Candidates
Begin
Input New sample Yes
Match Self? mples? Match Any Detector?
No Accept as New Detector
No
Yes Non-Self
Enough Detectors?
No End
Yes End (b) Generation Stage
(a) Detection Stage
Figure (1) Main concepts of Negative Selection Algorithm’(NSA)’[36]
18
Self
Figure (2) Overview of the proposed method
19
Input: 1- The program under test P and its input variable list X= (x1, x2, …, xn) where x S ; 2- The control flow graph CFG of P; 3- The parameters of the algorithm, Number of detectors max, τ; Output: 1- Set of test data D=(d1, d2, …, dn) which satisfied the path coverage; 2- The set of paths generated U=(u1, u2, …, un); Begin Step 1: Generate initial test data set randomly (Candidate population); Step 2: Test if the initial population reach to full coverage of path U Go to end Step 3: Generate a new test data x, where x S ; Step 4: Calculate the similarity of x with every test data di in D, di D which is represent the hamming distance and could be calculating from Eq. 1; N
f aff (di , x) (di x)
..
(1)
i 1
Step 5: Check the distance f aff di , x ; Step 6: if f aff di , x < τ then Remove the new test data x; Step 7: else add x to D; Step 8: end if Step 9: Repeat steps 3 to 8 until detecror number > max or D reach to full coverage of paths U; End
Figure (3) The NSA test data generation algorithm
20
Start
Input source code of SUT CFG Construction Generate Random Detector Candidate (test data)
Calculate Hamming distance with all other detectors from the equation h( x, y )
n
(X Y ) i 1
i
i
Get the minimum distance from among other detectors
Distance < τ?
Yes
Eliminate the new detector
No Add the new Detector to the Set of Detectors
No
Enough Detectors or full coverage?
Yes
- Coverage Percentage - Execution Time - No. of Generations - Set of Test Data
End
Figure (4) Flowchart of the proposed algorithm 21
22
(a)
(b)
Figure (5) Relationship between the number of test data that covers the equilateral path with (a) number of generations; (b) population size
23
Figure (6) The source code and control flow graph of Trityp
24
Figure (7) Example of test data and calculation of Trityp’s Hamming distance
25
Figure (8) Number of test data to cover Trityp program paths
Figure (9) Number of test data and testing time to cover all paths of Trityp program
26
(a)
(b)
(c)
(d)
Figure (10) Number of test data to cover the program paths for 10 generations in (a) Mid program; (b) Roots of Equation program; (c) Student Scale program; and (d) P-lie program
27
(a)
(b)
Figure (11) (a) Number of test data and (b) Testing time to cover all paths of the programs
Figure (12) Average number of test data to cover all paths of the programs
28
Figure (13) Path coverage percentage of the programs
29
Trityp
Mid QuadEq
STD-Scale P-lie MinMax
L-Search BubSort NumDays CalDay AllTrue32
Triangle Type: aims to receive three numbers as input and deciding the type of triangle they represent: equilateral, isosceles, scalene, or no triangle. Find the middle value between three values. Find the root of a quadratic equation and specify whether they are real or imaginary. Output the scale of a student based on the marks in three subjects. Find if x and y point lie on xaxis, y-axis or origin. Find the max and min value in array of integers and floats numbers. Searching a key in array of integer, floats, characters and string. Sort an array of elements in an increasing order. Calculating number of days between two dates. Calculates what day of the week for a specific date. Checks the Boolean values of an array if are all true.
30
Source
of Number Paths
Number of Decision Nodes
Description
No. of Loops
Abbreviation
Number Arguments
Benchmark
Lines of Code
of
Table (1) Description of the benchmark programs
3
18
0
6
7
[4, 5, 21, 44]
3
20
0
5
6
[5, 21]
3
28
0
6
7
[5, 32]
3
46
0
5
6
[21]
2
21
0
3
4
[21]
1
16
1
3
13
[4, 5, 21]
2
27
3
5
32
[4, 21]
1
20
2
1
15
[4, 42]
6
233
3
43
86
[5, 42, 44]
3
119
0
24
25
[1, 44]
1
7
1
32
232
[45]
Table (2) Experimental results of all programs compared with Alzabidi [21]
Program Name
Average Number of Generations needed for total coverage GA Random NSA
Average Number of Test Average Path data needed for total Coverage % coverage Random
NSA
GA [21]
Random
NSA
4945 23 4848 3816 2534 497.2 698.6 157.7 625.8 387 8658 NA NA
229 6 691 362 215 113.6 290.5 77.5 242.8 176.8 5338 NA 93966
11091 17787 7199 19070 14619 -
87 95
100 100
50 78 65 75 73 90 50 69 73 30 2
50 95 67 77 83 95 78 75 92 84 54
[21]
Trityp Mid P-Lie QuadEq STD- Scale MinMax L-Search L-SearchChar L-SearchString BubSort CalDay NumDays AllTrue32
50 1 49 39 26 5 7 2 7 4 87 NA NA
3 1 7 4 3 2 3 1 3 2 54 NA 940
10.3 1 NA 1 1 -
31
Table (3) Comparison with previous works Number of Generations Method
needed
for
total
coverage
Average
Average Number
Time (ms)
of Test data
NSA
3
31
229
ACO [44]
6
6.22
---
GA [18]
27
4000
24421
GA [21]
10
---
11091
32
Table (4) Statistical comparison of the results Program Trityp
Mid
P-lie
QuadEq
STD-Scale MinMax L-Search
L-SearchChar L-SearchString
BubSort
CalDay
AllTrue32
Method Runs Mean
Median Std-Dev
Min
Max
Random 10
4945
3187.5
3246.63
2520
10987
NSA
229
230.5
39.07
153
287
Random 10
23
23.5
7.69
13
35
NSA
6
5
3.8
3
15
Random 10
4848
4712.5 2037.12
1800
9048
NSA
691
675.5
590
845
Random 10
3816
3524.5 2509.91
1028
8023
NSA
362
383.5
91.06
225
460
Random 10
2534
2085.5 2406.89
1019
9122
NSA
215
213
74.24
118
293
Random 10
497.2
512
82.2905
321
618
NSA 10 Random 10
113.6
95
78
16
244
698.6
319
247.2
205
985
NSA
10
290.5
260
116.9
209
585
Random 10
157.7
137.5
48.2
101
235
NSA
77.5
70
32.9
43
147
Random 10
625.8
500
386.6
236
1290
NSA
242.8
221.5
110
107
410
Random 10
387
377
119.599
149
598
NSA
10
176.8
178.5
34.2403
132
215
Random 10
8658
7787.5 4720.3
3106
18168
NSA
5338
4436.5 5499.1
1930
19511
Random 10
NA
NA
NA
NA
NSA
93966
95600 5356.2
85630
99172
10
10
10
10
10
10
10
10
10
33
86.9
NA