- Email: [email protected]
Covering code behavior on input validation in functional testing
Information and Software Technology 51 (2009) 546–553
Contents lists available at ScienceDirect
Information and Software Technology journal homepage: www.elsevier.com/locate/infsof
Covering code behavior on input validation in functional testing Hui Liu *, Hee Beng Kuan Tan School of Electrical and Electronic Engineering, Block S2, Nanyang Technological University, Nanyang Avenue, Singapore 639798, Singapore
a r t i c l e
i n f o
Article history: Received 14 September 2007 Received in revised form 20 February 2008 Accepted 11 July 2008 Available online 24 August 2008 Index terms: Software testing Black-box testing White-box testing Input validation Test coverage
a b s t r a c t Input validation is the enforcement built in software systems to ensure that only valid input is accepted to raise external effects. It is essential and very important to a large class of systems and usually forms a major part of a data-intensive system. Most existing methods for input validation testing are specification-based. However, to test input validation more accurately, a code-based method is also required. In this paper, we propose an approach to extract path partition and input conditions from code for testing input validation. The path partition can be used to design white-box test cases for testing input validation. It can also be used to measure the coverage of input validation testing. The valid and invalid input conditions recovered can be used to check against the specifications and aid the test suite design in blackbox testing. We have also evaluated the proposed method through experimental study. Ó 2008 Elsevier B.V. All rights reserved.
1. Introduction Software testing techniques can be broadly classified into white-box testing and black-box testing. The former refers to the structural testing technique that designs test cases based on the information derived from source code. The latter refers to the functional testing technique that designs test cases based on the information from the specification. White-box and black-box testing are considered complementary to each other. Many researchers emphasize that, to test software more accurately, it is required to cover both specification and code behaviors [13]. A large class of systems process inputs submitted from users to raise external effects. Information systems and database applications belong to this class. These systems enforce that only valid inputs are accepted to raise external effects. Invalid inputs submitted are rejected and no external effects are raised. The enforcement in the systems is called input validation. For example, in a banking system, the amount withdrawn from a customer account submitted in any withdrawal transaction must be less than the available balance of the account. As such, input validation is incorporated into the system to enforce that only such valid withdrawal amount is accepted. Input validation plays a key role in the controlling user inputs submitted to a system. It plays a vital role on the robustness of the system. It is also an importance method for defending against today’s application attacks [21]. The design and implementation of input validation are usually carried out by a large population of * Corresponding author. Tel.: +65 67906528; fax: +65 67922971. E-mail addresses: [email protected] (H. Liu), [email protected] (H.B. Kuan Tan). 0950-5849/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.infsof.2008.07.001
application developers but not a small highly specialized software expert group. Therefore, to have an effective method for testing input validation is important for software quality assurance. We propose in this paper a method to recover implementation information from code to aid input validation testing. We propose a method to extract path partition and to recover valid and invalid input conditions from code. The path partition can be used in test case generation for testing input validation. It can also be used to measure the coverage of input validation testing. The recovered input conditions can be used to check against the specification and aid the test suite design in black-box testing. We have evaluated the proposed method through experimental study. The approach applies empirical software engineering, reverse engineering and program analysis together in software testing. The paper is organized as follows. Section 2 presents the empirical theory for the recovery of input validation information from code, and proposes a test coverage criterion to measure the adequacy of input validation testing. Section 3 discusses the proposed method and Section 4 reports our evaluation of the proposed method. Section 5 compares our work with related work. Section 6 concludes the paper.
2. An empirical theory for the recovery of code behavior on input validation We shall first review some terms that will be used throughout the paper. As defined by Sinha et al. [24], the control flow graph (CFG) of a program P is a directed graph G = (N, E), in which N contains a set of nodes and E = {(n, m) j n, m 2 N} contains edges that connecting the nodes. Each node in G represents a statement in
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
P, and each edge in the G represents possible flow of control between two statements in P. There is an entry node and an exit node in a CFG representing entry to and exit from P, respectively. A predicate node in G has two edges originate from it. The statement at a predicate node will be evaluated during the program execution to determine which out-going edge of the predicate node will be traversed. An out-going edge of a predicate node is called a branch and a branch predicate describes the conditions under which the branch will be traversed. In a control flow graph, the data dependence represents the data flow between program statements. A node y is data dependent on a node x if and only if there exists a variable v such that x defines v, y uses v and there is a path from x to y along with which v is not redefined. The control dependence is defined between conditional statement and the statements whose executions are controlled by the conditional statement. A node y dominates a node x if and only if every path from the entry node to x contains y. A node y postdominates a node x if and only if every path from x to the exit node contains y. A node y is control dependent on a node x if and only if x has successors x0 and x00 such that y postdominates x0 but y does not postdominate x00 . The transitive property of data dependence is inherent. We say a node y is transitively data dependent on a node x if there exists a sequence of nodes (w0 = x, w1,w2,. . .,wn = y) in the control flow graph such that n P 1 and wj is data dependent on wj1 for all j, 1 6 j 6 n. Likewise, the transitive control dependence can be defined based on a sequence of control dependence edges. A node y is transitively control dependent on node x if there exists a sequence of nodes (w0 = x, w1,w2,. . .,wn = y) in the control flow graph such that n P 1 and wj is control dependent on wj1 for all j, 1 6 j 6 n. In the CFG of a program, a node is called an effect node if it raises external effect. For example, a node that updates database is an effect node. A node u is called an input node if u reads an input directly from the external environment or an input received from the external environment is accessible at u and all other nodes at which the input is also accessible are dominated by u. In either case, the input is called the input received from u. For example, a node that reads a string directly from console is an input node of the first type. In a web application, a server program is activated upon the submission of an input through a form displayed. In this case, the input is usually comprised of many data items, and the first node of the server program is an input node of the second type. Through the empirical study, we observed that a certain type of node in a control flow graph of a program is critical to the implementation of input validation in a program. Such nodes are formally defined next. Let u be an input node of a program. A predicate node d is a validation node of u if the following two conditions hold: (1) d is transitively data dependent on a node that references to inputs received from u. (2) There is a path from d to an effect node that is transitively control dependent on d, and there is also a path from d to an exit node that does not contain any effect node. The two conditions emphasize that a validation node should check the validity of inputs submitted by users and decide whether the inputs should be used to raise any external effect. In addition, we say that an effect node f is influenced by an input node u if and only if f is transitively control dependent on a validation node of u, or f is transitively data dependent on a node that references to inputs received from u. For example, Fig. 1 shows a simplified program for cash withdrawal from an automated teller machine (ATM). In the control
547
flow graph of the program, node 1, 4 and 7 are three input nodes and node 12 is the only effect node. It is easy to identify that node 2 is the only validation node of node 1, node 5 is the only validation node of node 4, and node 8, 10 are validation nodes of node 7. Cleary, the effect node 12 is influenced by all three input nodes in the program. 2.1. Input validation partition As the code behavior on input validation is path-based and theoretically the number of paths through a CFG is infinite, practically, we need to base on a set of paths that satisfies a path coverage criterion for code-based input validation testing. Let J be a set of input nodes in the CFG of a program. Let S be a set of paths through the CFG that satisfies a path coverage criterion p. We then partition the paths in S that contain some validation nodes of the nodes in J in such a way that paths are put into the same class if they satisfy the following two conditions: (1) They contain the same set of validation nodes of any input node in J. (2) They follow the same branch at each of the above-mentioned validation nodes. Hence, the set of validation nodes and the branch traversed at each of these nodes form a partition criterion. The resulting partition is called an input validation partition of J under p. If J has only one input node u, we call the partition the input validation partition of u under p. For example, in the CFG shown in Fig. 1, the set of paths S that satisfies the basis path coverage criterion [26] is {(entry, 1, 2, 3, end), (entry, 1, 2, 4, 5, 6, end), (entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}. The set of input nodes J include node 1, 4, and 7. It is not difficult to verify that the input validation partition of J under the basis path coverage has five classes. Each class has exactly one path in S. Given a path coverage criterion p, we can measure the adequacy of a test suite for testing input validation based on the code characteristics. Let X be an input validation partition of a set of input nodes J under p. Clearly, a test suite is adequate for testing the input validation implemented in a program with regards to J and p if at least one path in each class in X is exercised. Hence, we can compute the input validation coverage of a test suite T as the percentage of the classes in X from which at least one path is exercised. That is:
IVCðT; PÞ ¼
CEðT; PÞ 100% CðT; PÞ
where C(T,P) is the total number of classes in X and CE(T,P) is the number of classes in X from which at least a path is exercised by a test case in T. 2.2. Recovery of valid and invalid input conditions In majority of cases, an input received from an input node is validated by checking it against some conditions using predicate nodes before any effect is raised. A condition that must be satisfied before the use of the input to raise effect is a valid condition of the input. Conversely, a condition to deny the use of the input to raise effect is an invalid condition of the input. Let u be an input node in the CFG of a program. Let Xu be the input validation partition of u under the Ct coverage with k = 0. The Ct coverage criterion is introduced by Bently and Miller [3] that defines a minimum iteration count k, so that a manageable set of paths can be obtained. For the case of k = 0, the Ct coverage
548
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
begin 1 read card number; 2 if (card number is not valid) then 3 show “card not valid” error; else 4 read PIN from user; 5 if (PIN does match card PIN) then 6 show “invalid PIN” error; else 7 read withdrawal amount M 8 if (M > withdrawal limit) then 9 show “limit exceeded” error; else 10 if (M > account balance) then 11 show “balance exceeded” error; else 12 debit the amount M from user account; endif endif endif endif end
(a) The pseudocode for the program
Fig. 1. A simplified ATM cash withdrawal program.
identifies all the non-iterative paths and iterative paths that do not repeat any loop. Clearly, a class K in the partition Xu will fall under one of the following two types: (1) If there is a path in K that contains an effect node influenced by u, all the paths in K accept the input received from u. (2) Otherwise, all the paths in K reject the input received from u. We call the first and the second type of classes a valid class and an invalid class, respectively of u. From the definition of input validation partition, the conjunction of branch predicates of the branches at all the validation nodes of u that any path in K follows is identical. We denote it as CK and call it the input condition of class K. Clearly, the checking of the input received from an input node u against a condition to decide whether an effect should be raised is carried out by the validation nodes of u. This directly implies that the input condition of a valid class is a valid condition of the input received from u. The input condition of an invalid class is an invalid condition of the input received from u. Next, we shall summarize this invariant property in Property 1.
Property 1. Let Xu be an input validation partition of an input node u under Ct k = 0 coverage criterion. Then, VCu = {the input condition of KjK 2 Xu and K is a valid class of u} and ICu = {the input condition of KjK 2 Xu and K is an invalid class of u} are the sets of valid and invalid input conditions, respectively of the input received from n, that are implemented in the program. For the program shown in Fig. 1, as discussed earlier, node 1, 4 and 7 are input nodes. Under the Ct k = 0 coverage criterion, the input validation partition of node 1 has two classes: {(entry, 1, 2, 3, end)} and {(entry, 1, 2, 4, 5, 6, end), (entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}, the input validation partition of node 2 also has two classes: {(entry, 1, 2, 4, 5, 6, end)} and {(entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}, and the input validation partition of node 7 has three classes: {(entry, 1, 2, 4, 5, 7, 8, 9, end)}, {(entry, 1, 2, 4, 5, 7, 8, 10, 11, end)}, and {(entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}. Since node 12 is the only effect node in the program, it is not difficult to identify the invalid classes and valid classes of each input node. For example, in the input validation partition of node 7, the first two classes are invalid classes of node 7, whereas the last class is a valid class of node 7. Since node 8 and
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
9 are validation nodes of node 7, valid and invalid conditions of node 7 can be derived by computing the conjunction of the branch predicates at the two nodes along a path in each class in the partition. Therefore, {(M > withdrawal limit), NOT(M > withdrawal limit) AND (M > account balance)} is the set of invalid conditions of the amount M received from node 7, and {NOT(M > withdrawal limit) AND NOT (M > account balance)} is the set of valid conditions for the amount M received from node 7. As input validation is a major feature in a system, the valid and invalid conditions of input should be obtainable from some form of system documentation. Therefore, the invalid and invalid input conditions extracted from code can be used to check against the corresponding conditions defined in the specification for any discrepancy to be revealed. In addition, the extracted input conditions provide the testers with true implementation information that supplements the specification, and hence can be used to aid the test suite design in black-box testing. 3. The proposed method Based on the discussion in Section 2, we propose a method to testing method that covers the code behavior on input validation in functional testing. The method comprises of two stages: (1) Compute valid and invalid input conditions from program source code; (2) Perform functional testing with the aid of the information computed in the first stage. 3.1. Stage 1 – Compute valid and invalid input conditions In this stage, all the valid and invalid input conditions for each input received in a program are automatically recovered from program source code. Fig. 2 shows the algorithm that comprises two steps. In Step 1, the CFG of a program is constructed first. Usually, source code specific tools can be used to perform static analysis, from which the control dependency, data dependency and other program characteristics can be identified. Next, the set of input nodes and effect nodes need to be identified. An effect node represents an output state or an action that is critical and major to a
program. For example, in database applications, the statements that perform database transactions or write to a file are obviously more critical to the ones that output warning messages. The identification of effect nodes would not be difficult to domain experts or developers who understand the program well. Hence, once the types of effects in a program have been decided, the effect nodes can be automatically identified by program analysis tools through standard programming interfaces. The input nodes can then be identified in a similar way. For programs that have multiple input nodes and effect nodes, it is also essential to identify and understand the complex relationships between those input nodes and effect nodes before the input conditions can be recovered. An input node usually influences more than one effect node and an effect node could be influenced by more than one input nodes. Such information can be obtained through data dependence and control dependence analysis or with the use of program slicing. In Step 2, the set of valid and invalid input conditions of each input received in the program are computed. To achieve that, three sub-steps are performed. First, a set S of paths is computed under the Ct coverage with k = 0 [3]. Next, for each input node u in the program, the input validation partition X u of u is computed as follows: for each path in S, if it contains at least one validation node of u, the sequence of validation nodes of u contained in the path and the branch traversed at each of these nodes are identified as a partition criterion. This criterion forms a class in the partition and is included in Xu if it has not been included yet. The path is then put into the class in Xu according to the partition criterion. If the path contains an effect node influenced by u, the corresponding class is classified as a valid class of u; otherwise, the corresponding class is classified as an invalid class of u. Finally, based on Property 1, the input condition of each class in Xu is computed and hence, the sets of valid and invalid conditions of the input received from u, respectively can be computed. 3.2. Stage 2 – Perform functional testing Equivalence class testing and cause-effect testing are two fundamental functional testing techniques [18,22]. Equivalence class
Comp_IV_Info (program P) Output: For each input node u, an input validation partition Ωu of u, and the sets VCu and ICu of valid and invalid conditions of input received from u respectively. Step 1.
549
Construct the CFG of P and identify the set U of input nodes and the set F of effect nodes. Next, compute the following information: 1) For each input node u in U, the set of effect nodes F(u) that are influenced by u. 2) For each input node u in U, the set of validation nodes D(u) of u.
Step 2.
Compute the valid and invalid conditions of input received from each input node u in U. 2.1 Compute a set S of paths under the Ct coverage with k=0. 2.2 Compute the input validation partition Ωu of u: For each path p in S, a) Exclude p from S if it does not contain any validation node of u. b) Identify the partition criterion Q that consists of the sequence of validation nodes of u in p and the branch traversed at each of these nodes. c) Include Q in set Ωu if it has not been included yet. d) Add p into the class of paths that links to the criterion Q. e) If p contains an effect node in F(u), Q is marked as a valid class of u; otherwise, Q is marked as an invalid class of u. 2.3 Compute the set VCu of valid conditions and the set ICu of invalid conditions of u: a) Extract the input conditions of all the valid class of u and include them in VCu. b) Extract the input conditions of all the invalid class of u and include them in ICu. Fig. 2. An algorithm to compute input validation information.
550
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
testing partitions the input space into equivalence classes according to the input conditions. Test cases are designed by selecting at least one condition from each equivalence class. Boundary value analysis complements the equivalence partitioning by selecting test cases at the edges of a class. Cause-effect testing uses cause-effect graph and decision table as a tool to design testing cases, in which input conditions form the causes and actions form the effects. A test case is then designed for each rule in the decision table. In principle, to derive a comprehensive test suite for testing input validation from black-box testing methods, all possible input conditions would be considered. However, for a software system with ordinary complexity, the number of input conditions can be very large; hence, such approach would be too expensive to be used with limited testing resources. In such cases, software testers often need to use their intuition and professional experiences to select effective test cases [6]. However, this approach relies on chance to determine the effectiveness of the resulting test suite as the assumption could fail for the specific implementation under test. Furthermore, even with adequate test cases, black-box testing alone may not reveal all faults in a system when the system implements the behavior that is not in the specification. Clearly, we can use the valid and invalid input conditions recovered from stage 1 to aid the testers in black-box input validation testing. Firstly, the valid and invalid input conditions can be used to check against the specifications for any discrepancies to be revealed. Secondly, the recovered input conditions provide the testers the implementation information of a program, and thus aid the design of test cases. Based on the implementation information, the testers can avoid the selection of unnecessary combinations of input conditions that are not implemented in the program. Consequently, fewer test cases could be selected for adequate testing. Furthermore, based on the input validation partition, white-box test cases can be designed to supplement the black-box test cases. The input validation coverage metric can also be used to measure the adequacy of the test suite designed.
Program Source Code
PathTracer Kaffe JVM J Trace
Program Analyzer Preprocessor
Test case execution Execution Traces
JABA CFG IV Miner IV Extractor
IV Partitions IV Coverage Analyzer
Program Slicer
Program Slices
Validation Conditions
Coverage Report IV: Input Validation
Fig. 3. The architecture of the prototype tool.
It is used to study the variables involved in these conditions for reconciling them against the specification. Path Tracer contains a JTrace which is built on top of the Kaffe Java virtual machine [15]. The JTrace uses the components in the open source tool JSlice [14] and is able to output the execution trace in terms of source line numbers and the corresponding class names. The execution trace is then input to IV coverage analyzer for the computation of the input validation coverage. 4.2. Experimental study
4. Evaluation We have conducted experimental study to access the feasibility and usefulness of the proposed method. To aid the experiment, we have developed a prototype tool based on the Java Architecture for Bytecode Analysis (JABA) from Georgia Institute of Technology [12]. 4.1. Prototype tool implementation The prototype tool is targeted for web database applications written in Java. As shown in Fig. 3, it consists of four components: a program analyzer, an input validation miner (IV miner), a path tracer and an input validation coverage analyzer (IV coverage analyzer). Program analyzer uses JABA’s API to analyze Java programs. By parsing a class file of a Java program, JABA performs control and data flow analysis and builds the control flow graph of the program. The Preprocessor is to prepare all the files in a system into class files for inputting to JABA to build the control flow graph of a program. This component implements Step 1 in the algorithm shown in Fig. 2. IV Miner includes two modules: an input validation extractor (IV extractor) and a program slicer. IV extractor implements Step 2 in the algorithm shown in Fig. 2. For each program that processes inputs submitted from user to raise external effect, for each input node in the program, IV Extractor generates a report containing the sets of valid and invalid conditions of inputs received from the input node. Program Slicer is a utility for computing program slices.
We have conducted two controlled experiments in which various types of faults related to the input validation were seeded into well-tested systems. From our analysis of functional testing techniques and further discussion with the software engineers in the industry, we come to the conclusion that the most frequently used methods to test input validation are equivalence class testing supplemented with boundary value analysis. Cause-effect testing is a testing method that explores all the combinations of input conditions; in practice, however, the complexity of the method and the large test suite it requires make it is difficult to be applied for most projects. Nevertheless, for the purpose of more thorough evaluation, in the experiments, we compare the proposed method with both methods. The first experiment was performed on an open source system called Smacs downloaded from Sourceforge [25]. Smacs is a webbased application for the management of the working of temporary staff in an organization. The size of the source code is 13462 LOC. The second experiment was performed on a system called JAuction that developed from an MSc Software Engineering project. JAuction provides the functionalities for hosting online auctions where registered users can both post and bid on items. The system was developed by following the Rational Unified Process and the size of the source code is 18342 LOC. Before the commencement of each experiment, we have prepared for both systems the software design specifications. In addition, each system has been thoroughly tested to ensure that the system behaves in accord with its specification. Besides the system used, the design of the two experiments is the same.
551
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
We then seeded into each system the faults that are commonly made in the implementation of input validation. We introduced the following seven types of faults: (1) Boolean operator fault (BOF): The changes of AND, OR and NOT operators in a validation node. (2) Relational operator fault (ROF): The changes of less than, greater than, less or equal to and greater or equal to operators in a validation node. (3) Variable and constant fault (VCF): A variable or constant in a validation node is replaced by another variable or constant. (4) Predicate insertion fault (PIF): An extraneous clause is inserted into a validation node. (5) Predicate omission fault (POF): A clause is omitted from a validation node. (6) Condition insertion fault (CIF): An extraneous validation node is inserted in to the code. (7) Condition omission fault (COF): A validation node is omitted from the code. For fault type (6), a trivial if condition was inserted right before an effect node such that it will not affect normal program execution. For fault type (7), we chose only to remove the validation node that has one branch leading to the exit node immediately after the checking failure; the node that prints system messages was also removed. For each system, we seeded 4 faults of each above-mentioned type into the source code, which are 28 faults in total. The faults were seeded into different functional units in the system under test such that no fault conceals another and all faults cause observable failures. The objective of the experiments is to uncover the faults seeded in the code through input validation testing using one of the following methods: (1) Equivalence class testing supplemented with boundary value analysis. (2) Cause-effect testing solely. (3) The proposed method. When using the proposed method, the prototype tool first computes the input validation partition and extracts the set of valid and invalid input conditions from code for each system under test. The testers are then required to use the computed information in equivalence class testing. No functional testing tools or test case generation tools were available in both experiments. Since equivalence class testing, boundary value analysis and cause-effect testing have been taught in the Software Engineering course in our school, we selected six students who have scored full marks in the quiz to participate in the experiments. All of them have no prior knowledge of the two systems under test; it is thus reasonable to assume that the six students are equally capable in conducting the experiments. The six students were equally divided into three groups. Each group was required to test the two systems using methods 1, 2 and 3, respectively. We had made sure that each group conducted the experiment independently from other groups. We also gave two-hour training to the students who were assigned with method 3 on the proposed method and the usage of the prototype tool. The students conducted the experiments in their spare time in our research laboratory. A summary of the results is shown in Table 1. The first column refers to the testing method used in the experiment. The column #Test cases gives the number of test cases in a test suite. The column #Faults gives the number of faults uncovered. The last column shows the program-level input validation coverage for each test suite. From the figures shown, we observe that in both experiments, the size of the test suite designed using the proposed method is
Table 1 Experiment results Method
#Test cases
#Faults
IVC (%)
(a) The results of experiment 1 1 169 2 344 3 152
16 23 27
60 77 79
(b) The results of experiment 2 1 276 2 450 3 215
17 21 28
65 78 82
the smallest, yet it manages to uncover almost all the faults seeded in the two systems. A test suite with higher input validation coverage tends to have better ability in detecting the faults seeded, though no test suite gives 100% input validation coverage due to the existence of infeasible paths. It is obvious, at least in this case study, that the fault detection capability of the proposed method is superior to that of the other two methods. A further examination of the types of faults uncovered is shown in Table 2. The figures clearly show the improvement of accuracy achieved by the proposed method in detecting input validation errors. Especially, the proposed method can be more effective in detecting predicate insertion fault (PIF) and condition insertion fault (CIF). Such faults in the code are usually difficult to find using blackbox testing methods as the program implements the behavior that is not in the specification. On the other hand, white-box testing methods may find faults with extraneous conditions inserted in the code but not the faults with conditions omitted from code. The proposed method uses the code information of input validation in functional testing, thus provides an effective testing that covers both the specification and the code characteristics of input validation. In the next, we summarize two more observations from this case study: (1) The recovered input conditions could be used to verify against the ones defined in the specification. This effectively aid to resolve discrepancies between the code and the specification before executing any test cases. The process may also help in detecting infeasible paths in a program through the identification of infeasible combination of input conditions recovered from code. Identification of such paths helps to achieve a more accurate coverage analysis. (2) Though there may not be a direct match between valid and invalid input conditions recovered from code with those in specification, the understanding of these conditions is not difficult for testers who have some fundamental programming skills as the conditions recovered from code are usually expressed in terms of variables that have clear one-to-one correspondence with input field names defined at the client programs or attribute names in database tables. In our
Table 2 Types of faults uncovered Method
BOF
ROF
VCF
PIF
POF
CIF
COF
(a) Faults uncovered in experiment 1 1 3 4 3 2 4 4 4 3 4 4 4
1 2 4
2 4 3
0 1 4
3 4 4
(b) Faults uncovered in experiment 2 1 2 4 4 2 4 3 4 3 4 4 4
0 2 4
3 4 4
0 0 4
4 4 4
552
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
experiments, the program slicing tool was only used for very few cases to study the variables involved for the understanding and verification of the input conditions.
path in each class needs to be exercised. This technique can effectively test input validation with less cost. To the best of our knowledge, currently, no such test coverage has been proposed yet. 6. Conclusion
5. Related work In our earlier work, we have proposed a code-based approach for testing input validation in web application [16,17]. The approach proposed in this paper is a major enhancement of the earlier work in which both the theory and method have been significantly improved. The most important contributions of this paper is that it introduces a property to infer valid and invalid input conditions from code and the idea of using the recovered input conditions to aid black-box input validation testing. In addition, we simplified and improved the definitions used in the theory, formalized and enhanced the input validation coverage measurement, and remove the redundancy and inefficiencies in earlier methods. In terms of domain of application, the earliest work related to input validation testing mostly focused on the automated generation of programs to test compilers [1]. Subsequently, Beizer [2] proposes a method for syntax testing that uses graph to specify user command and applies graph coverage techniques to manually generate test cases. Marick [18] also proposes an approach to syntax testing based on informal guidelines. However, these techniques cover only syntax that is just one part of input validation. Recently, two techniques that are more related to the proposed technique have been proposed for testing input validation in dataintensive systems [10,11,23]. The input validation testing (IVT) proposed by Hayes and Offutt [10,11] is a specification-based testing method that automatically analyzes user interface specification to generate test cases. Test cases for both valid and invalid inputs are generated. Motivated by the importance of security in Web applications, bypass testing was then proposed by Offutt et al. [23]. Bypass testing is a special kind of IVT to test the robustness and security of server software in Web applications. Through by passing client-side checking, test cases with invalid inputs are created to test the input validation feature of the server software. The proposed method shares the objective of testing input validation for data-intensive systems with IVT and bypass testing. However, both IVT and bypass testing are specification-based approaches. They generate test cases from the specification alone. Goodenough and Gerhart have pointed in their foundation paper for software testing [8] that neither specification-based nor code-based testing method can ensure the full correctness. The proposed method automatically recovers valid and invalid input conditions from code to aid black-box input validation testing. Consequently, the testing performed will also cover the code behavior on input validation. Therefore, the proposed method has the advantage of providing a more accurate and adequate testing for input validation. The proposed method extracts input validation partition and input conditions from code. The extracted information can be used as an input to existing test case generation techniques [4,7,9] to automatically generate test cases for testing. It can also be used in other testing methods such as GUI smoke testing [19,20] and bug locating [5]. In addition, we have proposed a coverage criterion to measure the adequacy of a test suite for testing input validation. Test coverage is an important measurement of the quality of a test suite. As input validation is critical to a large class of systems, it is important to verify that this feature is sufficiently tested before a system is deployed. In the proposed method, based on the structure of input validation implemented in a program, a set of paths under a particular path coverage criterion is partitioned into different classes. Hence, to test input validation adequately, only one
Input validation is critical to the control of the user inputs submitted to a system. In this paper, we have proposed a method to extract input validation partition and all the valid and invalid input conditions from program source code. Based on input validation partition, we have also developed a test coverage criterion to measure the adequacy of a test suite for testing input validation. In the proposed method, all the valid and invalid input conditions are automatically recovered from code to aid black-box testing. The recovered input conditions can be used to check against the specification for any inconsistencies to be revealed. They can aid the test case selection in black-box testing. They can also aid the design of white-box test cases that supplement the black-box testing. Hence, the resulting test suite covers both the specification and the code characteristics of the input validation. Our empirical evaluation has shown that the proposed method can effectively improve the accuracy and adequacy of the input validation testing. With the aid of the recovered input conditions, fewer test cases could be selected for adequate testing. Furthermore, the method can be more effective in detecting the extraneous behavior that implemented in the program but not described in the specification. Such faults are usually difficult to find using specification-based testing. Covering code behavior in functional testing has the advantage of achieving test accuracy with optimized testing effort. We have established such a method for testing input validation. Currently, we are exploring into this testing paradigm for other common software features. Acknowledgements We would like to thank Mary Jean Harrold from Georgia Institute of Technology for sharing the JABA program analysis tool. We also thank Abhik Roychoudhury and Tao Wang from Nantional University of Singapore for their help in the configuration of the JSlicer tool. Both tools help tremendously in building the prototype system. References [1] F. Bazzichi, I. Spadafora, An automatic generator for compiler testing, IEEE Transactions on Software Engineering SE-8 (4) (1982) 343–353. [2] B. Beizer, Software testing techniques, Van Nostrand Reinhold, 1990. p. xxvi, 550. [3] W.G. Bently, E.F. Miller, Ct coverage-initial results, Software Quality Journal 2 (1) (1993) 29–47. [4] P.M.S. Bueno, M. Jino, Automatic test data generation for program paths using genetic algorithms, International Journal of Software Engineering and Knowledge Engineering 12 (6) (2002) 691–709. [5] C. Cadar, et al., ‘‘EXE: Automatically generating inputs of death,” Association for Computing Machinery, New York, NY 10036-5701, United States, 2006, pp. 322–335. [6] E.Y.K. Chan, et al., ‘‘On the testing of particular input conditions, in: Proceedings of the 28th Annual International Computer Software and Applications Conference, IEEE Comput. Soc., 2004, pp. 318–325. [7] R. Ferguson, B. Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology 5 (1) (1996) 63– 86. [8] J.B. Goodenough, S.L. Gerhart, Toward a theory of test data selection, 1975, pp. 493–510. [9] N. Gupta, et al., Automated test data generation using an iterative relaxation method, in: Proceedings of the 6th International Symposium on the Foundations of Software Engineering (FSE-6), ACM, New York, NY, USA, 1998, pp. 231–244. [10] J.H. Hayes, A.J. Offutt, Increased software reliability through input validation analysis and testing, in: Proceedings of the 10th International
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
[11] [12] [13] [14] [15] [16]
[17]
[18]
Symposium on Software Reliability Engineering, Institute of Electrical and Electronics Engineers Computer Society, Los Alamitos, CA, USA, 1999, pp. 199–209. J.H. Hayes, J. Offutt, Input validation analysis and testing, Empirical Software Engineering 11 (4) (2006) 493–522. JABA, Java Architecture for Bytecode Analysis,, 2006. P. Jorgensen, Software testing: a craftman’s approach, CRC Press, 2002. p. 359. JSlice, Dynamic slicing tool for Java programs,
553
[19] A. Memon, et al., DART: a framework for regression testing nightly/daily builds of GUI applications, in: Proceedings International Conference on Software Maintenance, IEEE Comput. Soc., 2003, pp. 410–419. [20] A.M. Memon, Q. Xie, Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software, IEEE Transactions on Software Engineering 31 (10) (2005) 884–896. [21] MSDN, Design Guidelines for Secure Web Application,, 2008. [22] G.J. Myers et al., The art of software testing, John Wiley and Sons, 2004. p. xv, 234. [23] J. Offutt, et al., Bypass testing of Web applications, in: Proceedings of the 15th International Symposium on Software Reliability Engineering, IEEE Comput. Soc., 2004, pp. 187–197. [24] S. Sinha et al., Interprocedural control dependence, ACM Transactions on Software Engineering and Methodology 10 (2) (2001) 209–254. [25] Sourceforge, Open source website, . [26] A.H. Watson, T.J. McCabe, Structured testing: A software testing methodology using the cyclomatic complexity metric, NIST-SP-500-235, National Institute of Standards and Technology, 1996.
Contents lists available at ScienceDirect
Information and Software Technology journal homepage: www.elsevier.com/locate/infsof
Covering code behavior on input validation in functional testing Hui Liu *, Hee Beng Kuan Tan School of Electrical and Electronic Engineering, Block S2, Nanyang Technological University, Nanyang Avenue, Singapore 639798, Singapore
a r t i c l e
i n f o
Article history: Received 14 September 2007 Received in revised form 20 February 2008 Accepted 11 July 2008 Available online 24 August 2008 Index terms: Software testing Black-box testing White-box testing Input validation Test coverage
a b s t r a c t Input validation is the enforcement built in software systems to ensure that only valid input is accepted to raise external effects. It is essential and very important to a large class of systems and usually forms a major part of a data-intensive system. Most existing methods for input validation testing are specification-based. However, to test input validation more accurately, a code-based method is also required. In this paper, we propose an approach to extract path partition and input conditions from code for testing input validation. The path partition can be used to design white-box test cases for testing input validation. It can also be used to measure the coverage of input validation testing. The valid and invalid input conditions recovered can be used to check against the specifications and aid the test suite design in blackbox testing. We have also evaluated the proposed method through experimental study. Ó 2008 Elsevier B.V. All rights reserved.
1. Introduction Software testing techniques can be broadly classified into white-box testing and black-box testing. The former refers to the structural testing technique that designs test cases based on the information derived from source code. The latter refers to the functional testing technique that designs test cases based on the information from the specification. White-box and black-box testing are considered complementary to each other. Many researchers emphasize that, to test software more accurately, it is required to cover both specification and code behaviors [13]. A large class of systems process inputs submitted from users to raise external effects. Information systems and database applications belong to this class. These systems enforce that only valid inputs are accepted to raise external effects. Invalid inputs submitted are rejected and no external effects are raised. The enforcement in the systems is called input validation. For example, in a banking system, the amount withdrawn from a customer account submitted in any withdrawal transaction must be less than the available balance of the account. As such, input validation is incorporated into the system to enforce that only such valid withdrawal amount is accepted. Input validation plays a key role in the controlling user inputs submitted to a system. It plays a vital role on the robustness of the system. It is also an importance method for defending against today’s application attacks [21]. The design and implementation of input validation are usually carried out by a large population of * Corresponding author. Tel.: +65 67906528; fax: +65 67922971. E-mail addresses: [email protected] (H. Liu), [email protected] (H.B. Kuan Tan). 0950-5849/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.infsof.2008.07.001
application developers but not a small highly specialized software expert group. Therefore, to have an effective method for testing input validation is important for software quality assurance. We propose in this paper a method to recover implementation information from code to aid input validation testing. We propose a method to extract path partition and to recover valid and invalid input conditions from code. The path partition can be used in test case generation for testing input validation. It can also be used to measure the coverage of input validation testing. The recovered input conditions can be used to check against the specification and aid the test suite design in black-box testing. We have evaluated the proposed method through experimental study. The approach applies empirical software engineering, reverse engineering and program analysis together in software testing. The paper is organized as follows. Section 2 presents the empirical theory for the recovery of input validation information from code, and proposes a test coverage criterion to measure the adequacy of input validation testing. Section 3 discusses the proposed method and Section 4 reports our evaluation of the proposed method. Section 5 compares our work with related work. Section 6 concludes the paper.
2. An empirical theory for the recovery of code behavior on input validation We shall first review some terms that will be used throughout the paper. As defined by Sinha et al. [24], the control flow graph (CFG) of a program P is a directed graph G = (N, E), in which N contains a set of nodes and E = {(n, m) j n, m 2 N} contains edges that connecting the nodes. Each node in G represents a statement in
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
P, and each edge in the G represents possible flow of control between two statements in P. There is an entry node and an exit node in a CFG representing entry to and exit from P, respectively. A predicate node in G has two edges originate from it. The statement at a predicate node will be evaluated during the program execution to determine which out-going edge of the predicate node will be traversed. An out-going edge of a predicate node is called a branch and a branch predicate describes the conditions under which the branch will be traversed. In a control flow graph, the data dependence represents the data flow between program statements. A node y is data dependent on a node x if and only if there exists a variable v such that x defines v, y uses v and there is a path from x to y along with which v is not redefined. The control dependence is defined between conditional statement and the statements whose executions are controlled by the conditional statement. A node y dominates a node x if and only if every path from the entry node to x contains y. A node y postdominates a node x if and only if every path from x to the exit node contains y. A node y is control dependent on a node x if and only if x has successors x0 and x00 such that y postdominates x0 but y does not postdominate x00 . The transitive property of data dependence is inherent. We say a node y is transitively data dependent on a node x if there exists a sequence of nodes (w0 = x, w1,w2,. . .,wn = y) in the control flow graph such that n P 1 and wj is data dependent on wj1 for all j, 1 6 j 6 n. Likewise, the transitive control dependence can be defined based on a sequence of control dependence edges. A node y is transitively control dependent on node x if there exists a sequence of nodes (w0 = x, w1,w2,. . .,wn = y) in the control flow graph such that n P 1 and wj is control dependent on wj1 for all j, 1 6 j 6 n. In the CFG of a program, a node is called an effect node if it raises external effect. For example, a node that updates database is an effect node. A node u is called an input node if u reads an input directly from the external environment or an input received from the external environment is accessible at u and all other nodes at which the input is also accessible are dominated by u. In either case, the input is called the input received from u. For example, a node that reads a string directly from console is an input node of the first type. In a web application, a server program is activated upon the submission of an input through a form displayed. In this case, the input is usually comprised of many data items, and the first node of the server program is an input node of the second type. Through the empirical study, we observed that a certain type of node in a control flow graph of a program is critical to the implementation of input validation in a program. Such nodes are formally defined next. Let u be an input node of a program. A predicate node d is a validation node of u if the following two conditions hold: (1) d is transitively data dependent on a node that references to inputs received from u. (2) There is a path from d to an effect node that is transitively control dependent on d, and there is also a path from d to an exit node that does not contain any effect node. The two conditions emphasize that a validation node should check the validity of inputs submitted by users and decide whether the inputs should be used to raise any external effect. In addition, we say that an effect node f is influenced by an input node u if and only if f is transitively control dependent on a validation node of u, or f is transitively data dependent on a node that references to inputs received from u. For example, Fig. 1 shows a simplified program for cash withdrawal from an automated teller machine (ATM). In the control
547
flow graph of the program, node 1, 4 and 7 are three input nodes and node 12 is the only effect node. It is easy to identify that node 2 is the only validation node of node 1, node 5 is the only validation node of node 4, and node 8, 10 are validation nodes of node 7. Cleary, the effect node 12 is influenced by all three input nodes in the program. 2.1. Input validation partition As the code behavior on input validation is path-based and theoretically the number of paths through a CFG is infinite, practically, we need to base on a set of paths that satisfies a path coverage criterion for code-based input validation testing. Let J be a set of input nodes in the CFG of a program. Let S be a set of paths through the CFG that satisfies a path coverage criterion p. We then partition the paths in S that contain some validation nodes of the nodes in J in such a way that paths are put into the same class if they satisfy the following two conditions: (1) They contain the same set of validation nodes of any input node in J. (2) They follow the same branch at each of the above-mentioned validation nodes. Hence, the set of validation nodes and the branch traversed at each of these nodes form a partition criterion. The resulting partition is called an input validation partition of J under p. If J has only one input node u, we call the partition the input validation partition of u under p. For example, in the CFG shown in Fig. 1, the set of paths S that satisfies the basis path coverage criterion [26] is {(entry, 1, 2, 3, end), (entry, 1, 2, 4, 5, 6, end), (entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}. The set of input nodes J include node 1, 4, and 7. It is not difficult to verify that the input validation partition of J under the basis path coverage has five classes. Each class has exactly one path in S. Given a path coverage criterion p, we can measure the adequacy of a test suite for testing input validation based on the code characteristics. Let X be an input validation partition of a set of input nodes J under p. Clearly, a test suite is adequate for testing the input validation implemented in a program with regards to J and p if at least one path in each class in X is exercised. Hence, we can compute the input validation coverage of a test suite T as the percentage of the classes in X from which at least one path is exercised. That is:
IVCðT; PÞ ¼
CEðT; PÞ 100% CðT; PÞ
where C(T,P) is the total number of classes in X and CE(T,P) is the number of classes in X from which at least a path is exercised by a test case in T. 2.2. Recovery of valid and invalid input conditions In majority of cases, an input received from an input node is validated by checking it against some conditions using predicate nodes before any effect is raised. A condition that must be satisfied before the use of the input to raise effect is a valid condition of the input. Conversely, a condition to deny the use of the input to raise effect is an invalid condition of the input. Let u be an input node in the CFG of a program. Let Xu be the input validation partition of u under the Ct coverage with k = 0. The Ct coverage criterion is introduced by Bently and Miller [3] that defines a minimum iteration count k, so that a manageable set of paths can be obtained. For the case of k = 0, the Ct coverage
548
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
begin 1 read card number; 2 if (card number is not valid) then 3 show “card not valid” error; else 4 read PIN from user; 5 if (PIN does match card PIN) then 6 show “invalid PIN” error; else 7 read withdrawal amount M 8 if (M > withdrawal limit) then 9 show “limit exceeded” error; else 10 if (M > account balance) then 11 show “balance exceeded” error; else 12 debit the amount M from user account; endif endif endif endif end
(a) The pseudocode for the program
Fig. 1. A simplified ATM cash withdrawal program.
identifies all the non-iterative paths and iterative paths that do not repeat any loop. Clearly, a class K in the partition Xu will fall under one of the following two types: (1) If there is a path in K that contains an effect node influenced by u, all the paths in K accept the input received from u. (2) Otherwise, all the paths in K reject the input received from u. We call the first and the second type of classes a valid class and an invalid class, respectively of u. From the definition of input validation partition, the conjunction of branch predicates of the branches at all the validation nodes of u that any path in K follows is identical. We denote it as CK and call it the input condition of class K. Clearly, the checking of the input received from an input node u against a condition to decide whether an effect should be raised is carried out by the validation nodes of u. This directly implies that the input condition of a valid class is a valid condition of the input received from u. The input condition of an invalid class is an invalid condition of the input received from u. Next, we shall summarize this invariant property in Property 1.
Property 1. Let Xu be an input validation partition of an input node u under Ct k = 0 coverage criterion. Then, VCu = {the input condition of KjK 2 Xu and K is a valid class of u} and ICu = {the input condition of KjK 2 Xu and K is an invalid class of u} are the sets of valid and invalid input conditions, respectively of the input received from n, that are implemented in the program. For the program shown in Fig. 1, as discussed earlier, node 1, 4 and 7 are input nodes. Under the Ct k = 0 coverage criterion, the input validation partition of node 1 has two classes: {(entry, 1, 2, 3, end)} and {(entry, 1, 2, 4, 5, 6, end), (entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}, the input validation partition of node 2 also has two classes: {(entry, 1, 2, 4, 5, 6, end)} and {(entry, 1, 2, 4, 5, 7, 8, 9, end), (entry, 1, 2, 4, 5, 7, 8, 10, 11, end), (entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}, and the input validation partition of node 7 has three classes: {(entry, 1, 2, 4, 5, 7, 8, 9, end)}, {(entry, 1, 2, 4, 5, 7, 8, 10, 11, end)}, and {(entry, 1, 2, 4, 5, 7, 8, 10, 12, end)}. Since node 12 is the only effect node in the program, it is not difficult to identify the invalid classes and valid classes of each input node. For example, in the input validation partition of node 7, the first two classes are invalid classes of node 7, whereas the last class is a valid class of node 7. Since node 8 and
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
9 are validation nodes of node 7, valid and invalid conditions of node 7 can be derived by computing the conjunction of the branch predicates at the two nodes along a path in each class in the partition. Therefore, {(M > withdrawal limit), NOT(M > withdrawal limit) AND (M > account balance)} is the set of invalid conditions of the amount M received from node 7, and {NOT(M > withdrawal limit) AND NOT (M > account balance)} is the set of valid conditions for the amount M received from node 7. As input validation is a major feature in a system, the valid and invalid conditions of input should be obtainable from some form of system documentation. Therefore, the invalid and invalid input conditions extracted from code can be used to check against the corresponding conditions defined in the specification for any discrepancy to be revealed. In addition, the extracted input conditions provide the testers with true implementation information that supplements the specification, and hence can be used to aid the test suite design in black-box testing. 3. The proposed method Based on the discussion in Section 2, we propose a method to testing method that covers the code behavior on input validation in functional testing. The method comprises of two stages: (1) Compute valid and invalid input conditions from program source code; (2) Perform functional testing with the aid of the information computed in the first stage. 3.1. Stage 1 – Compute valid and invalid input conditions In this stage, all the valid and invalid input conditions for each input received in a program are automatically recovered from program source code. Fig. 2 shows the algorithm that comprises two steps. In Step 1, the CFG of a program is constructed first. Usually, source code specific tools can be used to perform static analysis, from which the control dependency, data dependency and other program characteristics can be identified. Next, the set of input nodes and effect nodes need to be identified. An effect node represents an output state or an action that is critical and major to a
program. For example, in database applications, the statements that perform database transactions or write to a file are obviously more critical to the ones that output warning messages. The identification of effect nodes would not be difficult to domain experts or developers who understand the program well. Hence, once the types of effects in a program have been decided, the effect nodes can be automatically identified by program analysis tools through standard programming interfaces. The input nodes can then be identified in a similar way. For programs that have multiple input nodes and effect nodes, it is also essential to identify and understand the complex relationships between those input nodes and effect nodes before the input conditions can be recovered. An input node usually influences more than one effect node and an effect node could be influenced by more than one input nodes. Such information can be obtained through data dependence and control dependence analysis or with the use of program slicing. In Step 2, the set of valid and invalid input conditions of each input received in the program are computed. To achieve that, three sub-steps are performed. First, a set S of paths is computed under the Ct coverage with k = 0 [3]. Next, for each input node u in the program, the input validation partition X u of u is computed as follows: for each path in S, if it contains at least one validation node of u, the sequence of validation nodes of u contained in the path and the branch traversed at each of these nodes are identified as a partition criterion. This criterion forms a class in the partition and is included in Xu if it has not been included yet. The path is then put into the class in Xu according to the partition criterion. If the path contains an effect node influenced by u, the corresponding class is classified as a valid class of u; otherwise, the corresponding class is classified as an invalid class of u. Finally, based on Property 1, the input condition of each class in Xu is computed and hence, the sets of valid and invalid conditions of the input received from u, respectively can be computed. 3.2. Stage 2 – Perform functional testing Equivalence class testing and cause-effect testing are two fundamental functional testing techniques [18,22]. Equivalence class
Comp_IV_Info (program P) Output: For each input node u, an input validation partition Ωu of u, and the sets VCu and ICu of valid and invalid conditions of input received from u respectively. Step 1.
549
Construct the CFG of P and identify the set U of input nodes and the set F of effect nodes. Next, compute the following information: 1) For each input node u in U, the set of effect nodes F(u) that are influenced by u. 2) For each input node u in U, the set of validation nodes D(u) of u.
Step 2.
Compute the valid and invalid conditions of input received from each input node u in U. 2.1 Compute a set S of paths under the Ct coverage with k=0. 2.2 Compute the input validation partition Ωu of u: For each path p in S, a) Exclude p from S if it does not contain any validation node of u. b) Identify the partition criterion Q that consists of the sequence of validation nodes of u in p and the branch traversed at each of these nodes. c) Include Q in set Ωu if it has not been included yet. d) Add p into the class of paths that links to the criterion Q. e) If p contains an effect node in F(u), Q is marked as a valid class of u; otherwise, Q is marked as an invalid class of u. 2.3 Compute the set VCu of valid conditions and the set ICu of invalid conditions of u: a) Extract the input conditions of all the valid class of u and include them in VCu. b) Extract the input conditions of all the invalid class of u and include them in ICu. Fig. 2. An algorithm to compute input validation information.
550
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
testing partitions the input space into equivalence classes according to the input conditions. Test cases are designed by selecting at least one condition from each equivalence class. Boundary value analysis complements the equivalence partitioning by selecting test cases at the edges of a class. Cause-effect testing uses cause-effect graph and decision table as a tool to design testing cases, in which input conditions form the causes and actions form the effects. A test case is then designed for each rule in the decision table. In principle, to derive a comprehensive test suite for testing input validation from black-box testing methods, all possible input conditions would be considered. However, for a software system with ordinary complexity, the number of input conditions can be very large; hence, such approach would be too expensive to be used with limited testing resources. In such cases, software testers often need to use their intuition and professional experiences to select effective test cases [6]. However, this approach relies on chance to determine the effectiveness of the resulting test suite as the assumption could fail for the specific implementation under test. Furthermore, even with adequate test cases, black-box testing alone may not reveal all faults in a system when the system implements the behavior that is not in the specification. Clearly, we can use the valid and invalid input conditions recovered from stage 1 to aid the testers in black-box input validation testing. Firstly, the valid and invalid input conditions can be used to check against the specifications for any discrepancies to be revealed. Secondly, the recovered input conditions provide the testers the implementation information of a program, and thus aid the design of test cases. Based on the implementation information, the testers can avoid the selection of unnecessary combinations of input conditions that are not implemented in the program. Consequently, fewer test cases could be selected for adequate testing. Furthermore, based on the input validation partition, white-box test cases can be designed to supplement the black-box test cases. The input validation coverage metric can also be used to measure the adequacy of the test suite designed.
Program Source Code
PathTracer Kaffe JVM J Trace
Program Analyzer Preprocessor
Test case execution Execution Traces
JABA CFG IV Miner IV Extractor
IV Partitions IV Coverage Analyzer
Program Slicer
Program Slices
Validation Conditions
Coverage Report IV: Input Validation
Fig. 3. The architecture of the prototype tool.
It is used to study the variables involved in these conditions for reconciling them against the specification. Path Tracer contains a JTrace which is built on top of the Kaffe Java virtual machine [15]. The JTrace uses the components in the open source tool JSlice [14] and is able to output the execution trace in terms of source line numbers and the corresponding class names. The execution trace is then input to IV coverage analyzer for the computation of the input validation coverage. 4.2. Experimental study
4. Evaluation We have conducted experimental study to access the feasibility and usefulness of the proposed method. To aid the experiment, we have developed a prototype tool based on the Java Architecture for Bytecode Analysis (JABA) from Georgia Institute of Technology [12]. 4.1. Prototype tool implementation The prototype tool is targeted for web database applications written in Java. As shown in Fig. 3, it consists of four components: a program analyzer, an input validation miner (IV miner), a path tracer and an input validation coverage analyzer (IV coverage analyzer). Program analyzer uses JABA’s API to analyze Java programs. By parsing a class file of a Java program, JABA performs control and data flow analysis and builds the control flow graph of the program. The Preprocessor is to prepare all the files in a system into class files for inputting to JABA to build the control flow graph of a program. This component implements Step 1 in the algorithm shown in Fig. 2. IV Miner includes two modules: an input validation extractor (IV extractor) and a program slicer. IV extractor implements Step 2 in the algorithm shown in Fig. 2. For each program that processes inputs submitted from user to raise external effect, for each input node in the program, IV Extractor generates a report containing the sets of valid and invalid conditions of inputs received from the input node. Program Slicer is a utility for computing program slices.
We have conducted two controlled experiments in which various types of faults related to the input validation were seeded into well-tested systems. From our analysis of functional testing techniques and further discussion with the software engineers in the industry, we come to the conclusion that the most frequently used methods to test input validation are equivalence class testing supplemented with boundary value analysis. Cause-effect testing is a testing method that explores all the combinations of input conditions; in practice, however, the complexity of the method and the large test suite it requires make it is difficult to be applied for most projects. Nevertheless, for the purpose of more thorough evaluation, in the experiments, we compare the proposed method with both methods. The first experiment was performed on an open source system called Smacs downloaded from Sourceforge [25]. Smacs is a webbased application for the management of the working of temporary staff in an organization. The size of the source code is 13462 LOC. The second experiment was performed on a system called JAuction that developed from an MSc Software Engineering project. JAuction provides the functionalities for hosting online auctions where registered users can both post and bid on items. The system was developed by following the Rational Unified Process and the size of the source code is 18342 LOC. Before the commencement of each experiment, we have prepared for both systems the software design specifications. In addition, each system has been thoroughly tested to ensure that the system behaves in accord with its specification. Besides the system used, the design of the two experiments is the same.
551
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
We then seeded into each system the faults that are commonly made in the implementation of input validation. We introduced the following seven types of faults: (1) Boolean operator fault (BOF): The changes of AND, OR and NOT operators in a validation node. (2) Relational operator fault (ROF): The changes of less than, greater than, less or equal to and greater or equal to operators in a validation node. (3) Variable and constant fault (VCF): A variable or constant in a validation node is replaced by another variable or constant. (4) Predicate insertion fault (PIF): An extraneous clause is inserted into a validation node. (5) Predicate omission fault (POF): A clause is omitted from a validation node. (6) Condition insertion fault (CIF): An extraneous validation node is inserted in to the code. (7) Condition omission fault (COF): A validation node is omitted from the code. For fault type (6), a trivial if condition was inserted right before an effect node such that it will not affect normal program execution. For fault type (7), we chose only to remove the validation node that has one branch leading to the exit node immediately after the checking failure; the node that prints system messages was also removed. For each system, we seeded 4 faults of each above-mentioned type into the source code, which are 28 faults in total. The faults were seeded into different functional units in the system under test such that no fault conceals another and all faults cause observable failures. The objective of the experiments is to uncover the faults seeded in the code through input validation testing using one of the following methods: (1) Equivalence class testing supplemented with boundary value analysis. (2) Cause-effect testing solely. (3) The proposed method. When using the proposed method, the prototype tool first computes the input validation partition and extracts the set of valid and invalid input conditions from code for each system under test. The testers are then required to use the computed information in equivalence class testing. No functional testing tools or test case generation tools were available in both experiments. Since equivalence class testing, boundary value analysis and cause-effect testing have been taught in the Software Engineering course in our school, we selected six students who have scored full marks in the quiz to participate in the experiments. All of them have no prior knowledge of the two systems under test; it is thus reasonable to assume that the six students are equally capable in conducting the experiments. The six students were equally divided into three groups. Each group was required to test the two systems using methods 1, 2 and 3, respectively. We had made sure that each group conducted the experiment independently from other groups. We also gave two-hour training to the students who were assigned with method 3 on the proposed method and the usage of the prototype tool. The students conducted the experiments in their spare time in our research laboratory. A summary of the results is shown in Table 1. The first column refers to the testing method used in the experiment. The column #Test cases gives the number of test cases in a test suite. The column #Faults gives the number of faults uncovered. The last column shows the program-level input validation coverage for each test suite. From the figures shown, we observe that in both experiments, the size of the test suite designed using the proposed method is
Table 1 Experiment results Method
#Test cases
#Faults
IVC (%)
(a) The results of experiment 1 1 169 2 344 3 152
16 23 27
60 77 79
(b) The results of experiment 2 1 276 2 450 3 215
17 21 28
65 78 82
the smallest, yet it manages to uncover almost all the faults seeded in the two systems. A test suite with higher input validation coverage tends to have better ability in detecting the faults seeded, though no test suite gives 100% input validation coverage due to the existence of infeasible paths. It is obvious, at least in this case study, that the fault detection capability of the proposed method is superior to that of the other two methods. A further examination of the types of faults uncovered is shown in Table 2. The figures clearly show the improvement of accuracy achieved by the proposed method in detecting input validation errors. Especially, the proposed method can be more effective in detecting predicate insertion fault (PIF) and condition insertion fault (CIF). Such faults in the code are usually difficult to find using blackbox testing methods as the program implements the behavior that is not in the specification. On the other hand, white-box testing methods may find faults with extraneous conditions inserted in the code but not the faults with conditions omitted from code. The proposed method uses the code information of input validation in functional testing, thus provides an effective testing that covers both the specification and the code characteristics of input validation. In the next, we summarize two more observations from this case study: (1) The recovered input conditions could be used to verify against the ones defined in the specification. This effectively aid to resolve discrepancies between the code and the specification before executing any test cases. The process may also help in detecting infeasible paths in a program through the identification of infeasible combination of input conditions recovered from code. Identification of such paths helps to achieve a more accurate coverage analysis. (2) Though there may not be a direct match between valid and invalid input conditions recovered from code with those in specification, the understanding of these conditions is not difficult for testers who have some fundamental programming skills as the conditions recovered from code are usually expressed in terms of variables that have clear one-to-one correspondence with input field names defined at the client programs or attribute names in database tables. In our
Table 2 Types of faults uncovered Method
BOF
ROF
VCF
PIF
POF
CIF
COF
(a) Faults uncovered in experiment 1 1 3 4 3 2 4 4 4 3 4 4 4
1 2 4
2 4 3
0 1 4
3 4 4
(b) Faults uncovered in experiment 2 1 2 4 4 2 4 3 4 3 4 4 4
0 2 4
3 4 4
0 0 4
4 4 4
552
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
experiments, the program slicing tool was only used for very few cases to study the variables involved for the understanding and verification of the input conditions.
path in each class needs to be exercised. This technique can effectively test input validation with less cost. To the best of our knowledge, currently, no such test coverage has been proposed yet. 6. Conclusion
5. Related work In our earlier work, we have proposed a code-based approach for testing input validation in web application [16,17]. The approach proposed in this paper is a major enhancement of the earlier work in which both the theory and method have been significantly improved. The most important contributions of this paper is that it introduces a property to infer valid and invalid input conditions from code and the idea of using the recovered input conditions to aid black-box input validation testing. In addition, we simplified and improved the definitions used in the theory, formalized and enhanced the input validation coverage measurement, and remove the redundancy and inefficiencies in earlier methods. In terms of domain of application, the earliest work related to input validation testing mostly focused on the automated generation of programs to test compilers [1]. Subsequently, Beizer [2] proposes a method for syntax testing that uses graph to specify user command and applies graph coverage techniques to manually generate test cases. Marick [18] also proposes an approach to syntax testing based on informal guidelines. However, these techniques cover only syntax that is just one part of input validation. Recently, two techniques that are more related to the proposed technique have been proposed for testing input validation in dataintensive systems [10,11,23]. The input validation testing (IVT) proposed by Hayes and Offutt [10,11] is a specification-based testing method that automatically analyzes user interface specification to generate test cases. Test cases for both valid and invalid inputs are generated. Motivated by the importance of security in Web applications, bypass testing was then proposed by Offutt et al. [23]. Bypass testing is a special kind of IVT to test the robustness and security of server software in Web applications. Through by passing client-side checking, test cases with invalid inputs are created to test the input validation feature of the server software. The proposed method shares the objective of testing input validation for data-intensive systems with IVT and bypass testing. However, both IVT and bypass testing are specification-based approaches. They generate test cases from the specification alone. Goodenough and Gerhart have pointed in their foundation paper for software testing [8] that neither specification-based nor code-based testing method can ensure the full correctness. The proposed method automatically recovers valid and invalid input conditions from code to aid black-box input validation testing. Consequently, the testing performed will also cover the code behavior on input validation. Therefore, the proposed method has the advantage of providing a more accurate and adequate testing for input validation. The proposed method extracts input validation partition and input conditions from code. The extracted information can be used as an input to existing test case generation techniques [4,7,9] to automatically generate test cases for testing. It can also be used in other testing methods such as GUI smoke testing [19,20] and bug locating [5]. In addition, we have proposed a coverage criterion to measure the adequacy of a test suite for testing input validation. Test coverage is an important measurement of the quality of a test suite. As input validation is critical to a large class of systems, it is important to verify that this feature is sufficiently tested before a system is deployed. In the proposed method, based on the structure of input validation implemented in a program, a set of paths under a particular path coverage criterion is partitioned into different classes. Hence, to test input validation adequately, only one
Input validation is critical to the control of the user inputs submitted to a system. In this paper, we have proposed a method to extract input validation partition and all the valid and invalid input conditions from program source code. Based on input validation partition, we have also developed a test coverage criterion to measure the adequacy of a test suite for testing input validation. In the proposed method, all the valid and invalid input conditions are automatically recovered from code to aid black-box testing. The recovered input conditions can be used to check against the specification for any inconsistencies to be revealed. They can aid the test case selection in black-box testing. They can also aid the design of white-box test cases that supplement the black-box testing. Hence, the resulting test suite covers both the specification and the code characteristics of the input validation. Our empirical evaluation has shown that the proposed method can effectively improve the accuracy and adequacy of the input validation testing. With the aid of the recovered input conditions, fewer test cases could be selected for adequate testing. Furthermore, the method can be more effective in detecting the extraneous behavior that implemented in the program but not described in the specification. Such faults are usually difficult to find using specification-based testing. Covering code behavior in functional testing has the advantage of achieving test accuracy with optimized testing effort. We have established such a method for testing input validation. Currently, we are exploring into this testing paradigm for other common software features. Acknowledgements We would like to thank Mary Jean Harrold from Georgia Institute of Technology for sharing the JABA program analysis tool. We also thank Abhik Roychoudhury and Tao Wang from Nantional University of Singapore for their help in the configuration of the JSlicer tool. Both tools help tremendously in building the prototype system. References [1] F. Bazzichi, I. Spadafora, An automatic generator for compiler testing, IEEE Transactions on Software Engineering SE-8 (4) (1982) 343–353. [2] B. Beizer, Software testing techniques, Van Nostrand Reinhold, 1990. p. xxvi, 550. [3] W.G. Bently, E.F. Miller, Ct coverage-initial results, Software Quality Journal 2 (1) (1993) 29–47. [4] P.M.S. Bueno, M. Jino, Automatic test data generation for program paths using genetic algorithms, International Journal of Software Engineering and Knowledge Engineering 12 (6) (2002) 691–709. [5] C. Cadar, et al., ‘‘EXE: Automatically generating inputs of death,” Association for Computing Machinery, New York, NY 10036-5701, United States, 2006, pp. 322–335. [6] E.Y.K. Chan, et al., ‘‘On the testing of particular input conditions, in: Proceedings of the 28th Annual International Computer Software and Applications Conference, IEEE Comput. Soc., 2004, pp. 318–325. [7] R. Ferguson, B. Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology 5 (1) (1996) 63– 86. [8] J.B. Goodenough, S.L. Gerhart, Toward a theory of test data selection, 1975, pp. 493–510. [9] N. Gupta, et al., Automated test data generation using an iterative relaxation method, in: Proceedings of the 6th International Symposium on the Foundations of Software Engineering (FSE-6), ACM, New York, NY, USA, 1998, pp. 231–244. [10] J.H. Hayes, A.J. Offutt, Increased software reliability through input validation analysis and testing, in: Proceedings of the 10th International
H. Liu, H.B. Kuan Tan / Information and Software Technology 51 (2009) 546–553
[11] [12] [13] [14] [15] [16]
[17]
[18]
Symposium on Software Reliability Engineering, Institute of Electrical and Electronics Engineers Computer Society, Los Alamitos, CA, USA, 1999, pp. 199–209. J.H. Hayes, J. Offutt, Input validation analysis and testing, Empirical Software Engineering 11 (4) (2006) 493–522. JABA, Java Architecture for Bytecode Analysis,
553
[19] A. Memon, et al., DART: a framework for regression testing nightly/daily builds of GUI applications, in: Proceedings International Conference on Software Maintenance, IEEE Comput. Soc., 2003, pp. 410–419. [20] A.M. Memon, Q. Xie, Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software, IEEE Transactions on Software Engineering 31 (10) (2005) 884–896. [21] MSDN, Design Guidelines for Secure Web Application,