- Email: [email protected]
On the effectiveness of classification trees for test case construction1
INFSOF 3965
Information and Software Technology 40 (1998) 765–775
On the effectiveness of classification trees for test case construction 夽 T.Y. Chen*, P.L. Poon Department of Computer Science, University of Melbourne, Parkville 3052, Australia Received 21 April 1998; accepted 26 August 1998
Abstract The notion of the classification-hierarchy table and the classification-tree construction algorithm provide a systematic approach to the construction of classification trees from given sets of classifications and their associated classes. Using classification trees, the set of all possible test cases can be constructed from functional specifications. This paper extends their study by introducing a metric to measure the effectiveness of a classification tree with respect to the construction of test cases, and providing ways to improve this effectiveness. 䉷 1998 Elsevier Science B.V. All rights reserved. Keywords: Classification-hierarchy table; Classification-tree method; Software testing; Specification-based testing; Test case selection
1. Introduction Various people [1–3] have argued that the expense of software testing is often underestimated although it may account for up to 50% of the project cost. Because testing is an expensive process, its effectiveness should be improved by means of systematic planning, execution and monitoring. As it is well-known that the construction of the set of all possible test cases S, affects the comprehensiveness and, hence, the quality of the test [4–6], numerous researchers have developed their own methods for this construction process. Two approaches to this construction are the black box and the white box testing. For the former approach, most techniques are developed for formal specifications only [7– 9], despite the fact that many real-life specifications are written in an informal way. On the other hand, the few techniques developed for both formal and informal specifications include the classification-tree method [5], the category-partition method [6] and the decision-table method [10]. The classification-tree method is an extension of the category-partition method. Also, the decision-table method can be regarded as a degenerated form of the classification-tree method. With respect to the generation of S, the classification-tree method is Corresponding author. Tel.: 61 3 9287 9101; fax: 61 3 9348 1184; e-mail: [email protected] 夽 This project was partially supported by a research grant from the Australian Research Council.
more effective. Readers may refer to [11] for a detailed comparison between these methods. In general, the classification-tree method developed by Grochtmann and Grimm [5], helps construct S from functional specifications (hereafter referred to as the ‘specifications’), via the construction of classification trees. However, their tree construction method is rather ad hoc, resulting in the variation of classification trees constructed from the same specification according to the tester’s own expertise and experience. This problem is subsequently solved by Chen and Poon [12,13] using the notion of classification-hierarchy table (which captures the hierarchical relation for each pair of distinct classifications), and a method for the construction of classification trees from given sets of classifications and their associated classes. However, their tree construction methodology does not take into account the effectiveness of the classification trees, with respect to the construction of test cases. This paper attempts to solve the above problem by: (i) defining a metric to measure the effectiveness of a classification tree; (ii) identifying some relationships between a classification tree and this effectiveness metric; and (iii) providing ways to improve this effectiveness metric of a classification tree. The rest of this paper is structured as follows. Section 2 gives an overview of the classification-tree method. Section 3 introduces an effectiveness metric of a classification tree and its underlying rationale. Section 4 describes some structures of a classification tree which could lead to a poor effectiveness metric. Section 5 presents two restructuring
0950-5849/98/$ - see front matter 䉷 1998 Elsevier Science B.V. All rights reserved. PII: S0950-584 9(98)00107-4
766
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Fig. 1. TPGM and part of its combination table.
algorithms for a classification tree so as to improve its effectiveness metric. Finally, Section 6 concludes the paper.
2. An overview of the classification-tree method A classification tree, T, is a graphical representation of the hierarchical relation between various classifications. The following describes the major steps of the classification-tree method: 1. From the specification, identify all classifications (defined as the different criteria for partitioning the input domain of the program to be tested) and their associated classes (defined as the disjoint subsets of values for each classification). 2. From the sets of classifications and their associated classes, construct T. 3. From T, construct the corresponding combination table. 4. From the combination table, identify all the feasible combinations of classes. Each combination of classes represents one potential test case. Example 1 illustrates this method.
Example 1. Suppose we have program PGM, with its specification as shown below, for testing: 1. PGM has six input variables, namely, A, B, C, D, E and F. 2. Each variable A and C has three possible values (e.g. a1, a2 and a3 for A), whereas each remaining variable has two possible values (e.g. b1 and b2 for B). 3. The input domain of PGM may contain any combination of possible values from some of these variables except the following:
i. (A is a1 or a3) and (E is e1 or e2). ii. (A is a2 or a3) and (D is d1 or d2). iii. (A is a2) and (C is c3). iv. (A is a3) and (B is b2). v. (B is b1) and (F is f1 or f2). vi. (C is c1 or c2) and (E is e1 or e2). vii. (C is c2 or c3) and (D is d1 or d2). viii. (D is d1 or d2) and (E is e1 or e2). Suppose all of the classifications and classes for PGM are simply defined as the input variables and their possible values, respectively. Thus, for example, A is taken as a classification with a1, a2 and a3 as its three associated classes (throughout this paper, classifications and classes are denoted by capital and small letters, respectively). Obviously, with these classifications and their associated classes, one approach to the construction of SPGM (which denotes S for PGM) is to select and combine one class of each classification, so that each combination of classes represents one test case. Using this approach, the size of SPGM will be 144 ( 3 2 × 2 4). However, a drawback of this approach is the possible occurrence of many invalid combinations of classes (e.g. A is a1 and E is e1), as contradictions to the constraints between variables stated in (3) of the specification of PGM. Basically, the classification-tree method aims at reducing the number of invalid combinations of classes, via the enforcement of constraints between classifications/classes in T. Suppose a classification tree TPGM for PGM is constructed as depicted in Fig. 1, either on an ad hoc basis or using Chen and Poon’s methodology [12, 13]. In Fig. 1, the small circle at the top of TPGM represents the input domain of PGM and is called the general root node. The classifications A, B and C directly under the general root node are called top level classifications. Also, every
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
classification in TPGM is the parent classification of those classes (known as its child classes) which are directly under it. Similarly, every class in TPGM is the parent class of those classifications (known as its child classifications) which are directly under it. For example, A is the parent classification of a1, a2 and a3, which in turn are the child classes of A; b2 is the parent class of F, which in turn is the child classification of b2. In TPGM every parent class has only one child classification. However, in some other T’s, a parent class may have two or more child classifications (refer to [5, 12, 13] for details). In Fig. 1, each column of the combination table corresponds exactly to a terminal node (which is a class) of TPGM. With the combination table, the construction rule states that a test case can be formed by selecting a combination of classes in TPGM via the following recursive process: 1. One and only one child class of each top level classification is selected. 2. For every selected class, one and only one child class of each child classification is selected. A test case constructed in this way is known as a ‘potential test case’. For example, row 1 of the combination table in Fig. 1 represents a potential test case where A is a1, B is b1, C is c1 and D is d1. We use Pi (i ⭓ 1) to denote a path in TPGM. For example, P9 denotes the path C–c1 –D–d1, in TPGM. Thus, the potential test case corresponding to row 1 is formed by combining all the classes which lie on P1, P6 and P9. Only part of the combination table is shown in Fig. 1. The complete combination table contains a total of 75 potential test cases. When compared to the 144 test cases which would have been constructed simply by selecting and combining one class of each classification, 69 test cases with invalid combinations of classes are effectively filtered out by TPGM. For example, in TPGM, it is impossible to combine b1 and f1 [which is invalid according to (3.v) of the specification of PGM] to form part of a potential test case, according to the construction rule of potential test cases.
3. The effectiveness of a classification tree Occasionally, not all the constraints between classifications can be enforced in T. Therefore, all the potential test cases constructed from the combination table have to be validated against the specification to ensure their consistency with the specification. Those found to be inconsistent with the specification are referred to as illegitimate test cases, which should be disregarded for testing. The remaining ones are referred to as legitimate test cases. Example 2 illustrates this concept.
767
• a1, b2, c1, d1, d2, f1 (formed by combining P1, P7 and P10). • a2, b2, c3, e1, f1 (formed by combining P3, P7 and P12). These two potential test cases are illegitimate, because the first one contains both d1 and d2 (which should be disjointed), and the second one contains both a2 and c3 [which contradicts the constraint (3.iii) of the specification of PGM]. In fact, out of the 75 potential test cases constructed from TPGM, 68 are illegitimate. Only the seven potential test cases listed below, are legitimate: • • • • • • •
a1, a1, a1, a1, a1, a1, a3,
b1, b2, b2, b1, b2, b2, b1,
c1, d1 (by combining P1, P6 and P9). c1, d1, f1 (by combining P1, P7 and P9). c1, d1, f2 (by combining P1, P8 and P9). c1, d2 (by combining P2, P6 and P10). c1, d2, f1 (by combining P2, P7 and P10). c1, d2, f2 (by combining P2, P8 and P10). c2 (by combining P5, P6 and P11).
We wish to point out that we are actually interested in the legitimate test cases, and T merely serves as a means to construct them. Based on this rationale, we define a metric ET (it is first introduced in [14]) to measure the effectiveness of T. This effectiveness metric is defined as: ET
l NT p NT
1
p l and NT denote the total number of legitimate test where NT cases and the total number of potential test cases constructed from T, respectively. p l ⭐ NT . The Obviously, we have 0 ⬍ ET ⭐ 1 as 0 ⬍ NT l value of NT can only be known after the identification of all illegitimate test cases from the potential test cases. p can be directly computed from T using the However, NT following equations. p : Equations for computing NT Suppose T has k (k ⭓ 1) top level classifications denoted as L1,…,Lk. Let ti denote the subtree (in T) with Li (1 ⭐ i ⭐ k) as its root, and N(ti) denote the total number of possible p is defined as: combinations of classes for ti. Hence, NT p NT
k Y
N ti
2
i1
where N(ti) is defined by the following recursive definitions. Let SXti and Sxti denote the subtrees (in ti) with classification X and class x as their roots, respectively. Also, let N SXti and N Sxti represent the total number of possible combinations of classes for SXti and Sxti , respectively. Obviously, ti SLtii , and N ti N SLtii . • For N SXti :
Example 2. The following are two potential test cases constructed from TPGM:
Suppose X has m child classes in which m1 are nonterminal classes (denoted as xj’s, 1 ⭐ j ⭐ m1) and m2 are
768
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
terminal classes (note m m1 ⫹ m2). Then, N SXti m2 ⫹
m1 X
x
N Stji
4. Some characteristics of ET 3
j1
• For N Sxti : Suppose x has n child classifications denoted as Yj’s (1 ⱕ j ⱕ n). Then, N Sxti
n Y
Y
N Stij
4
j1
p depends on the number of Eqs. (2)–(4) show that NT terminal nodes and the structure of T. Example 3 illustrates p and ETPGM . how to calculate NT PGM
Example 3. Refer to TPGM in Fig. 1. Let t1, t2 and t3 denote the subtrees with classifications A, B and C as their roots, respectively. Thus, N SD t1 2 N SEt1 2 N Sat11 N SD t1 2 N Sat12
N SEt1
2
N SAt1 1 ⫹ N Sat11 ⫹ N Sat12 1 ⫹ 2 ⫹ 2 5 Similarly, N SBt2 and N SCt3 are calculated as 3 and 5, respectively. Hence, p N t1 × N t2 × N t3 N SAt1 × N SBt2 × N SCt3 NT PGM
5 × 3 × 5 75 From Eq. (1), ETPGM is calculated as 0.09. [ (75 ⫺ 68)/75]. This small value of ETPGM ) means that TPGM is very ineffective in generating legitimate test cases. It should be noted that we deliberately chose TPGM (with a small value of ETPGM ) as an example to illustrate our restructuring algorithms to be presented later. Obviously, a small value of ET is undesirable as more effort is required in identifying illegitimate test cases from the potential test cases. Since this identification process is still performed manually, it is also more likely that mistakes will be made. This may affect the completeness of the set of legitimate test cases (if some of them are mistakenly classified as illegitimate and are consequently ignored) and it may in turn affect the comprehensiveness of the testing. Therefore, it is highly desirable to develop ways of improving the value of ET.
Obviously, before we can develop ways to improve the effectiveness of T, we need to know those features of T that may result in a small value of ET. An obvious approach is to examine all the 68 illegitimate test cases constructed from TPGM. Our examination of them reveals the following types of illegitimacy: 1. 2. 3. 4. 5. 6. 7.
Coexistence Coexistence Coexistence Coexistence Coexistence Coexistence Coexistence
of of of of of of of
d1 and d2. e1 and e2. (d1 or d2) and (e1 or e2). (d1 or d2) and (a2, a3, c2 or c3). (e1 or e2) and (a1, a3, c1 or c2). a2 and c3. a3 and b2.
Each of these results from some constraints mentioned in the specification of PGM which are not enforced in TPGM. In fact, all the above types of illegitimacy can be grouped into two categories depending on the causes of their occurrences: (i) existence of duplicated classifications under different Li’s; and (ii) existence of ‘partially dependent’ classifications. We further explain these two categories in the following subsections. 4.1. Existence of duplicated classifications under different Li’s 4.1.1. Illegitimacy types 1 and 2 Refer to TPGM in Fig. 1. Resulting from the duplication of D under both A and C, it is possible to select d1 and d2 from: • P1 and P10, respectively; or • P9 and P2, respectively. This results in the coexistence of d1 and d2, leading to illegitimacy type 1. Similarly, the duplication of E under both A and C allows the selection of both e1 and e2, leading to illegitimacy type 2. Intuitively, the number of illegitimate test cases will become larger when the number of duplicated classifications or subtrees (with more than one classification) under different Li’s grows. 4.1.2. Illegitimacy types 3, 4 and 5 E D E With SD t1 , St1 , St3 and St3 in TPGM, the following invalid combinations of classes would be constructed: • • • •
d1 and d1 and d2 and d2 and
e1 e2 e1 e2
[by [by [by [by
selecting selecting selecting selecting
(P1 (P1 (P2 (P2
and and and and
P12) or P13) or P12) or P13) or
(P9 and P3)]. (P9 and P4)]. (P10 and P3)]. (P10 and P4)].
This results in illegitimacy type 3, as a violation of the constraint (3.viii) of the specification of PGM. In fact, the duplications of D and E under A and C also lead to the violation of the constraints (3.i), (3.ii), (3.vi) and (3.vii) of
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
769
the specification. For these constraints, (3.ii) and (3.vii) correspond to illegitimacy type 4, whereas (3.i) and (3.vi) correspond to illegitimacy type 5.
exclusive classes a2 and c3, whereas those of type 7 contain the mutually exclusive classes a3 and b2.
4.2. Existence of partially dependent classifications
5. Our restructuring techniques
4.2.1. Illegitimacy types 6 and 7 Suppose X and Y are two distinct classifications. Then X is said to be ‘partially dependent’ on Y if all of the following conditions are met:
Obviously, simply knowing ET is not good enough. A more important issue is how to improve ET. The following subsections provide two algorithms for restructuring T in order to improve ET.
1. There exist a class x of X, distinct classes yi and yj (where i,j ⱖ 1) of Y such that x and yi may coexist, and x and yj cannot coexist. 2. There does not exist a class x 0 of X such that x 0 cannot coexist with any class of Y. 3. There does not exist a class y of Y such that y cannot coexist with any class of X. With the above relationship, X will not be placed under Y in T, and vice versa. Also, x and yj are said to be ‘mutually exclusive’. With respect to this hierarchical relation, we have the following proposition.
Proposition 1. If X is partially dependent on Y, then Y must also be partially dependent on X.
Proof. Suppose X is partially dependent on Y. Conditions (1) and (3) imply that there must be a class x 00 of X such that x 00 may coexist with yj of Y. Obviously, x and x 00 are different. This relationship between x 00 and yj, together with Conditions (2) and (3), imply that Y must be partially dependent on X.A
Example 4. 1:
Refer to the specification of PGM in Example
• When A is a2, C may be c1 or c2, but not c3. • There does not exist a class of A such that it cannot coexist with any class of C. • There does not exist a class of C such that it cannot coexist with any class of A. Hence, A is partially dependent on C, and classes a2 and c3 are mutually exclusive. By Proposition 1, C is also partially dependent on A. Caused by similar reasons, A and B are partially dependent on each other, and classes a3 and b2 are mutually exclusive.
5.1. For duplicated classifications under different Li’s As can be seen in Section 4.1, the duplications of D and E under A as well as under C (in TPGM) result in the cases of illegitimacy types 1–5. An obvious approach to this problem is to discard all D’s and all E’s from TPGM. In practice, this approach does not work as it will affect the completeness of the set of legitimate test cases. For examE D E ple, if SD t1 , St1 , St3 and St3 are deleted from TPGM in Fig. 1, then six out of the seven legitimate test cases listed in Section 3 will not be constructed. To eliminate the occurrence of illegitimate test cases resulting from the duplication of classifications under different Li’s, we propose the following algorithm: Tree restructuring algorithm for duplicated classifications (remove_duplicate): Refer to the notation used in Eqs. (2)–(4). Suppose there exists a subtree SXti (1 ⱕ i ⱕ k and k ⱖ 2) of T such that x is a child class of classification X, and x has Y as one of its child classifications. Also, suppose there exists a subtree SYtj (1 ⱕ j ⱕ k and j 苷 i) of T. Note that SYti SYtj . If SYti and SYtj occur only once in ti and tj, respectively, then T can be restructured as shown below. 1. Initially set the tree Y1 SXti . Then, prune SYY1 from Y1. 2. Construct the tree Y2 with X as its root, x as the unique child class of X, and SYti as the unique subtree of x (denoted as X–x–SYti ). 3. Replace SYtj with Y2. 4. Replace SXti according to one of the following steps: i. If Y1 X–x, replace SXti with a null tree (equivalent to pruning SXti from ti). ii. If Y1 苷 X–x and x is a terminal node of Y1, delete x from Y1 and then replace SXti with the newly formed Y1. iii. If Y1 苷 X–x and x is not a terminal node of Y1, replace SXti with Y1. In remove_duplicate, it should be noted that:
In fact, illegitimacy types 6 and 7 result from the occurrence of the partially dependent classifications A and C, and classifications A and B, respectively. More specifically, all the illegitimate test cases of type 6 contain the mutually
• Step (4.i) will be performed if x is the unique child class of X and Y is the unique child classification of x, in ti of T. • Step (4.ii) will be performed if x is not the unique child
770
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
0 Fig. 2. Construction sequence of TPGM .
class of X and Y is the unique child classification of x, in ti of T. • Step (4.iii) will be performed if Y is not the unique child classification of x, in ti of T. Let T 0 denote the classification tree after restructuring T using remove_duplicate. The following are the two important properties of remove_duplicate: p p 1. NT 0 ⱕ NT . 2. All legitimate test cases which can be constructed from T can still be constructed from T 0 , after reformatting all l l 0 NT ). the relevant potential test cases (hence, NT
Now, let us prove the first property and defer the proof of the second property until after the presentation of the reformatting algorithm.
Proposition 2.
p p NT 0 ⱕ NT .
Proof. Refer to remove_duplicate. Without loss of generality, let us assume that k 2 (i.e. T has two top level p N ti × N tj . classifications). Thus, NT
Obviously, N Y2 N SYti [hence, N Y2 N SYtj ]. Since: (i) Y1 is initially set to SXti ; and (ii) SYY1 is then pruned from Y1, Y1 is in fact a subtree of SXti . Thus, N Y1 ⱕ N SXti . Let ti 0 and tj 0 be ti and tj, respectively, after the application of remove_duplicate on T. Since N Y2 N SYtj , therefore, N(tj 0 ) N(tj). Since we have N Y1 ⱕ N SXti , therefore, N(ti 0 ) ⱕ N(ti). Thus, it follows from Eq. (2) that Ntp0 ⱕ Ntp .A Now, let us illustrate how to apply remove_duplicate.
Example 5. Refer to TPGM in Fig. 1. Again, let t1, t2 and t3 denote the subtrees with A, B and C as their roots, respecD tively. It should be noted that SD t1 and St3 occur only once in t1 and t3, respectively. The following is the sequence of restructuring applied to TPGM with the initialization of Y1 to SAt1 in step (1) of remove_duplicate: 1. 2. 3. 4.
Initially set Y1 SAt1 . Then, prune SD Y1 from Y1. Set Y2 to A–a1 –SD . t1 Replace SD t3 with Y2. Delete a1 from Y1 and then replace SAt1 with the newly
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
00 Fig. 3. TPGM after restructuring.
formed Y1 (because Y1 苷 A–a1 and a1 is a terminal node of Y1). The above restructuring steps and the newly formed classification tree T 0 PGM are depicted in Fig. 2. However, a different classification tree may be produced if Y1 is initialized as SCt3 instead of SAt1 in step (1) of remove_duplicate. In this case, the new sequence of restructuring would then be: 1. 2. 3. 4.
Initially set Y1 SCt3 . Then, prune SD Y1 from Y1. . Set Y2 to C–c1 –SD t3 Replace SD t1 with Y2. Delete c1 from Y1 and then replace SCt3 with the newly formed Y1 (because Y1 苷 C–c1 and c1 is a terminal node of Y1).
Fig. 3 depicts the classification tree T 00PGM formed using the above sequence of restructuring. Example 5 shows that remove_duplicate is nondeterministic because of the various ways of initializing p p and NT Y1 in its step (1). Using Eqs. (2)–(4), NT 0 00 PGM
PGM
are both calculated as 45. Thus, either T 0PGM or T 00PGM can be used for constructing the set of all potential test cases. However, in some situations where the restructured classification trees (formed by different initialization of Y1 in remove_duplicate) may have different N p’s, then the restructured classification tree with the smallest N p should be chosen. For Example 5, suppose T 0PGM in Fig. 2 is chosen for constructing the set of all potential test cases. Obviously, it is quite straightforward to construct the corresponding combination table from which all potential test cases can be identified. A close examination of all the 45 potential test cases constructed from T 0PGM , reveals that some illegitimate test cases occur because of the coexistence of (a1 and a2)
771
or (a1 and a3). These types of illegitimacy are not induced from the combination table of TPGM in Fig. 1 because a1, a2 and a3 are under the same top level classification A before restructuring. Therefore, only one of them can be selected at a time to form part of a potential test case. However, a1 and (a2 or a3) are now separately placed under two different top level classifications C and A of T 0PGM , respectively. This makes the selection of (a1 and a2) or (a1 and a3) possible. In fact, for the seven legitimate test cases constructed from TPGM , only one [i.e. the legitimate test case (a3, b1, c2)] of them can be constructed from T 0PGM . The remaining six legitimate test cases are converted into illegitimate ones because of the coexistence of (a1 and a2) or (a1 and a3). In view of this type of illegitimacy caused by restructuring of T, some potential test cases constructed from T 0 should be reformatted using the following algorithm (which is an extended version of the one in [14]) before the validation of their legitimacy against the specification: Test case reformatting algorithm (reformat_test_ case): Refer to remove_duplicate. Again, let T and T 0 denote the classification tree before and after being restructured, respectively. If the condition in step (4.ii) of remove_duplicate is satisfied (thus, SXti in T is replaced with a modified Y1 which does not contain x), then: 1. Let um (m ⱖ 1) be a child class of Li in T such that um is not an ancestor class (the ancestor classes of a classification H is defined as all the classes above H in T) of Y in SYti (of T), and vn (n ⱖ 1) be a child class of X in SXti (of T) such that vn 苷 x. When Li X, um vn. Construct a set of classes g which contains all the classes appearing in every Sutim and every Svtni , of T. 2. For any potential test case constructed from T 0 which contains classes from Y2 in tj 0 of T 0 , reformat this potential test case by deleting all the classes in g from it. else No reformatting is required. Let us illustrate how to apply reformat_test_case.
Example 6. Refer to Figs. 1 and 2. Note ti, tj, X and x in remove_duplicate correspond to t1, t3, A and a1 of TPGM in Fig. 1. It can be seen from Fig. 2 that SAt1 of TPGM is replaced with a modified Y1 in step (4.ii) of remove_duplicate. Additionally, 18 potential test cases [e.g. the potential test case (a1, a2, b1, c1, d1, e1)] contain one or more classes (a1, d1 and d2) selected from Y2 in t3 0 of T 0PGM . Therefore, these potential test cases should be reformatted as follows: 1. In this case, um vn as X Li. The two um’s (or vn’s) in step (1) of reformat_test_case correspond to a2 and a3 of TPGM in Fig. 1. Thus, g {a2, a3, e1, e2}.
772
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
2. Delete a2, a3, e1 and e2 (if any) from the 18 potential test cases which contain one or more classes selected from Y2 in t3 0 of T 0PGM . For example, one of these potential test cases (a1, a2, b1, c1, d1, e1) becomes (a1, b1, c1, d1) after the deletion process. After the application of reformat_test_case, all potential test cases should then be validated against the specification. After the removal of all illegitimate test cases, seven legitimate test cases remain (same as those listed in Section 3). From Eq. (1), ET 0PGM is calculated as 0.16 ( 7=45). Compared with ETPGM 0:09, the improvement is about 78%. For each potential test case constructed from T, there is a corresponding set of paths in T. Let Fti denote the set of those paths appearing in ti of T. For example, in TPGM of Fig. 1, {A–a1 –D–d1} is a Ft1 (denoted as F1). In this example, F1, contains only one path because D is the unique child classification of a1. If, however, a1 has more than one child classification, then F1 would contain more than one path. Now, we are ready to prove the second property, i.e. with the use of remove_duplicate and reformat_test_case, all legitimate test cases constructed from T are preserved in T 0 .
Proposition 3. All legitimate test cases which can be constructed from T can still be constructed from T 0 , after the application of reformat_test_case (i.e. l l 0 NT ). NT
Proof. Again, let us assume that k 2, and the two subtrees (in T) with Li and Lj (1 ⭐ i,j ⭐ 2 and i 苷 j) as their roots are ti and tj, respectively. If a Fti contains any class of SYti , then it is denoted as F1ti . Otherwise, it is denoted as F2ti . Since every potential test case is constructed by combining all the classes which appear in the paths of a Fti and a Ftj , it must belong to one of the following types: 1. One formed by combining all the classes in the paths of a F1ti and a F1tj . 2. One formed by combining all the classes in the paths of a F1ti and a F2tj . 3. One formed by combining all the classes in the paths of a F2ti and a F1tj . 4. One formed by combining all the classes in the paths of a F2ti and a F2tj . Potential test cases of type 1 may be legitimate or illegitimate. To analyse the impact of remove_duplicate on the legitimate test cases of this type, we have to consider the following situations:
• The condition of step (4.i) of remove_duplicate is satisfied (thus, SXti in T is replaced with a null tree) • If (X in ti of T) 苷 Li Although subpaths (which contain classes in SXti of T) of some elements of F1ti are deleted as a result of the replacement of SXti in T with a null tree, these deleted subpaths reappear in Y2 which replaces SYtj in T. Thus, all the potential test cases (and hence the legitimate test cases) of type 1 constructed from T can still be formed from T 0 by combining all the classes appearing in the paths of F2ti 0 and F1tj 0 . • If (X in ti of T) Li It is obvious that SXti ti . Thus, the replacement of SXti in T with a null tree is equivalent to the pruning of ti from T. After the pruning process, all F1ti ’s are removed and T 0 tj 0 . However, all the potential test cases (and hence, the legitimate test cases) of type 1 constructed from T can still be formed from T 0 , because x and SYtj are contained in Y2 which replaces SYtj in T. • The condition of step (4.ii) of remove_duplicate is satisfied (thus, SXti in T is replaced with a modified Y1 which does not contain x) As a result of the replacement of SXti in T with the modified Y1, subpaths (which contain classes in Sxti of T) of some elements of Ft1i will be deleted. However, these deleted subpaths appear in Y2 which replaces SYtj in T. Also, since Sxti (and hence SYti ) does not appear in ti 0 , F1ti 0 Ø. As a result, all classes in the paths of F1tj 0 have to be combined with all classes in the paths of F2ti 0 to form a potential test case. However, any potential test case containing this combination is illegitimate because all the classes in SYtj , (some of these classes must exist in certain elements of F1tj 0 ) cannot coexist with any class from any Sutim0 and Svtni 0 of T 0 (note some classes of Sutim0 or Svtni 0 must appear in some elements of F2ti 0 ). Obviously, any illegitimate test case of this type can be converted into a legitimate one through the removal of all the classes in every Sutim Sutim0 and every Svtni Svtni 0 (i.e. steps (1) and (2) of reformat_test_case). Thus, no legitimate test cases constructed from T will be omitted from T 0 . • The condition of step (4.iii) of remove_duplicate is satisfied (thus, SXti in T is replaced with Y1) Although subpaths (which contain classes in SYti of T) of some elements of F1ti are deleted because of the replacement of SXti in T with Y1, these deleted subpaths reappear in Y2 which replaces SYtj in T. Therefore, all the potential test cases (and hence, the legitimate test cases) of type 1 constructed from T are still being formed from T 0 via the combination of all the classes appearing in the paths of F2ti 0 and F1tj 0 .
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Potential test cases of type 2 are illegitimate because some classes appearing in the paths of F2tj cannot coexist with any class appearing in SYtj . For a similar reason, potential test cases of type 3 are also illegitimate. Finally, let us look at potential test cases of type 4. They remain unchanged after the restructuring process because F2ti and F2tj are left intact, as seen in remove_ duplicate. Therefore, all legitimate test cases constructed from T can still be formed from T 0 after the application of l l 0 NT .A reformat_test_case. As a result, NT 5.2. For partially dependent classifications in different t i’s In general, illegitimate test cases caused by a pair of partially dependent classifications are difficult, if not impossible, to remove, through restructuring of T (with the preservation of all legitimate test cases). However, ET can be improved by restructuring T when the duplicated classifications are under mutually exclusive classes of their respective parent classifications (which are partially dependent on each other). Under this special circumstance, the following restructuring algorithm can be used to improve ET: Tree restructuring algorithm for partially dependent classifications (remove_partial_dependence): Suppose T has k (k ⱖ 2) top level classifications denoted as L1,…,Lk; and ti and tj (1 ⱕ i,j ⱕ k) denote the distinct subtrees (in T) with Li and Lj as their roots, respectively. Also, suppose there exist two identical subtrees SYti and SYtj in T. If: (i) SYti and SYtj occur only once in ti and tj, respectively; (ii) some ancestor classifications (the ancestor classifications of a subtree S or a classification H, is defined as all the classifications above S or H, in T, respectively) M and N of SYti and SYtj , respectively, are partially dependent on each other; and (iii) SYti and SYtj have m (a child class of M) and n (a child class of N) as one of their ancestor classes, respectively, such that m and n cannot coexist (i.e. m and n are mutually exclusive classes), then T can be restructured as shown below: 1. If m is the unique child class of M, then prune SM ti from from T. T. Otherwise, prune Sm ti 2. If n is the unique child class of N, then prune SNtj from T. Otherwise, prune Sntj from T. It should be noted that ti and tj in the restructured classification tree cannot be null. Otherwise, T must be incorrect because no legitimate test cases can be constructed from it. Let T 0 and T 000 denote the classification trees after restructuring T using remove_duplicate and remove_partial_dependence, respectively. We have the following relationships: p p p • NT 000 ⱕ NT 0 ⱕ NT .
773
• All legitimate test cases which can be constructed from l l 000 NT ). T can also be constructed from T 000 (i.e. NT Now, let us prove these relationships.
Proposition 4.
p p p NT 000 ⱕ NT 0 ⱕ NT .
Proof. Refer to remove_duplicate and remove_partial_dependence. Suppose, without loss of generality, that k 2, and the two top level classifications of T are L1 and L2. Let t1 and t2 correspond to ti and tj mentioned in these two algorithms, respectively; t1 0 and t2 0 be t1 and t2, respectively, after restructuring T with remove_duplicate; and t1 000 and t2 000 be t1 and t2, respectively, after restructuring T with remove_partial_dependence. Also, suppose that the parent classes of SYt1 and SYt2 in T are x and w, respectively, and that the parent classifications of x and w are X and W, respectively. It should be noted that X, x, W and w may be M, m, N and n, respectively, as mentioned in remove_partial_dependence. Suppose T 0 is formed with the initialization of Y1 to SXt1 in step (1) of remove_duplicate. Then t1 0 of T 0 is formed by pruning either SXt1 , Sxt1 or SYt1 (corresponding to step (4.i), (4.ii) or (4.iii) of remove_duplicate, respectively) from t1 of T. Now, let us consider t1 000 of T 000 . The subtree t1 000 is t1 m with SM t1 or St1 pruned. Since M X or M is an ancestor classification of X, thus, we have either t1 000 t1 0 or t1 000 is a subtree of t1 0 . Therefore, N t1 000 ⱕ N t1 0 . Now, consider t2 0 and t2 000 . From the proof of Proposition 2, we know that N t2 0 N t2 . Since t2 000 is t2 with SNt2 or Snt2 pruned, t2 000 is in fact a subtree of t2 (hence, N t2 000 ⱕ N t2 ). It follows that N t2 000 ⱕ N t2 0 . p p 0 0 000 and NT Since NT 0 N t1 × N t2 000 N t1 × 000 000 0 N t2 , and we know that N t1 ⱕ N t1 and p p N t2 000 ⱕ N t2 0 , so we must have NT 000 ⱕ NT 0 . From p p p Proposition 2, NT 000 ⱕ NT 0 ⱕ NT immediately follows.A Proposition 4 implies that whenever possible, remove_partial_dependence should be used instead of remove_duplicate for improving ET.
Proposition 5. All legitimate test cases which can be constructed from T can also be constructed from T 000 l l 000 ⱕ NT ). (i.e. NT
Proof. We follow the proof of Proposition 4. For each potential test case constructed from T, let G be its corresponding set of paths in T. Also, let Gt1 and Gt2 denote the sets of those paths appearing in t1 and t2 of T, respectively. If a Gt1 contains m, then it is denoted as G1t1 . Otherwise, it is denoted as G2t2 . Similarly, if a Gt2 contains n, then it is denoted as G1t2 . Otherwise, it is denoted as G2t2 . Since every potential test case is constructed by combining
774
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Fig. 4. T 000PGM after restructuring.
all classes which appear in the paths of a Gt1 and a Gt2 , it must belong to one of the following types: 1. One formed by combining all the classes in the paths of a G1t1 and a G1t2 . 2. One formed by combining all the classes in the paths of a G1t1 and a G2t2 . 3. One formed by combining all the classes in the paths of a G2t1 and a G1t2 . 4. One formed by combining all the classes in the paths of a G2t1 and a G2t2 . All the potential test cases of type 1 are illegitimate because they contain both m and n, which are mutually exclusive classes. Since SYt1 and SYt2 occur only once in t1 and t2, respectively, Some classes in the paths of G1t1 and G2t1 cannot coexist with some classes in the paths of G2t2 and G1t2 , respectively. Hence, all potential test cases of types 2 and 3 are also illegitimate. For potential test cases of type 4, they may be legitimate or illegitimate. Since G2t1 and G2t2 are invariants with respect to the application of remove_partial_dependence, therefore, all the legitimate test cases constructed from T l l 000 NT ).A are retained in T 000 (i.e. NT Example 7 illustrates the application of remove_partial_dependence and the resultant set of legitimate test cases. Example 7. Refer to TPGM in Fig. 1. Let t1, t2 and t3 denote the subtrees with A, B and C as their roots, respectively. Since: (i) SEt1 and SEt3 occur only once in t1 and t3, respectively; (ii) the parent classifications A and C of SEt1 and SEt3 , respectively, are partially dependent on each other; and (iii) the parent classes a2 and c3 of SEt1 and SEt3 , respectively, are mutually exclusive, we should apply remove_partial_dependence to restructure TPGM in order to improve ETPGM . The restructuring steps are as follows: 1. A and a2 in TPGM correspond to M and m in remove_partial_dependence, respectively. Since a2 is not the unique child class of A, Sat12 is pruned from TPGM.
2. C and c3 in TPGM correspond to N and n in remove_partial_dependence, respectively. Since c3 is not the unique child class of C, Sct33 is pruned from TPGM. 000 after restructuring The resultant classification tree T PGM is depicted in Fig. 4. p 000 of Fig. 4, NT is calculated as 27 using From T PGM 000 PGM Eqs. (2)–(4). Also, the seven legitimate test cases constructed from TPGM can still be formed from T 000PGM. For example, the legitimate test case (a1, b1, c1, d1) constructed from TPGM, can be formed by combining all the classes which lie on P1 000 , P4 000 and P7 000 in Fig. 4. By Eq. (1), ET 000PGM is calculated as 0.26( 7/27). When comparing with ETPGM 0:09, the improvement is about 189%. It should be noted that, in Fig. 4, ET 000PGM can be further improved by restructuring T 000PGM using remove_ duplicate for removing the duplication of the classification D. If this happens, ET 000PGM will be further improved to 0.78 (about 767% improvement).
6. Conclusion As testing plays an important role in the assurance of the correctness (and hence the quality) of software, a systematic way of identifying test cases from specifications is essential. It is because the comprehensiveness of the selected test cases will affect the quality of testing. In view of this, the classification-tree method developed by Grochtmann and Grimm [5] provides a useful direction in pursuit of testing. Unfortunately, their tree construction method is rather ad hoc. This problem has motivated Chen and Poon [12, 13] to develop the classification-hierarchy table and its associated tree construction algorithm, by which classification trees can be constructed systematically from specifications. However, their construction algorithm does not take into consideration the effectiveness of classification trees. In this paper, we have introduced a metric ET to measure the effectiveness of a classification tree. Intuitively speaking,
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
ET is a measure of the cost-effectiveness of constructing the set of legitimate test cases from a classification tree. Simply knowing the value of ET for a classification tree is insufficient. A more important issue is how to improve ET. To address this issue, this paper has investigated the relationship between the structure of a classification tree and its ET value, focusing on those features which could result in a poor ET value. We observe that there are two sources leading to a poor ET value, the existence of duplicated classifications under different top level classifications, and partially dependent classifications. From this observation, we have proposed two algorithms to improve ET through restructuring the classification trees, prior to the construction of potential test cases. References [1] R. Ferguson, B. Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology 5 (1) (1996) 63–86. [2] B. Korel, Automated test data generation for programs with procedures, in: ISSTA’96: Proceedings of the 1996 International Symposium on Software Testing and Analysis, January 1996, pp. 209-215. [3] J. Sanders, E. Curran, Software Quality: A Framework for Success in Software Development and Support. Addison-Wesley, Reading, MA, 1994. [4] T. Chusho, Test data selection and quality estimation based on the concept of essential branches for path testing, IEEE Transactions on Software Engineering 13 (5) (1987) 509–517.
775
[5] M. Grochtmann, K. Grimm, Classification trees for partition testing, Software Testing, Verification and Reliability 3 (1993) 63–82. [6] T.J. Ostrand, M.J. Balcer, The category-partition method for specifying and generating functional tests, Communications of the ACM 31 (6) (1988) 676–686. [7] T.Y. Chen, M.F. Lau, Two test data selection strategies towards testing of boolean specifications, in: COMPSAC’97. Proceedings of the Twenty-First Annual International Computer Software and Applications Conference, IEEE Computer Society Press, August 1997, pp. 608–611. [8] J.J. Chilenski, S.P. Miller, Applicability of modified condition/decision coverage to software testing, Software Engineering Journal 9 (5) (1994) 193–200. [9] E.J. Weyuker, T. Goradia, A. Singh, Automatically generating test data from a boolean specification, IEEE Transactions on Software Engineering 20 (5) (1994) 353–363. [10] R. Weber, EDP Auditing Conceptual Foundations and Practice, 2nd ed., McGraw-Hill, New York, 1988. [11] T.Y. Chen, P.L. Poon, S.F. Tang, A systematic method for auditing user acceptance tests, IS Audit and Control Journal 5 (1998) 31– 36. [12] T.Y. Chen, P.L. Poon, Classification-hierarchy table: a methodology for constructing the classification tree, in: ASWEC’96: Proceedings of the Australian Software Engineering Conference, IEEE Computer Society Press, July 1996, pp. 93–104. [13] T.Y. Chen, P.L. Poon, Construction of classification trees via the classification-hierarchy table, Information and Software Technology 39 (13) (1997) 889–896. [14] T.Y. Chen, P.L. Poon, Improving the quality of classification trees via restructuring, in: APSEC’96: Proceedings of the Asia-Pacific Software Engineering Conference, IEEE Computer Society Press, December 1996, pp. 83–92.
Information and Software Technology 40 (1998) 765–775
On the effectiveness of classification trees for test case construction 夽 T.Y. Chen*, P.L. Poon Department of Computer Science, University of Melbourne, Parkville 3052, Australia Received 21 April 1998; accepted 26 August 1998
Abstract The notion of the classification-hierarchy table and the classification-tree construction algorithm provide a systematic approach to the construction of classification trees from given sets of classifications and their associated classes. Using classification trees, the set of all possible test cases can be constructed from functional specifications. This paper extends their study by introducing a metric to measure the effectiveness of a classification tree with respect to the construction of test cases, and providing ways to improve this effectiveness. 䉷 1998 Elsevier Science B.V. All rights reserved. Keywords: Classification-hierarchy table; Classification-tree method; Software testing; Specification-based testing; Test case selection
1. Introduction Various people [1–3] have argued that the expense of software testing is often underestimated although it may account for up to 50% of the project cost. Because testing is an expensive process, its effectiveness should be improved by means of systematic planning, execution and monitoring. As it is well-known that the construction of the set of all possible test cases S, affects the comprehensiveness and, hence, the quality of the test [4–6], numerous researchers have developed their own methods for this construction process. Two approaches to this construction are the black box and the white box testing. For the former approach, most techniques are developed for formal specifications only [7– 9], despite the fact that many real-life specifications are written in an informal way. On the other hand, the few techniques developed for both formal and informal specifications include the classification-tree method [5], the category-partition method [6] and the decision-table method [10]. The classification-tree method is an extension of the category-partition method. Also, the decision-table method can be regarded as a degenerated form of the classification-tree method. With respect to the generation of S, the classification-tree method is Corresponding author. Tel.: 61 3 9287 9101; fax: 61 3 9348 1184; e-mail: [email protected] 夽 This project was partially supported by a research grant from the Australian Research Council.
more effective. Readers may refer to [11] for a detailed comparison between these methods. In general, the classification-tree method developed by Grochtmann and Grimm [5], helps construct S from functional specifications (hereafter referred to as the ‘specifications’), via the construction of classification trees. However, their tree construction method is rather ad hoc, resulting in the variation of classification trees constructed from the same specification according to the tester’s own expertise and experience. This problem is subsequently solved by Chen and Poon [12,13] using the notion of classification-hierarchy table (which captures the hierarchical relation for each pair of distinct classifications), and a method for the construction of classification trees from given sets of classifications and their associated classes. However, their tree construction methodology does not take into account the effectiveness of the classification trees, with respect to the construction of test cases. This paper attempts to solve the above problem by: (i) defining a metric to measure the effectiveness of a classification tree; (ii) identifying some relationships between a classification tree and this effectiveness metric; and (iii) providing ways to improve this effectiveness metric of a classification tree. The rest of this paper is structured as follows. Section 2 gives an overview of the classification-tree method. Section 3 introduces an effectiveness metric of a classification tree and its underlying rationale. Section 4 describes some structures of a classification tree which could lead to a poor effectiveness metric. Section 5 presents two restructuring
0950-5849/98/$ - see front matter 䉷 1998 Elsevier Science B.V. All rights reserved. PII: S0950-584 9(98)00107-4
766
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Fig. 1. TPGM and part of its combination table.
algorithms for a classification tree so as to improve its effectiveness metric. Finally, Section 6 concludes the paper.
2. An overview of the classification-tree method A classification tree, T, is a graphical representation of the hierarchical relation between various classifications. The following describes the major steps of the classification-tree method: 1. From the specification, identify all classifications (defined as the different criteria for partitioning the input domain of the program to be tested) and their associated classes (defined as the disjoint subsets of values for each classification). 2. From the sets of classifications and their associated classes, construct T. 3. From T, construct the corresponding combination table. 4. From the combination table, identify all the feasible combinations of classes. Each combination of classes represents one potential test case. Example 1 illustrates this method.
Example 1. Suppose we have program PGM, with its specification as shown below, for testing: 1. PGM has six input variables, namely, A, B, C, D, E and F. 2. Each variable A and C has three possible values (e.g. a1, a2 and a3 for A), whereas each remaining variable has two possible values (e.g. b1 and b2 for B). 3. The input domain of PGM may contain any combination of possible values from some of these variables except the following:
i. (A is a1 or a3) and (E is e1 or e2). ii. (A is a2 or a3) and (D is d1 or d2). iii. (A is a2) and (C is c3). iv. (A is a3) and (B is b2). v. (B is b1) and (F is f1 or f2). vi. (C is c1 or c2) and (E is e1 or e2). vii. (C is c2 or c3) and (D is d1 or d2). viii. (D is d1 or d2) and (E is e1 or e2). Suppose all of the classifications and classes for PGM are simply defined as the input variables and their possible values, respectively. Thus, for example, A is taken as a classification with a1, a2 and a3 as its three associated classes (throughout this paper, classifications and classes are denoted by capital and small letters, respectively). Obviously, with these classifications and their associated classes, one approach to the construction of SPGM (which denotes S for PGM) is to select and combine one class of each classification, so that each combination of classes represents one test case. Using this approach, the size of SPGM will be 144 ( 3 2 × 2 4). However, a drawback of this approach is the possible occurrence of many invalid combinations of classes (e.g. A is a1 and E is e1), as contradictions to the constraints between variables stated in (3) of the specification of PGM. Basically, the classification-tree method aims at reducing the number of invalid combinations of classes, via the enforcement of constraints between classifications/classes in T. Suppose a classification tree TPGM for PGM is constructed as depicted in Fig. 1, either on an ad hoc basis or using Chen and Poon’s methodology [12, 13]. In Fig. 1, the small circle at the top of TPGM represents the input domain of PGM and is called the general root node. The classifications A, B and C directly under the general root node are called top level classifications. Also, every
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
classification in TPGM is the parent classification of those classes (known as its child classes) which are directly under it. Similarly, every class in TPGM is the parent class of those classifications (known as its child classifications) which are directly under it. For example, A is the parent classification of a1, a2 and a3, which in turn are the child classes of A; b2 is the parent class of F, which in turn is the child classification of b2. In TPGM every parent class has only one child classification. However, in some other T’s, a parent class may have two or more child classifications (refer to [5, 12, 13] for details). In Fig. 1, each column of the combination table corresponds exactly to a terminal node (which is a class) of TPGM. With the combination table, the construction rule states that a test case can be formed by selecting a combination of classes in TPGM via the following recursive process: 1. One and only one child class of each top level classification is selected. 2. For every selected class, one and only one child class of each child classification is selected. A test case constructed in this way is known as a ‘potential test case’. For example, row 1 of the combination table in Fig. 1 represents a potential test case where A is a1, B is b1, C is c1 and D is d1. We use Pi (i ⭓ 1) to denote a path in TPGM. For example, P9 denotes the path C–c1 –D–d1, in TPGM. Thus, the potential test case corresponding to row 1 is formed by combining all the classes which lie on P1, P6 and P9. Only part of the combination table is shown in Fig. 1. The complete combination table contains a total of 75 potential test cases. When compared to the 144 test cases which would have been constructed simply by selecting and combining one class of each classification, 69 test cases with invalid combinations of classes are effectively filtered out by TPGM. For example, in TPGM, it is impossible to combine b1 and f1 [which is invalid according to (3.v) of the specification of PGM] to form part of a potential test case, according to the construction rule of potential test cases.
3. The effectiveness of a classification tree Occasionally, not all the constraints between classifications can be enforced in T. Therefore, all the potential test cases constructed from the combination table have to be validated against the specification to ensure their consistency with the specification. Those found to be inconsistent with the specification are referred to as illegitimate test cases, which should be disregarded for testing. The remaining ones are referred to as legitimate test cases. Example 2 illustrates this concept.
767
• a1, b2, c1, d1, d2, f1 (formed by combining P1, P7 and P10). • a2, b2, c3, e1, f1 (formed by combining P3, P7 and P12). These two potential test cases are illegitimate, because the first one contains both d1 and d2 (which should be disjointed), and the second one contains both a2 and c3 [which contradicts the constraint (3.iii) of the specification of PGM]. In fact, out of the 75 potential test cases constructed from TPGM, 68 are illegitimate. Only the seven potential test cases listed below, are legitimate: • • • • • • •
a1, a1, a1, a1, a1, a1, a3,
b1, b2, b2, b1, b2, b2, b1,
c1, d1 (by combining P1, P6 and P9). c1, d1, f1 (by combining P1, P7 and P9). c1, d1, f2 (by combining P1, P8 and P9). c1, d2 (by combining P2, P6 and P10). c1, d2, f1 (by combining P2, P7 and P10). c1, d2, f2 (by combining P2, P8 and P10). c2 (by combining P5, P6 and P11).
We wish to point out that we are actually interested in the legitimate test cases, and T merely serves as a means to construct them. Based on this rationale, we define a metric ET (it is first introduced in [14]) to measure the effectiveness of T. This effectiveness metric is defined as: ET
l NT p NT
1
p l and NT denote the total number of legitimate test where NT cases and the total number of potential test cases constructed from T, respectively. p l ⭐ NT . The Obviously, we have 0 ⬍ ET ⭐ 1 as 0 ⬍ NT l value of NT can only be known after the identification of all illegitimate test cases from the potential test cases. p can be directly computed from T using the However, NT following equations. p : Equations for computing NT Suppose T has k (k ⭓ 1) top level classifications denoted as L1,…,Lk. Let ti denote the subtree (in T) with Li (1 ⭐ i ⭐ k) as its root, and N(ti) denote the total number of possible p is defined as: combinations of classes for ti. Hence, NT p NT
k Y
N ti
2
i1
where N(ti) is defined by the following recursive definitions. Let SXti and Sxti denote the subtrees (in ti) with classification X and class x as their roots, respectively. Also, let N SXti and N Sxti represent the total number of possible combinations of classes for SXti and Sxti , respectively. Obviously, ti SLtii , and N ti N SLtii . • For N SXti :
Example 2. The following are two potential test cases constructed from TPGM:
Suppose X has m child classes in which m1 are nonterminal classes (denoted as xj’s, 1 ⭐ j ⭐ m1) and m2 are
768
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
terminal classes (note m m1 ⫹ m2). Then, N SXti m2 ⫹
m1 X
x
N Stji
4. Some characteristics of ET 3
j1
• For N Sxti : Suppose x has n child classifications denoted as Yj’s (1 ⱕ j ⱕ n). Then, N Sxti
n Y
Y
N Stij
4
j1
p depends on the number of Eqs. (2)–(4) show that NT terminal nodes and the structure of T. Example 3 illustrates p and ETPGM . how to calculate NT PGM
Example 3. Refer to TPGM in Fig. 1. Let t1, t2 and t3 denote the subtrees with classifications A, B and C as their roots, respectively. Thus, N SD t1 2 N SEt1 2 N Sat11 N SD t1 2 N Sat12
N SEt1
2
N SAt1 1 ⫹ N Sat11 ⫹ N Sat12 1 ⫹ 2 ⫹ 2 5 Similarly, N SBt2 and N SCt3 are calculated as 3 and 5, respectively. Hence, p N t1 × N t2 × N t3 N SAt1 × N SBt2 × N SCt3 NT PGM
5 × 3 × 5 75 From Eq. (1), ETPGM is calculated as 0.09. [ (75 ⫺ 68)/75]. This small value of ETPGM ) means that TPGM is very ineffective in generating legitimate test cases. It should be noted that we deliberately chose TPGM (with a small value of ETPGM ) as an example to illustrate our restructuring algorithms to be presented later. Obviously, a small value of ET is undesirable as more effort is required in identifying illegitimate test cases from the potential test cases. Since this identification process is still performed manually, it is also more likely that mistakes will be made. This may affect the completeness of the set of legitimate test cases (if some of them are mistakenly classified as illegitimate and are consequently ignored) and it may in turn affect the comprehensiveness of the testing. Therefore, it is highly desirable to develop ways of improving the value of ET.
Obviously, before we can develop ways to improve the effectiveness of T, we need to know those features of T that may result in a small value of ET. An obvious approach is to examine all the 68 illegitimate test cases constructed from TPGM. Our examination of them reveals the following types of illegitimacy: 1. 2. 3. 4. 5. 6. 7.
Coexistence Coexistence Coexistence Coexistence Coexistence Coexistence Coexistence
of of of of of of of
d1 and d2. e1 and e2. (d1 or d2) and (e1 or e2). (d1 or d2) and (a2, a3, c2 or c3). (e1 or e2) and (a1, a3, c1 or c2). a2 and c3. a3 and b2.
Each of these results from some constraints mentioned in the specification of PGM which are not enforced in TPGM. In fact, all the above types of illegitimacy can be grouped into two categories depending on the causes of their occurrences: (i) existence of duplicated classifications under different Li’s; and (ii) existence of ‘partially dependent’ classifications. We further explain these two categories in the following subsections. 4.1. Existence of duplicated classifications under different Li’s 4.1.1. Illegitimacy types 1 and 2 Refer to TPGM in Fig. 1. Resulting from the duplication of D under both A and C, it is possible to select d1 and d2 from: • P1 and P10, respectively; or • P9 and P2, respectively. This results in the coexistence of d1 and d2, leading to illegitimacy type 1. Similarly, the duplication of E under both A and C allows the selection of both e1 and e2, leading to illegitimacy type 2. Intuitively, the number of illegitimate test cases will become larger when the number of duplicated classifications or subtrees (with more than one classification) under different Li’s grows. 4.1.2. Illegitimacy types 3, 4 and 5 E D E With SD t1 , St1 , St3 and St3 in TPGM, the following invalid combinations of classes would be constructed: • • • •
d1 and d1 and d2 and d2 and
e1 e2 e1 e2
[by [by [by [by
selecting selecting selecting selecting
(P1 (P1 (P2 (P2
and and and and
P12) or P13) or P12) or P13) or
(P9 and P3)]. (P9 and P4)]. (P10 and P3)]. (P10 and P4)].
This results in illegitimacy type 3, as a violation of the constraint (3.viii) of the specification of PGM. In fact, the duplications of D and E under A and C also lead to the violation of the constraints (3.i), (3.ii), (3.vi) and (3.vii) of
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
769
the specification. For these constraints, (3.ii) and (3.vii) correspond to illegitimacy type 4, whereas (3.i) and (3.vi) correspond to illegitimacy type 5.
exclusive classes a2 and c3, whereas those of type 7 contain the mutually exclusive classes a3 and b2.
4.2. Existence of partially dependent classifications
5. Our restructuring techniques
4.2.1. Illegitimacy types 6 and 7 Suppose X and Y are two distinct classifications. Then X is said to be ‘partially dependent’ on Y if all of the following conditions are met:
Obviously, simply knowing ET is not good enough. A more important issue is how to improve ET. The following subsections provide two algorithms for restructuring T in order to improve ET.
1. There exist a class x of X, distinct classes yi and yj (where i,j ⱖ 1) of Y such that x and yi may coexist, and x and yj cannot coexist. 2. There does not exist a class x 0 of X such that x 0 cannot coexist with any class of Y. 3. There does not exist a class y of Y such that y cannot coexist with any class of X. With the above relationship, X will not be placed under Y in T, and vice versa. Also, x and yj are said to be ‘mutually exclusive’. With respect to this hierarchical relation, we have the following proposition.
Proposition 1. If X is partially dependent on Y, then Y must also be partially dependent on X.
Proof. Suppose X is partially dependent on Y. Conditions (1) and (3) imply that there must be a class x 00 of X such that x 00 may coexist with yj of Y. Obviously, x and x 00 are different. This relationship between x 00 and yj, together with Conditions (2) and (3), imply that Y must be partially dependent on X.A
Example 4. 1:
Refer to the specification of PGM in Example
• When A is a2, C may be c1 or c2, but not c3. • There does not exist a class of A such that it cannot coexist with any class of C. • There does not exist a class of C such that it cannot coexist with any class of A. Hence, A is partially dependent on C, and classes a2 and c3 are mutually exclusive. By Proposition 1, C is also partially dependent on A. Caused by similar reasons, A and B are partially dependent on each other, and classes a3 and b2 are mutually exclusive.
5.1. For duplicated classifications under different Li’s As can be seen in Section 4.1, the duplications of D and E under A as well as under C (in TPGM) result in the cases of illegitimacy types 1–5. An obvious approach to this problem is to discard all D’s and all E’s from TPGM. In practice, this approach does not work as it will affect the completeness of the set of legitimate test cases. For examE D E ple, if SD t1 , St1 , St3 and St3 are deleted from TPGM in Fig. 1, then six out of the seven legitimate test cases listed in Section 3 will not be constructed. To eliminate the occurrence of illegitimate test cases resulting from the duplication of classifications under different Li’s, we propose the following algorithm: Tree restructuring algorithm for duplicated classifications (remove_duplicate): Refer to the notation used in Eqs. (2)–(4). Suppose there exists a subtree SXti (1 ⱕ i ⱕ k and k ⱖ 2) of T such that x is a child class of classification X, and x has Y as one of its child classifications. Also, suppose there exists a subtree SYtj (1 ⱕ j ⱕ k and j 苷 i) of T. Note that SYti SYtj . If SYti and SYtj occur only once in ti and tj, respectively, then T can be restructured as shown below. 1. Initially set the tree Y1 SXti . Then, prune SYY1 from Y1. 2. Construct the tree Y2 with X as its root, x as the unique child class of X, and SYti as the unique subtree of x (denoted as X–x–SYti ). 3. Replace SYtj with Y2. 4. Replace SXti according to one of the following steps: i. If Y1 X–x, replace SXti with a null tree (equivalent to pruning SXti from ti). ii. If Y1 苷 X–x and x is a terminal node of Y1, delete x from Y1 and then replace SXti with the newly formed Y1. iii. If Y1 苷 X–x and x is not a terminal node of Y1, replace SXti with Y1. In remove_duplicate, it should be noted that:
In fact, illegitimacy types 6 and 7 result from the occurrence of the partially dependent classifications A and C, and classifications A and B, respectively. More specifically, all the illegitimate test cases of type 6 contain the mutually
• Step (4.i) will be performed if x is the unique child class of X and Y is the unique child classification of x, in ti of T. • Step (4.ii) will be performed if x is not the unique child
770
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
0 Fig. 2. Construction sequence of TPGM .
class of X and Y is the unique child classification of x, in ti of T. • Step (4.iii) will be performed if Y is not the unique child classification of x, in ti of T. Let T 0 denote the classification tree after restructuring T using remove_duplicate. The following are the two important properties of remove_duplicate: p p 1. NT 0 ⱕ NT . 2. All legitimate test cases which can be constructed from T can still be constructed from T 0 , after reformatting all l l 0 NT ). the relevant potential test cases (hence, NT
Now, let us prove the first property and defer the proof of the second property until after the presentation of the reformatting algorithm.
Proposition 2.
p p NT 0 ⱕ NT .
Proof. Refer to remove_duplicate. Without loss of generality, let us assume that k 2 (i.e. T has two top level p N ti × N tj . classifications). Thus, NT
Obviously, N Y2 N SYti [hence, N Y2 N SYtj ]. Since: (i) Y1 is initially set to SXti ; and (ii) SYY1 is then pruned from Y1, Y1 is in fact a subtree of SXti . Thus, N Y1 ⱕ N SXti . Let ti 0 and tj 0 be ti and tj, respectively, after the application of remove_duplicate on T. Since N Y2 N SYtj , therefore, N(tj 0 ) N(tj). Since we have N Y1 ⱕ N SXti , therefore, N(ti 0 ) ⱕ N(ti). Thus, it follows from Eq. (2) that Ntp0 ⱕ Ntp .A Now, let us illustrate how to apply remove_duplicate.
Example 5. Refer to TPGM in Fig. 1. Again, let t1, t2 and t3 denote the subtrees with A, B and C as their roots, respecD tively. It should be noted that SD t1 and St3 occur only once in t1 and t3, respectively. The following is the sequence of restructuring applied to TPGM with the initialization of Y1 to SAt1 in step (1) of remove_duplicate: 1. 2. 3. 4.
Initially set Y1 SAt1 . Then, prune SD Y1 from Y1. Set Y2 to A–a1 –SD . t1 Replace SD t3 with Y2. Delete a1 from Y1 and then replace SAt1 with the newly
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
00 Fig. 3. TPGM after restructuring.
formed Y1 (because Y1 苷 A–a1 and a1 is a terminal node of Y1). The above restructuring steps and the newly formed classification tree T 0 PGM are depicted in Fig. 2. However, a different classification tree may be produced if Y1 is initialized as SCt3 instead of SAt1 in step (1) of remove_duplicate. In this case, the new sequence of restructuring would then be: 1. 2. 3. 4.
Initially set Y1 SCt3 . Then, prune SD Y1 from Y1. . Set Y2 to C–c1 –SD t3 Replace SD t1 with Y2. Delete c1 from Y1 and then replace SCt3 with the newly formed Y1 (because Y1 苷 C–c1 and c1 is a terminal node of Y1).
Fig. 3 depicts the classification tree T 00PGM formed using the above sequence of restructuring. Example 5 shows that remove_duplicate is nondeterministic because of the various ways of initializing p p and NT Y1 in its step (1). Using Eqs. (2)–(4), NT 0 00 PGM
PGM
are both calculated as 45. Thus, either T 0PGM or T 00PGM can be used for constructing the set of all potential test cases. However, in some situations where the restructured classification trees (formed by different initialization of Y1 in remove_duplicate) may have different N p’s, then the restructured classification tree with the smallest N p should be chosen. For Example 5, suppose T 0PGM in Fig. 2 is chosen for constructing the set of all potential test cases. Obviously, it is quite straightforward to construct the corresponding combination table from which all potential test cases can be identified. A close examination of all the 45 potential test cases constructed from T 0PGM , reveals that some illegitimate test cases occur because of the coexistence of (a1 and a2)
771
or (a1 and a3). These types of illegitimacy are not induced from the combination table of TPGM in Fig. 1 because a1, a2 and a3 are under the same top level classification A before restructuring. Therefore, only one of them can be selected at a time to form part of a potential test case. However, a1 and (a2 or a3) are now separately placed under two different top level classifications C and A of T 0PGM , respectively. This makes the selection of (a1 and a2) or (a1 and a3) possible. In fact, for the seven legitimate test cases constructed from TPGM , only one [i.e. the legitimate test case (a3, b1, c2)] of them can be constructed from T 0PGM . The remaining six legitimate test cases are converted into illegitimate ones because of the coexistence of (a1 and a2) or (a1 and a3). In view of this type of illegitimacy caused by restructuring of T, some potential test cases constructed from T 0 should be reformatted using the following algorithm (which is an extended version of the one in [14]) before the validation of their legitimacy against the specification: Test case reformatting algorithm (reformat_test_ case): Refer to remove_duplicate. Again, let T and T 0 denote the classification tree before and after being restructured, respectively. If the condition in step (4.ii) of remove_duplicate is satisfied (thus, SXti in T is replaced with a modified Y1 which does not contain x), then: 1. Let um (m ⱖ 1) be a child class of Li in T such that um is not an ancestor class (the ancestor classes of a classification H is defined as all the classes above H in T) of Y in SYti (of T), and vn (n ⱖ 1) be a child class of X in SXti (of T) such that vn 苷 x. When Li X, um vn. Construct a set of classes g which contains all the classes appearing in every Sutim and every Svtni , of T. 2. For any potential test case constructed from T 0 which contains classes from Y2 in tj 0 of T 0 , reformat this potential test case by deleting all the classes in g from it. else No reformatting is required. Let us illustrate how to apply reformat_test_case.
Example 6. Refer to Figs. 1 and 2. Note ti, tj, X and x in remove_duplicate correspond to t1, t3, A and a1 of TPGM in Fig. 1. It can be seen from Fig. 2 that SAt1 of TPGM is replaced with a modified Y1 in step (4.ii) of remove_duplicate. Additionally, 18 potential test cases [e.g. the potential test case (a1, a2, b1, c1, d1, e1)] contain one or more classes (a1, d1 and d2) selected from Y2 in t3 0 of T 0PGM . Therefore, these potential test cases should be reformatted as follows: 1. In this case, um vn as X Li. The two um’s (or vn’s) in step (1) of reformat_test_case correspond to a2 and a3 of TPGM in Fig. 1. Thus, g {a2, a3, e1, e2}.
772
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
2. Delete a2, a3, e1 and e2 (if any) from the 18 potential test cases which contain one or more classes selected from Y2 in t3 0 of T 0PGM . For example, one of these potential test cases (a1, a2, b1, c1, d1, e1) becomes (a1, b1, c1, d1) after the deletion process. After the application of reformat_test_case, all potential test cases should then be validated against the specification. After the removal of all illegitimate test cases, seven legitimate test cases remain (same as those listed in Section 3). From Eq. (1), ET 0PGM is calculated as 0.16 ( 7=45). Compared with ETPGM 0:09, the improvement is about 78%. For each potential test case constructed from T, there is a corresponding set of paths in T. Let Fti denote the set of those paths appearing in ti of T. For example, in TPGM of Fig. 1, {A–a1 –D–d1} is a Ft1 (denoted as F1). In this example, F1, contains only one path because D is the unique child classification of a1. If, however, a1 has more than one child classification, then F1 would contain more than one path. Now, we are ready to prove the second property, i.e. with the use of remove_duplicate and reformat_test_case, all legitimate test cases constructed from T are preserved in T 0 .
Proposition 3. All legitimate test cases which can be constructed from T can still be constructed from T 0 , after the application of reformat_test_case (i.e. l l 0 NT ). NT
Proof. Again, let us assume that k 2, and the two subtrees (in T) with Li and Lj (1 ⭐ i,j ⭐ 2 and i 苷 j) as their roots are ti and tj, respectively. If a Fti contains any class of SYti , then it is denoted as F1ti . Otherwise, it is denoted as F2ti . Since every potential test case is constructed by combining all the classes which appear in the paths of a Fti and a Ftj , it must belong to one of the following types: 1. One formed by combining all the classes in the paths of a F1ti and a F1tj . 2. One formed by combining all the classes in the paths of a F1ti and a F2tj . 3. One formed by combining all the classes in the paths of a F2ti and a F1tj . 4. One formed by combining all the classes in the paths of a F2ti and a F2tj . Potential test cases of type 1 may be legitimate or illegitimate. To analyse the impact of remove_duplicate on the legitimate test cases of this type, we have to consider the following situations:
• The condition of step (4.i) of remove_duplicate is satisfied (thus, SXti in T is replaced with a null tree) • If (X in ti of T) 苷 Li Although subpaths (which contain classes in SXti of T) of some elements of F1ti are deleted as a result of the replacement of SXti in T with a null tree, these deleted subpaths reappear in Y2 which replaces SYtj in T. Thus, all the potential test cases (and hence the legitimate test cases) of type 1 constructed from T can still be formed from T 0 by combining all the classes appearing in the paths of F2ti 0 and F1tj 0 . • If (X in ti of T) Li It is obvious that SXti ti . Thus, the replacement of SXti in T with a null tree is equivalent to the pruning of ti from T. After the pruning process, all F1ti ’s are removed and T 0 tj 0 . However, all the potential test cases (and hence, the legitimate test cases) of type 1 constructed from T can still be formed from T 0 , because x and SYtj are contained in Y2 which replaces SYtj in T. • The condition of step (4.ii) of remove_duplicate is satisfied (thus, SXti in T is replaced with a modified Y1 which does not contain x) As a result of the replacement of SXti in T with the modified Y1, subpaths (which contain classes in Sxti of T) of some elements of Ft1i will be deleted. However, these deleted subpaths appear in Y2 which replaces SYtj in T. Also, since Sxti (and hence SYti ) does not appear in ti 0 , F1ti 0 Ø. As a result, all classes in the paths of F1tj 0 have to be combined with all classes in the paths of F2ti 0 to form a potential test case. However, any potential test case containing this combination is illegitimate because all the classes in SYtj , (some of these classes must exist in certain elements of F1tj 0 ) cannot coexist with any class from any Sutim0 and Svtni 0 of T 0 (note some classes of Sutim0 or Svtni 0 must appear in some elements of F2ti 0 ). Obviously, any illegitimate test case of this type can be converted into a legitimate one through the removal of all the classes in every Sutim Sutim0 and every Svtni Svtni 0 (i.e. steps (1) and (2) of reformat_test_case). Thus, no legitimate test cases constructed from T will be omitted from T 0 . • The condition of step (4.iii) of remove_duplicate is satisfied (thus, SXti in T is replaced with Y1) Although subpaths (which contain classes in SYti of T) of some elements of F1ti are deleted because of the replacement of SXti in T with Y1, these deleted subpaths reappear in Y2 which replaces SYtj in T. Therefore, all the potential test cases (and hence, the legitimate test cases) of type 1 constructed from T are still being formed from T 0 via the combination of all the classes appearing in the paths of F2ti 0 and F1tj 0 .
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Potential test cases of type 2 are illegitimate because some classes appearing in the paths of F2tj cannot coexist with any class appearing in SYtj . For a similar reason, potential test cases of type 3 are also illegitimate. Finally, let us look at potential test cases of type 4. They remain unchanged after the restructuring process because F2ti and F2tj are left intact, as seen in remove_ duplicate. Therefore, all legitimate test cases constructed from T can still be formed from T 0 after the application of l l 0 NT .A reformat_test_case. As a result, NT 5.2. For partially dependent classifications in different t i’s In general, illegitimate test cases caused by a pair of partially dependent classifications are difficult, if not impossible, to remove, through restructuring of T (with the preservation of all legitimate test cases). However, ET can be improved by restructuring T when the duplicated classifications are under mutually exclusive classes of their respective parent classifications (which are partially dependent on each other). Under this special circumstance, the following restructuring algorithm can be used to improve ET: Tree restructuring algorithm for partially dependent classifications (remove_partial_dependence): Suppose T has k (k ⱖ 2) top level classifications denoted as L1,…,Lk; and ti and tj (1 ⱕ i,j ⱕ k) denote the distinct subtrees (in T) with Li and Lj as their roots, respectively. Also, suppose there exist two identical subtrees SYti and SYtj in T. If: (i) SYti and SYtj occur only once in ti and tj, respectively; (ii) some ancestor classifications (the ancestor classifications of a subtree S or a classification H, is defined as all the classifications above S or H, in T, respectively) M and N of SYti and SYtj , respectively, are partially dependent on each other; and (iii) SYti and SYtj have m (a child class of M) and n (a child class of N) as one of their ancestor classes, respectively, such that m and n cannot coexist (i.e. m and n are mutually exclusive classes), then T can be restructured as shown below: 1. If m is the unique child class of M, then prune SM ti from from T. T. Otherwise, prune Sm ti 2. If n is the unique child class of N, then prune SNtj from T. Otherwise, prune Sntj from T. It should be noted that ti and tj in the restructured classification tree cannot be null. Otherwise, T must be incorrect because no legitimate test cases can be constructed from it. Let T 0 and T 000 denote the classification trees after restructuring T using remove_duplicate and remove_partial_dependence, respectively. We have the following relationships: p p p • NT 000 ⱕ NT 0 ⱕ NT .
773
• All legitimate test cases which can be constructed from l l 000 NT ). T can also be constructed from T 000 (i.e. NT Now, let us prove these relationships.
Proposition 4.
p p p NT 000 ⱕ NT 0 ⱕ NT .
Proof. Refer to remove_duplicate and remove_partial_dependence. Suppose, without loss of generality, that k 2, and the two top level classifications of T are L1 and L2. Let t1 and t2 correspond to ti and tj mentioned in these two algorithms, respectively; t1 0 and t2 0 be t1 and t2, respectively, after restructuring T with remove_duplicate; and t1 000 and t2 000 be t1 and t2, respectively, after restructuring T with remove_partial_dependence. Also, suppose that the parent classes of SYt1 and SYt2 in T are x and w, respectively, and that the parent classifications of x and w are X and W, respectively. It should be noted that X, x, W and w may be M, m, N and n, respectively, as mentioned in remove_partial_dependence. Suppose T 0 is formed with the initialization of Y1 to SXt1 in step (1) of remove_duplicate. Then t1 0 of T 0 is formed by pruning either SXt1 , Sxt1 or SYt1 (corresponding to step (4.i), (4.ii) or (4.iii) of remove_duplicate, respectively) from t1 of T. Now, let us consider t1 000 of T 000 . The subtree t1 000 is t1 m with SM t1 or St1 pruned. Since M X or M is an ancestor classification of X, thus, we have either t1 000 t1 0 or t1 000 is a subtree of t1 0 . Therefore, N t1 000 ⱕ N t1 0 . Now, consider t2 0 and t2 000 . From the proof of Proposition 2, we know that N t2 0 N t2 . Since t2 000 is t2 with SNt2 or Snt2 pruned, t2 000 is in fact a subtree of t2 (hence, N t2 000 ⱕ N t2 ). It follows that N t2 000 ⱕ N t2 0 . p p 0 0 000 and NT Since NT 0 N t1 × N t2 000 N t1 × 000 000 0 N t2 , and we know that N t1 ⱕ N t1 and p p N t2 000 ⱕ N t2 0 , so we must have NT 000 ⱕ NT 0 . From p p p Proposition 2, NT 000 ⱕ NT 0 ⱕ NT immediately follows.A Proposition 4 implies that whenever possible, remove_partial_dependence should be used instead of remove_duplicate for improving ET.
Proposition 5. All legitimate test cases which can be constructed from T can also be constructed from T 000 l l 000 ⱕ NT ). (i.e. NT
Proof. We follow the proof of Proposition 4. For each potential test case constructed from T, let G be its corresponding set of paths in T. Also, let Gt1 and Gt2 denote the sets of those paths appearing in t1 and t2 of T, respectively. If a Gt1 contains m, then it is denoted as G1t1 . Otherwise, it is denoted as G2t2 . Similarly, if a Gt2 contains n, then it is denoted as G1t2 . Otherwise, it is denoted as G2t2 . Since every potential test case is constructed by combining
774
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
Fig. 4. T 000PGM after restructuring.
all classes which appear in the paths of a Gt1 and a Gt2 , it must belong to one of the following types: 1. One formed by combining all the classes in the paths of a G1t1 and a G1t2 . 2. One formed by combining all the classes in the paths of a G1t1 and a G2t2 . 3. One formed by combining all the classes in the paths of a G2t1 and a G1t2 . 4. One formed by combining all the classes in the paths of a G2t1 and a G2t2 . All the potential test cases of type 1 are illegitimate because they contain both m and n, which are mutually exclusive classes. Since SYt1 and SYt2 occur only once in t1 and t2, respectively, Some classes in the paths of G1t1 and G2t1 cannot coexist with some classes in the paths of G2t2 and G1t2 , respectively. Hence, all potential test cases of types 2 and 3 are also illegitimate. For potential test cases of type 4, they may be legitimate or illegitimate. Since G2t1 and G2t2 are invariants with respect to the application of remove_partial_dependence, therefore, all the legitimate test cases constructed from T l l 000 NT ).A are retained in T 000 (i.e. NT Example 7 illustrates the application of remove_partial_dependence and the resultant set of legitimate test cases. Example 7. Refer to TPGM in Fig. 1. Let t1, t2 and t3 denote the subtrees with A, B and C as their roots, respectively. Since: (i) SEt1 and SEt3 occur only once in t1 and t3, respectively; (ii) the parent classifications A and C of SEt1 and SEt3 , respectively, are partially dependent on each other; and (iii) the parent classes a2 and c3 of SEt1 and SEt3 , respectively, are mutually exclusive, we should apply remove_partial_dependence to restructure TPGM in order to improve ETPGM . The restructuring steps are as follows: 1. A and a2 in TPGM correspond to M and m in remove_partial_dependence, respectively. Since a2 is not the unique child class of A, Sat12 is pruned from TPGM.
2. C and c3 in TPGM correspond to N and n in remove_partial_dependence, respectively. Since c3 is not the unique child class of C, Sct33 is pruned from TPGM. 000 after restructuring The resultant classification tree T PGM is depicted in Fig. 4. p 000 of Fig. 4, NT is calculated as 27 using From T PGM 000 PGM Eqs. (2)–(4). Also, the seven legitimate test cases constructed from TPGM can still be formed from T 000PGM. For example, the legitimate test case (a1, b1, c1, d1) constructed from TPGM, can be formed by combining all the classes which lie on P1 000 , P4 000 and P7 000 in Fig. 4. By Eq. (1), ET 000PGM is calculated as 0.26( 7/27). When comparing with ETPGM 0:09, the improvement is about 189%. It should be noted that, in Fig. 4, ET 000PGM can be further improved by restructuring T 000PGM using remove_ duplicate for removing the duplication of the classification D. If this happens, ET 000PGM will be further improved to 0.78 (about 767% improvement).
6. Conclusion As testing plays an important role in the assurance of the correctness (and hence the quality) of software, a systematic way of identifying test cases from specifications is essential. It is because the comprehensiveness of the selected test cases will affect the quality of testing. In view of this, the classification-tree method developed by Grochtmann and Grimm [5] provides a useful direction in pursuit of testing. Unfortunately, their tree construction method is rather ad hoc. This problem has motivated Chen and Poon [12, 13] to develop the classification-hierarchy table and its associated tree construction algorithm, by which classification trees can be constructed systematically from specifications. However, their construction algorithm does not take into consideration the effectiveness of classification trees. In this paper, we have introduced a metric ET to measure the effectiveness of a classification tree. Intuitively speaking,
T.Y. Chen, P.L. Poon / Information and Software Technology 40 (1998) 765–775
ET is a measure of the cost-effectiveness of constructing the set of legitimate test cases from a classification tree. Simply knowing the value of ET for a classification tree is insufficient. A more important issue is how to improve ET. To address this issue, this paper has investigated the relationship between the structure of a classification tree and its ET value, focusing on those features which could result in a poor ET value. We observe that there are two sources leading to a poor ET value, the existence of duplicated classifications under different top level classifications, and partially dependent classifications. From this observation, we have proposed two algorithms to improve ET through restructuring the classification trees, prior to the construction of potential test cases. References [1] R. Ferguson, B. Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology 5 (1) (1996) 63–86. [2] B. Korel, Automated test data generation for programs with procedures, in: ISSTA’96: Proceedings of the 1996 International Symposium on Software Testing and Analysis, January 1996, pp. 209-215. [3] J. Sanders, E. Curran, Software Quality: A Framework for Success in Software Development and Support. Addison-Wesley, Reading, MA, 1994. [4] T. Chusho, Test data selection and quality estimation based on the concept of essential branches for path testing, IEEE Transactions on Software Engineering 13 (5) (1987) 509–517.
775
[5] M. Grochtmann, K. Grimm, Classification trees for partition testing, Software Testing, Verification and Reliability 3 (1993) 63–82. [6] T.J. Ostrand, M.J. Balcer, The category-partition method for specifying and generating functional tests, Communications of the ACM 31 (6) (1988) 676–686. [7] T.Y. Chen, M.F. Lau, Two test data selection strategies towards testing of boolean specifications, in: COMPSAC’97. Proceedings of the Twenty-First Annual International Computer Software and Applications Conference, IEEE Computer Society Press, August 1997, pp. 608–611. [8] J.J. Chilenski, S.P. Miller, Applicability of modified condition/decision coverage to software testing, Software Engineering Journal 9 (5) (1994) 193–200. [9] E.J. Weyuker, T. Goradia, A. Singh, Automatically generating test data from a boolean specification, IEEE Transactions on Software Engineering 20 (5) (1994) 353–363. [10] R. Weber, EDP Auditing Conceptual Foundations and Practice, 2nd ed., McGraw-Hill, New York, 1988. [11] T.Y. Chen, P.L. Poon, S.F. Tang, A systematic method for auditing user acceptance tests, IS Audit and Control Journal 5 (1998) 31– 36. [12] T.Y. Chen, P.L. Poon, Classification-hierarchy table: a methodology for constructing the classification tree, in: ASWEC’96: Proceedings of the Australian Software Engineering Conference, IEEE Computer Society Press, July 1996, pp. 93–104. [13] T.Y. Chen, P.L. Poon, Construction of classification trees via the classification-hierarchy table, Information and Software Technology 39 (13) (1997) 889–896. [14] T.Y. Chen, P.L. Poon, Improving the quality of classification trees via restructuring, in: APSEC’96: Proceedings of the Asia-Pacific Software Engineering Conference, IEEE Computer Society Press, December 1996, pp. 83–92.