Model discrimination—another perspective on model-robust designs

Model discrimination—another perspective on model-robust designs

Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583 www.elsevier.com/locate/jspi Model discrimination—another perspective on model-r...

157KB Sizes 0 Downloads 64 Views

Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583 www.elsevier.com/locate/jspi

Model discrimination—another perspective on model-robust designs Bradley A. Jonesa , William Lib,∗ , Christopher J. Nachtsheimb , Kenny Q. Yec a SAS Institute, SAS Campus Drive, Cary, NC 27513, USA b Operations and Management Science Department, University of Minnesota, Minneapolis, MN 55455, USA c Department of Epidemiology and Population Health, Albert Einstein College of Medicine,1300 Morris Park Ave, Bronx, NY 10461, USA

Available online 6 October 2006

Abstract Recent progress in model-robust designs has focused on maximizing estimation capacities. However, for a given design, two competing models may be both estimable and yet difficult or impossible to discriminate in the model selection procedure. In this paper, we propose several criteria for gauging the capability of a design for model discrimination. The criteria are then used to evaluate a class of 18-run orthogonal designs in terms of their model-discriminating capabilities. We demonstrate that designs having the same estimation capacity may differ considerably with respect to model-discrimination capabilities. The best designs according to the proposed model-discrimination criteria are obtained and tabulated for practical use. © 2006 Elsevier B.V. All rights reserved. Keywords: Estimation capacity; Information capacity; Model discrimination; Model-robust design

1. Introduction In recent years, many authors evaluated designs from the perspective of model estimation. Consider a model space F = {f1 , . . . , fd } of d candidate linear models, where the functional f indicates which effects are present in the model. Given a factorial design, their focus has been on the two types of criteria: 1. The percentage of models in F that are estimable. 2. The average estimation efficiency of all models in F. Consideration of the first aspect motivated Sun (1993) to propose the criterion of estimation capacity (EC), which is defined as the ratio of the number of estimable models to the total number of models in the model space F. Early work on the second aspect can be traced back to Läuter (1974), who proposed maximizing the average of the log of the determinant over F. The same problem was considered by Cook and Nachtsheim (1982), who proposed the use

∗ Corresponding author.

E-mail address: [email protected] (W. Li) URL: http://www.csom.umn.edu/∼wli (W. Li). 0378-3758/$ - see front matter © 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.jspi.2006.09.006

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

1577

of linear optimality criteria such as the trace of the integrated variance. In the context of the two-level factorial designs, Sun (1993) proposed the criterion of information capacity (IC), which amounts to the average D-efficiency over all estimable models. Li and Nachtsheim (2000) modified the definition of IC to be the average D-efficiency over all models ¯ in F. Another criterion proposed by Li and Nachtsheim (2000) is the D-criterion, which considers the D-efficiency relative to the exact D-optimal design for each model. Li and Nachtsheim (2000) defined a model-robust design to be the one that sequentially maximizes EC and IC. That is, they first search for designs maximizing the EC. Ideally, these designs are those satisfying EC = 100%. Then from these designs, they select the one that maximizes the IC-criterion. By definition, model-robust designs have high EC and high estimation efficiency. However, there is another important aspect of model-robust designs which was not considered in Li and Nachtsheim (2000), namely, how well can estimable models be distinguished from each other? In general, given a design D, consider two models f1 and f2 with model matrices X1 and X2 . If the hat matrix of X1 , H1 = X1 (X1T X1 )−1 X1T is equal to the hat matrix of X2 , H2 = X2 (X2T X2 )−1 X2T , then their predictions yˆ 1 = H1 y and yˆ 2 = H2 y are the same on design D regardless of the values of the response y. Hence, these two models cannot be distinguished in the design. We call these two models fully aliased in design D. It is easy to provide examples of designs that have fully aliased models. Consider, for example, the 24−1 fractional factorial design defined by I = 1234. For the model space F comprised of all main effects and two two-factor interactions, the models 1 + 2 + 3 + 4 + 12 + 13 and 1 + 2 + 3 + 4 + 12 + 24 are both estimable and fully aliased. Although the issue of model discrimination has not been a focus in the recent development of model-robust designs, considerable work in the development of experimental designs for discriminating among regression model has been done. See, for example, Hunter and Reiner (1965), Box and Hill (1967), Hill et al. (1968), Atkinson and Cox (1974), Atkinson and Fedorov (1975a, b). A comprehensive review of early contributions is given by Hill (1978). The emphasis in these early studies is different from ours in two aspects. First, these authors focused on designs that best discriminate between two or among several nonlinear rival models. Second, these studies frequently employed sequential experimental design techniques. For model-robust designs, we consider only linear regression models and assume that a single experiment is to be conducted. Additionally, since our purpose is factor screening, we focus on finding designs such that candidate models are minimally aliased. Recently, Meyer et al. (1996) used a Bayesian criterion to choose follow-up runs after a factorial design to dealias candidate models. Their criterion, based on Kullback–Leibler information, can be traced back to Box and Hill (1967). Bingham and Chipman (2002) proposed another Bayesian criterion for choosing optimal designs for model selection. Their criterion is based on Hellinger distance between predictive densities. Both Bayesian criteria require specification of a prior distribution for the model coefficients and the errors, and consequently require intensive computation. Non-Bayesian criteria for model discrimination have also been addressed in the literature, most notably the T-optimality criterion proposed by Atkinson and Federov (1975a, b). The T-optimal criterion assumed that the true model is known as well as its parameters. More practical versions were also given in their papers including sequential design procedures and a Bayesian version. What is common to all of these criteria is that they are based on differences in the predictions that arise from all pairs of candidate model. This is also the basis of the criteria we propose in this article. Another paper deserving mention here is Srivastava (1975), which proposed a criterion, called resolving power, to measure the model-discrimination capability of a design. The purpose of his work is to find optimal “model search designs” that have capability to both estimate and discriminate a set of candidate models. His criterion, however, depends on the assumption that the noise in the linear models is negligible, and is therefore not applicable to the situations envisioned here. Moreover, his criterion does not quantitatively measure the degree of model aliasing. The problem of model aliasing was also noticed by Miller and Sitter (2001) in discussing foldover designs, but they did not deal with the problem directly. In the next section, we first propose three measures of model aliasing between two linear models. These measures are non-Bayesian and can be calculated quickly and easily using standard matrix routines. We will also discuss their statistical justification in detail. These measures are then extended to design criteria that account for model-discrimination capability of factorial designs. Section 3 will first discuss how model-discrimination criteria and EC criteria are combined to select optimal model-robust designs. We then combine these criteria to select model-robust 18-run orthogonal designs. The results show that model-discrimination capability is an important aspect of model-robust designs. Concluding remarks are given in Section 4.

1578

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

2. Design criteria for model discrimination In this section, we will first discuss three non-Bayesian measures of model aliasing: subspace angle (SA), maximum prediction difference (MPD), and expected prediction difference (EPD). We then develop design criteria based on those measures. 2.1. Measures of aliasing between two models 2.1.1. SA As discussed in the previous session, two models are fully aliased in a design when the hat matrices, H1 and H2 , of the model matrices X1 and X2 are the same. Equivalently, it follows that two models are fully aliased when the linear spaces V (X1 ) and V (X2 ) spanned by columns of model matrices X1 and X2 are the same. To measure the degree of model aliasing, it is natural to consider a geometrical measure for the “closeness” between the two model spaces V (X1 ) and V (X2 ). One such measure is the SA, which is a generalization of the angle between two planes in a three-dimensional Euclidean space   a12 = max min arccosv1 , v2  . (1) v1 ∈V (X1 )

v2 ∈V (X2 )

Note that for a given vector v1 ∈ V (X1 ), the vector in V (X2 ) with which v1 has the least inner-product is its projection on V (X2 ). Therefore, (1) is equivalent to a12 = max

v1 ∈V (X1 )

arccosv1 , H2 v1 .

(2)

Note that in (2), the value of the SA remains the same if v1 is restricted to unit vectors, i.e. v1  = 1. Then the SA can be interpreted geometrically as the arcsin value of the largest possible Euclidean distance between a unit vector in one subspace and its projection to the other. In terms of linear regression model, the SA can be interpreted as follows. Let f1 be the true model with the model matrix X1 and assume that the noise term is negligible. The observed response y is then obtained in V (X1 ). Let f2 be another model and yˆ 2 be its fitted value. The SA a12 is the arcsin of the largest possible value of y − yˆ 2 , the L2 -distance between a normalized response y and its corresponding fit of the other model. That is, a12 = arcsin

max

y=1,y∈V (X1 )

y − yˆ 2 .

(3)

Since arcsin is strictly monotonic, the larger the SA between V (X1 ) and V (X2 ), the larger the distance between a normalized response y ∈ V (X1 ) and its fitted value on yˆ 2 = H2 y ∈ V (X2 ). Note that a12 = a21 . Hence, this measure does not require the specification of which model is considered true. Following the implementation of function SUBSPACE of MATLAB, the computation of the SA can be done very efficiently through QR decomposition and single value decomposition (SVD). Let X1 = Q1 R1 and X2 = Q2 R2 , where Ri are triangular matrices and QTi Qi = I. The SA between V (X1 ) and V (X2 ) is then a12 = arccos smin , where smin is the minimum singular value of P = QT1 Q2 . If the two subspaces are the same, then smin = 1 and the SA reaches its minimum a12 = 0. If there exist two vectors in each subspace such that they are orthogonal, then smin = 0 and the SA reaches its maximum a12 = /2. Since arccos is strictly monotonic, in searching for a good design based on SA, one only needs smin without taking the extra step to get values of a12 . 2.1.2. MPD Consider two model matrices, X1 and X2 . Let Hi denote their corresponding hat matrices. The difference between the predictions yˆ 1 − yˆ 2 = (H1 − H2 )y = 0 for any response y if H1 − H2 = 0. A reasonable measure for aliasing of two models is the maximum differences between the predictions of two models over all normalized responses. That is, the MPD: max ˆy1 − yˆ 2 2 = max yT (H1 − H2 )(H1 − H2 )y.

y=1

y=1

(4)

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

1579

To compute the above criterion, one only needs to find the maximum absolute eigenvalue of H1 − H2 , denoted by ||max . Since H1 − H2 is symmetric, this is also equivalent to the maximum absolute singular value of H1 − H2 . In fact, it is not difficult to show the following, the proof of which is contained in the appendix. Theorem 1. The MPD between two linear models as defined in (4) is no larger than 1. Notice that MPD and SA are closely related. As shown in (4), MPD allows response y to be any normalized vectors in Rn . In contrast, the SA measure given in (3) restricts y to be a normalized vector in the subspace V (X1 ). Notice that yˆ 1 = y for y ∈ V (X1 ), the value of SA is never larger than arcsin(maxy=1 ˆy1 − yˆ 2 ) since the latter does not restrict the response to a subspace. 2.1.3. EPD The MPD measure considers only the best possible separation between two models. It is natural to consider a measure that averages over all normalized responses. The EPD is defined as E(ˆy1 − yˆ 2 2 | y = 1) = E(yT Dy | y = 1), where D = (H1 − H2 )(H1 − H2 ). It is the expected value of  difference between two predictions where the response y is uniformly distributed over the unit n-sphere. Notice that y=1 yi yj dy = 0 for i  = j , so that EPD can be calculated as follows: E(ˆy1 − yˆ 2 2 | y = 1) = E(yT Dy | y = 1)     1 1 T 2 y Dy dy =  y1 dy · Trace(D) = y=1 dy y=1 y=1 dy y=1 1 = Trace(D). (5) n    The last equality in (5) is true because y=1 y12 dy = (1/n) y=1 (y12 + y22 + · · · + yn2 ) dy; hence, (1/ y=1 dy)  2 y=1 y1 dy = 1/n. Therefore, EPD has a clear advantage over MPD and SA because it is much easier to compute. In general, all three measures are much easier to compute than the Bayesian measures used by Meyer et al. (1996) and Bingham and Chipman (2002). Although all three measures have clear statistical interpretations, we slightly favor EPD over the other two because it is easier to compute, and it is a less conservative measure. 2.2. Design criteria for model discrimination Each of the three model aliasing measures discussed in this section can be extended to a design criterion that measures the overall model-discrimination capability of a design. Consider the model space F = {f1 , . . . , fd }. The model aliasing  d measures can be applied to all r = 2 model pairs. We now define six model-discrimination criteria for a design over the model space F, each of which takes either the minimum or the average of one of previous three model aliasing measures over all r model pairs. Consider all model pairs fi and fj , whose corresponding model matrices are, respectively, given by Xi and Xj . Then the six criteria are given as follows: (minimum SA)MSA = min SA{V (Xi ), V (Xj )}, 1

(average SA)ASA = SA{V (Xi ), V (Xj )}, r (minimum MPD)MMPD = min MPD{fi , fj }, 1

(average MPD)AMPD = MPD{fi , fj }, r (minimum EPD)MEPD = min EPD{fi , fj }, 1

(average EPD)AEPD = EPD{fi , fj }. r

(6) (7) (8) (9) (10) (11)

In all six criteria, larger values of a design indicate better model-discrimination capabilities. In the next section, we will demonstrate how these criteria can be combined with the EC and IC criteria to select model-robust designs. For

1580

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

simplicity, we will focus on three of them: MSA, MMPD, and AEPD. We will also calculate and report the IC-criterion of Li and Nachtsheim (2000), which is the average D-efficiency over all models. 3. Orthogonal 18-run model-robust designs Given a set of candidate designs and a set of candidate models, we propose a three-step procedure to select optimal model-robust designs as follows: Step 1: We evaluate designs in terms of the EC criterion. In particular, we identify designs having EC = 100%. We call these designs model-robust designs. Step 2: Among the designs identified in step 1, we further screen out designs for which MMPD = 0. Note that for such designs, there exists at least one pair of models that is fully aliased. Step 3: Among the designs selected in steps 1 and 2, we search for designs that are effective in terms of both IC and model-discrimination criteria. The above procedure can be applied to general cases as long as a set of candidate designs and the model space are given. For illustration, we consider a class of 18-run orthogonal designs with three-level factors. The candidate designs are non-isomorphic projections of the L18 orthogonal array listed in Table 1. For three-level designs with quantitative factors, Cheng and Ye (2004) pointed out that permuting the levels of one or more factors might generate designs with different geometric structures. In that paper, -word length pattern (-WLP) is proposed to rank multi-level factorial designs with quantitative factors. Two designs with different -WLPs are geometrically non-isomorphic (Cheng and Ye, 2004), but two geometrically non-isomorphic designs might have the same -WLP. The -WLP criterion can be used to classify all k-column projections (with level-permutation) of the L18 array listed in Table 1 to sets of geometrically non-isomorphic designs. Cheng and Ye (2004) obtained such designs for k = 3 and 4. In this paper, we extended their results and obtained geometrically non-isomorphic designs for k = 5, 6, and 7 using the same methods. The numbers of these designs are summarized in Tables 2 and 3. We will now find model-robust designs from these candidate designs by considering both model-estimation and model-discrimination properties. Consider two model spaces F1 and F2 , which consist of models containing all main effects of k factors plus one or two two-factor (linear-by-linear) interactions, respectively. The main effects and the interaction effects are obtained under the linear-quadratic system described in Wu and Hamada (2000). Table 2 lists the numbers of model-robust designs, for which EC = 100%, for models spaces F1 and F2 . We first note that no L18 (37 ) designs in our candidate set have EC = 100% for either F1 or F2 . In fact, among 29 candidate designs, the largest EC values are only 61.9% for F1 and 30.5% for F2 . Thus, we do not recommend use of these designs for the model space F1 or F2 . Table 1 L18 (37 ) orthogonal array 1

2

3

4

5

6

7

0 0 0 1 1 1 2 2 2 0 0 0 1 1 1 2 2 2

0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2

0 1 2 0 1 2 1 2 0 2 0 1 1 2 0 2 0 1

0 1 2 1 2 0 0 1 2 2 0 1 2 0 1 1 2 0

0 1 2 1 2 0 2 0 1 1 2 0 0 1 2 2 0 1

0 1 2 2 0 1 1 2 0 1 2 0 2 0 1 0 1 2

0 1 2 2 0 1 2 0 1 0 1 2 1 2 0 1 2 0

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

1581

Table 2 Summary of L18 (3k ) designs for model robustness k

# of designs

n1

n2

3 4 5 6 7

8 21 43 41 29

7 21 43 16 0

6 18 22 0 0

n1 and n2 are numbers of model-robust designs (for which EC =100% for model spaces F1 and F2 , respectively).

Table 3 Numbers of model-robust L18 (3k ) designs, for which MMPD > 0 k

F1

F2

3 4 5 6 7

7 18 23 0 —

6 15 9 — —

Table 4 Summary of model-discriminating L18 (3k ) designs k

3 4 5

Design # in catalogue

3.1 3.4 4.2 4.5 5.29 5.25

F1

F2

MSA

MMPD

AEPD

IC

MSA

MMPD

AEPD

IC

1.427 1.318 1.318 1.206 .927 .795

.9897 .9682 .9682 .9340 .8000 .7141

.1088 .1088 .1065 .1049 .1005 .0967

.9834 .9945 .9716 .9767 .9389 .9616

1.446 1.318 1.231 1.126 .652 .762

.9922 .9682 .9428 .9025 .6070 .6901

.1094 .1088 .1453 .1434 .1468 .1423

.9685 .9878 .9454 .9529 .8826 .9192

Designs with high EC do not necessarily have good model-discrimination properties. For example, consider the L18 (35 ) designs for F2 . Among the 43 non-isomorphic designs, 22 have EC = 100%, as shown in Table 2. However, 13 out of 22 designs have MMPD = 0, i.e. at least one pair of candidate models are fully aliased. Thus, from the perspective of model discrimination, these 13 designs are undesirable. Table 3 lists the numbers of designs for which EC = 100% and MMPD > 0. It can be seen that the numbers of designs that can be used for the purpose of model discrimination are relatively small, compared to the total numbers of designs with EC = 100% listed in Table 2. In particular, when k = 5, only 23 out of the 43 designs having EC = 100% satisfy MMPD > 0 for F1 ; and 9 of the 22 designs having EC = 100% satisfy MMPD > 0 for F2 . When k = 6, none of the 16 designs having EC = 100% satisfy MMPD > 0 for F1 . Thus, we recommend that the L18 (36 ) designs not be used for either F1 and F2 . Finally, we evaluate the remaining designs that satisfy both (1) EC = 100% and (2) MMPD > 0. For each k = 3, 4, 5 we select two effective designs that have good model-discrimination capabilities and high IC values for both F1 and F2 . The results are summarized in Table 4, which shows four criterion values—MSA, MMPD, AEPD and IC of the selected designs. The catalogues of all L18 (3k ) designs mentioned in Table 2 are provided at web site www.csom.umn.edu/∼wli: for k = 3, the first design (design #3.1 in the catalogue) in Table 4 corresponds to the one that has the best modeldiscrimination value among eight candidate designs. It has the largest MSA values—1.427 for F1 and 1.446 for F2

1582

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

according to Table 4. It also has the largest MMPD and AEPD values for both F1 and F2 . The second design (design #3.4 in the catalogue) corresponds to the one having the highest IC values—.9945 for F1 and .9878 for F2 . The two designs are chosen similarly for k = 4: the first (design #4.2 in the catalogue of 21 candidate designs) has the best MSA, MMPD, and AEPD values for both F1 and F2 , and the second (design #4.5 in the above catalogue) has the best IC value for both F1 and F2 . The case for k = 5 is more complex for two reasons. First, the best design according to one model-discrimination criterion may not be the best with respect to another. For instance, design #5.25 (in the catalogue of 43 candidate designs) has (MMPD, AEPD) = (.6901, .1423) for model space F2 . In comparison, design #5.29 in the same catalogue has (MMPD, AEPD) = (.6070, .1468) for model space F2 . The former is MMPD-optimal, and the latter is AEPD-optimal. The second reason is that the best design for F1 may not be optimal for F2 . For instance, design #5.25 is MMPDoptimal for F2 , but design #5.29 is MMPD-optimal for F1 . The results of these two designs are given in Table 4. Notice that design #5.25 is also IC-optimal for both F1 and F2 . We conclude the discussion by noting different model-robust designs (for which EC = 100%) may have similar IC but very different model-discrimination properties. Consider, for example, the MSA criterion for the case of k = 5 and F2 . Among nine model-robust designs, the MSA values range from .289 (design #5.8 in the catalogue) to .762 (design #5.25). The IC values of these designs are similar—.8927 for the former and .9192 for the latter. This demonstrates that model-discrimination criteria proposed in this paper reflect another important perspective of model-robust designs. 4. Concluding remarks In this paper we extend the work on model-robust designs from EC and IC, as shown in Sun (1993) and Li and Nachtsheim (2000) from the perspective of model discrimination. We see an analogy with the extension of the maximum resolution criterion (Box and Hunter, 1961) to the minimum aberration criterion (Fries and Hunter, 1980) for orthogonal designs. Because designs with the same resolution may have different WLPs, the minimum aberration criterion was proposed to further characterize or discriminate between fractional factorial designs. In selecting model-robust designs, we note that designs having the same EC may differ significantly in terms of their model-discrimination capabilities. In this paper, we focus on designs having EC = 100%. If none of the candidate designs satisfy EC = 100%, the proposed model-discrimination criteria can still be used to select among designs having high EC values. Note that the model-aliasing measures make little sense when the two models are not both estimable. Model-discrimination criteria only characterize aliasing among estimable models. Acknowledgments This research was supported by the Supercomputing Institute for Digital Simulation and Advanced Computation at the University of Minnesota. The research of William Li and Christopher J. Nachtsheim is supported by the Research and Teaching Supplements System in Carlson School of Management at the University of Minnesota. The research of Kenny Q. Ye is supported in part by National Science Foundation grant DMS-0306306. We thank the editor and two referees for the helpful comments. Appendix Proof of Theorem 1. For a fixed response y ∈ Rn such that y = 1, consider the sphere in Rn with the center at y/2 and the diameter 1. Now consider the distance between yˆ and y/2. Knowing that y − yˆ and yˆ are orthogonal, we can show that ˆy − y/2 = 1/2 as follows: ˆy − y/2 = 2ˆy − y/2 = (ˆy − y) + yˆ /2   2 2 2 = 21 . ˆy − y + ˆy =

(12)

Note that the last two equalities in (12) use the Pythagorean theorem. The above equation shows that both yˆ 1 and yˆ 2 are on the sphere. Geometrically, the distance between any two points on a sphere can be no greater than the diameter. The proof is now completed. 

B.A. Jones et al. / Journal of Statistical Planning and Inference 137 (2007) 1576 – 1583

1583

References Atkinson, A.C., Cox, D.R., 1974. Planning experiments for discriminating between models. J. Roy. Statist. Soc. Ser. B 36, 321–348. Atkinson, A.C., Fedorov, V.V., 1975a. The design of experiments for discriminating between two rival models. Biometrika 62, 57–70. Atkinson, A.C., Fedorov, V.V., 1975b. Optimal design: experiments for discriminating between two several models. Biometrika 62, 289–303. Bingham, D.R., Chipman, H.A., 2002. Optimal designs for model selection. Technometrics, to appear. Box, G.E.P., Hill, W.J., 1967. Discrimination among mechanistic models. Technometrics 9, 57–71. Box, G.E.P., Hunter, J.S., 1961. The 2k−p fractional factorial designs. Technometrics 3, 311–351, 449–458. Cheng, S.W., Ye, K.Q., 2004. Geometric isomorphism and minimum aberration for factorial designs with quantitative factors. Ann. Statist. 32, 2168–2185. Cook, R.D., Nachtsheim, C.J., 1982. Model robust, linear-optimal designs. Technometrics 24, 49–54. Fries, A., Hunter, W.G., 1980. Minimum aberration 2k−p designs. Technometrics 22, 601–608. Hill, P.D.H., 1978. Experimental design procedures for regression model discrimination. Technometrics 20, 15–21. Hill, W.J., Hunter, W.G., Wichern, D.W., 1968. A joint design criterion for the dual problem of model discrimination and parameter estimation. Technometrics 10, 145–160. Hunter, W.G., Reiner, A.M., 1965. Designs for discriminating between two rival models. Technometrics 7, 307–323. Läuter, E., 1974. Experimental planning in a class of models. Math. Oper. Statist. 5, 673–708. Li, W., Nachtsheim, C.J., 2000. Model-robust factorial designs. Technometrics 42, 345–352. Meyer, R.D., Steinberg, D.S., Box, G.E.P, 1996. Follow-up designs to resolve the confounding in multifactor experiments. Technometrics 38, 303–313. Miller, A., Sitter, R.R., 2001. Using the folded-over 12-run Plackett-Burman designs to consider interactions. Technometrics 43, 44–55. Srivastava, J.N., 1975. Designs for searching non-negligible effects. In: Srivastava, J.N. (Ed.), A Survey of Statistical Designs and Linear Models. North-Holland, Amsterdam. Sun, D.X., 1993. Estimation capacity and related topics in experimental designs. Unpublished Ph.D. Dissertation. Department of Statistics and Actuarial Science, University of Waterloo, Canada. Wu, C.F.J., Hamada, M., 2000. Experiments: Planning, Analysis and Parameter Design Optimization. Wiley, New York.