Experimental design influences on stated choice outputs: An empirical study in air travel choice

Transportation Research Part A 45 (2011) 63–79

Contents lists available at ScienceDirect

Transportation Research Part A journal homepage: www.elsevier.com/locate/tra

Experimental design influences on stated choice outputs: An empirical study in air travel choice Michiel C.J. Bliemer a,b,⇑, John M. Rose b,1 a b

Delft University of Technology, Department of Transport and Planning, Faculty of Civil Engineering and Geosciences, P.O. Box 5048, 2600 GA Delft, The Netherlands The University of Sydney, Institute of Transport and Logistics Studies, Faculty of Economics and Business, NSW 2006, Australia

a r t i c l e

i n f o

Article history: Received 9 February 2010 Received in revised form 12 August 2010 Accepted 17 September 2010

Keywords: Experimental design Stated choice Air travel Orthogonal design D-efficient design Sample size

a b s t r a c t Discrete choice experiments are conducted in the transport field to obtain data for investigating travel behaviour and derived measures such as the value of travel time savings. The multinomial logit (MNL) and other more advanced discrete choice models (e.g., the mixed MNL model) have often been estimated on data from stated choice experiments and applied for planning and policy purposes. Determining efficient underlying experimental designs for these studies has become an increasingly important stream of research, in which the objective is to generate stated choice tasks that maximize the collected information, yielding more reliable parameter estimates. These theoretical advances have not been rigorously tested in practice, such that claims on whether the theoretical efficiency gains translate into practice cannot be made. Using an extensive empirical study of air travel choice behaviour, this paper presents for the first time results of different stated choice experimental design approaches, in which respective estimation results are compared. We show that D-efficient designs keep their promise in lowering standard errors in estimating, thereby requiring smaller sample sizes, ceteris paribus, compared to a more traditional orthogonal design. The parameter estimates found using an orthogonal design or an efficient design turn out to be statistically different in several cases, mainly attributed to more or less dominant alternatives existing in the orthogonal design. Furthermore, we found that small designs with a limited number of choice tasks performs just as good (or even better) than a large design. Finally, we show that theoretically predicted sample sizes using the so-called S-estimates provide a good lower bound. This paper will enable practitioners in better understanding the potential benefits of efficient designs, and enables policy makers to make decisions based on more reliable parameter estimates. Ó 2010 Elsevier Ltd. All rights reserved.

1. Introduction 1.1. Choice experiments Discrete choice experiments (DCE) have grown to become the primary source of data for obtaining estimates of behavioural importance such as consumer preferences for various transport goods and services or willingness to pay (WTP) measures for specific attributes such as travel time savings. In a discrete choice experiment, a respondent is being asked in one or multiple choice tasks to select their most preferred alternative from a given set of alternatives that are characterized by ⇑ Corresponding author at: Delft University of Technology, Department of Transport and Planning, Faculty of Civil Engineering and Geosciences, P.O. Box 5048, 2600 GA Delft, The Netherlands. Tel.: +31 15 2784874; fax: +31 15 2783179. E-mail addresses: [email protected] (M.C.J. Bliemer), [email protected] (J.M. Rose). 1 Tel.: +61 2 93510168; fax: +61 2 93510088. 0965-8564/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.tra.2010.09.003

64

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

(hypothesized) levels of attributes. Using these choice observations, the analyst tries to capture preferences for and tradeoffs between attributes (e.g., toll and travel time). As an example, consider the very simple choice experiment depicted in Fig. 1 in which the respondent has to choose between two alternative modes of transport, namely car and train. Each alternative is characterized by only two attributes, travel time and travel cost. In the experiment we aim to find preferences for these attributes by estimating the following two utility functions in a logit model setting:

V car;s ¼ btime TT car;s þ bcost TC car;s ;

ð1Þ

V train;s ¼ btrain þ btime TT train;s þ bcost TC train;s ;

where Vm,s is the systematic utility for mode m in choice task s, TTm,s is the level of the travel time for mode m in choice task s, TCm,s is the level of the travel cost for mode m in choice task s, and btrain, btime and bcost are unknown (preference) parameters that are to be estimated. In this example we assume that all respondents face the same choice tasks, however, this could be extended to respondent specific choice tasks. If we assume that both alternatives have identically and independently extreme value type I distributed random unobserved components, the probability Pm,s of choosing mode m in choice task s, given certain travel times and costs, is given by the following multinomial logit (MNL) model (see McFadden, 1974):

expðV m;s Þ Pm;s ¼ P : expðV m0 ;s Þ

ð2Þ

m0

In order to estimate btrain, btime and bcost (typically through maximum likelihood estimation), choices associated with choice tasks such as in Fig. 1 are needed. In a stated choice context, it is up to the analyst to develop such choice tasks. The set of choice tasks (which constitutes the experimental design) affects the parameter estimates and their reliability, hence constructing these experimental designs should be done carefully. Over time, as the discrete choice modelling literature has matured, a number of econometrically more advanced models able to uncover an increasing degree of behavioural richness have been developed, typified by the rapid progression from simple MNL model, to nested logit (NL), cross-correlated NL and mixed multinomial logit (MMNL) models (see e.g., Train, 2009). At the same time, advancements in the construction of experimental designs that underlie DCE have been limited and somewhat more erratic in terms of acceptance within the wider literature. This is not to suggest however that advancements have not been made. Unfortunately, whilst information related to which alternatives, attributes and attribute levels to use may come from secondary data sources or qualitative research such as focus groups and in-depth interviews, the precise method used to construct the underlying experimental design remain solely at the discretion of the researcher. Whilst there exist multiple possible construction methods, each of which make different assumptions during the design generation process, unlike estimation problems where a number of texts describe the advantages and disadvantages of various models (see e.g., Train, 2009), there regrettably exists little guidance as to which particular method to select when generating an experimental design for DCE type studies. A poor choice of experimental design, or one based on an inadequate or incorrect set of assumptions, may result in poor data quality. In turn, poor quality data may result in erroneous conclusions being reached, or at the very least, less reliable parameter estimates at a given sample size. At issue however, is that the advancements that have been made in relation to experimental design construction for DCEs have largely been theoretical in nature, with little empirical work demonstrating whether the theoretical advantages of the various identified design strategies actually translate into practice.

Experimental design car

Choice task 1 (of 9)

train

car

train

s

TT

TC

TT

TC

Travel time

10 min.

15 min.

1

10

1.00

15

0.50

Travel cost

1.00 euro

0.50 euro

2

20

1.50

20

0.50

3

15

2.00

25

0.50

4

15

1.50

15

1.00

5

10

2.00

20

1.00

6

20

1.00

25

1.00

7

20

2.00

15

1.50

8

15

1.00

20

1.50

9

10

1.50

25

1.50

Your choice:

Fig. 1. Example of a stated choice experiment.

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

65

1.2. Orthogonal designs As such, the most widely used type of design for DCEs remain what we term orthogonal fractional designs or simply orthogonal designs (see also Section 2.1). The orthogonality (or otherwise) of an experimental design relates to the correlation structure between the attributes of the design with designs in which all between attribute correlations are zero being said to be orthogonal. In other words, orthogonal designs ensure that the attribute levels are nicely spread over all choice tasks, and that attribute level combinations do not exhibit a certain (positively or negatively correlated) pattern. In some cases, this definition of an orthogonal design may be relaxed to define orthogonality as occurring when all attribute correlations are zero within alternatives but not necessarily between alternatives; see Louviere et al. (2000) for a discussion on sequential versus simultaneous generation of orthogonal designs. As an example, the experimental design in Fig. 1 is orthogonal (in the main effects), as each column of attribute levels is independent (uncorrelated) of any other column in the design. That is, corr(TTcar, TCcar) = 0, corr(TTcar, TTtrain) = 0, corr(TTcar, TCtrain) = 0, and corr(TTtrain, TCtrain) = 0. The popular use of orthogonal designs stems firstly from historical impetus, and secondly from a lack of guidance within the literature as to whether such designs represent best practice. In relation to historical impetus, experimental design theory as related to early DCE was simply borrowed from existing experimental design theory dealing primarily with linear models, in particular, ANOVA and regression type models (see e.g., Louviere and Woodworth, 1983). This resulted in the large scale use of orthogonal designs which continued unquestioned for many years to come. The lack of empirical evidence for or against the use of such designs stems from the typical use within study of a single experimental design. The failure to vary different experimental designs, whilst understandable given that most empirical studies are focused on some particular practical goal other than testing the impact of different experimental designs, makes comparisons of different design construction techniques impossible, as well as preventing the offering of a robust set of guidelines for best practice in generating experimental designs specifically for DCEs. This coupled with the fact that orthogonal designs appear to have worked well over the years has also resulted in their widespread use for DCE studies. 1.3. Efficient designs Nevertheless, alternatives to simple orthogonal designs have been available for some time. Fowkes and Wardman (1988) questioned the use of fractional factorial designs based on orthogonal arrays and discuss the importance of experiments that are realistic and make sense to the respondents as well as improving parameter estimates. They introduced an additional requirement in which a reasonable coverage of so-called ‘boundary values’ is obtained. With Fowkes et al. (1993), Bunch et al. (1996) and Huber and Zwerina (1996), new experimental design theories began to emerge in the early to mid 1990s specifically for the generation of experiments for the non-linear discrete choice models often associated with DCE. Further theoretical advancements in the field of experimental designs have been made since including, but not limited to work undertaken by Toner et al. (1998), Sándor and Wedel (2001, 2002, 2005), Kanninen (2002), Street and Burgess (2004), Burgess and Street (2005), Kessels et al. (2006), Ferrini and Scarpa (2007), Rose et al. (2008), Scarpa and Rose (2008), Bliemer et al. (2009), Rose et al. (2009), and Bliemer and Rose (2010a, 2010b). The types of designs constructed by these researchers, typically referred to as D-efficient or D-optimal designs, each have the common goal of seeking to minimize the determinant of the asymptotic variance–covariance (AVC) matrix of models estimated on data collected using the designs, which essentially minimizes the standard errors, yielding more reliable parameter estimates. In other words, a (D)efficient design aims to maximize the (Fisher) information about the preferences of the respondents over all observations. Differences in the construction methods posed by each of the above researchers however lie solely in the assumptions made about how to a priori estimate what the AVC matrix of a design might look like without having first collected any data. These assumptions typically correspond to a priori parameter values (called parameter priors), which can be zero, fixed nonzero values, or even distributions, and to the a priori model type (e.g., multinomial logit, mixed logit, nested logit). Instead of applying the determinant to the AVC matrix, other measures derived from the AVC matrix have been proposed, leading to e.g., A-efficient designs that considers the trace of the AVC matrix (see Huber and Zwerina, 1996) or S-efficient designs that considers the maximum t-ratio (see Bliemer and Rose, 2005, 2009b). As an example, consider again the experimental design in Fig. 1. Some choice tasks may provide more information about preferences than others. For example, in the last choice task in the design, with a much smaller travel time for car than for train at equal travel costs, most respondents are likely to choose the car alternative. The best choice tasks are the ones that will clearly exhibit trade-offs in the attributes and do not contain strongly dominant alternatives. Not that such an assessment can only be made if (approximate) values for the parameters are known, at least the sign of the parameters. Mathematically, a D-efficient design minimizes the D-error, which is defined as

D-error ¼ detðXÞ1=K ;

ð3Þ

where X is the AVC matrix and K is the number of parameters to be estimated. This AVC matrix can be calculated as the inverse of the Fisher information matrix, defined as the negative of the expected Hessian matrix of the log-likelihood function L corresponding with the assumed logit model,

66

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

X ¼ I1 ;

with I ¼ E

! @2L : @b@b0

ð4Þ

In case of the MNL model, the log-likelihood function can be written as

L¼

XX s

ym;s log Pm;s ;

ð5Þ

m

where ym,s is the choice indicator that equals one if alternative m is chosen in choice task s, and zero otherwise. These choice observations are unknown at the time of constructing the experimental design, however, y drops out of the equations (at least for the MNL, NL, and cross-sectional ML models) when taking the expectation of the second derivatives in Eq. (4). So-called Bayesian D-efficient designs (see e.g., Kessels et al., 2008) optimize the D-error over a range of assumed parameter priors (given by random distributions), such that the D-error defined in Eq. (3) will become the expected D-error over these pffiffiffiffiffiffiffiffi distributions. Let sek ¼ Xkk denote the standard error of the kth parameter estimate, given by the roots of the diagonals of the AVC matrix. Asymptotic t-ratios can therefore be computed as tk = bk/sek. S-efficient designs aim to maximize the minimum t-ratio over all parameter estimates. The objective of this paper is not to detail the advancements in generating experimental designs for DCEs. This has been done elsewhere (see e.g., Bliemer and Rose, 2009a; Rose and Bliemer, 2008, 2009). Rather, the objective of this paper is to empirically compare the results obtained from the two different experimental designs types that have been proposed and mainly used within the literature. In this respect, this paper joins a very small but growing area of research that seeks to clarify whether the theoretical advancements claimed by researchers in the area of experimental design generation for DCEs translates into practice. It is hoped that such research will ultimately produce practical guidelines for the construction of DCE experimental designs. 1.4. Paper outline The remainder of the paper is organised as follows. In the next section, we provide a brief literature review of several of the top tier transportation journals, in terms of the types of experimental designs that have been used in the past. In this section, we also provide a review of the literature exploring the empirical impacts of different experimental design types. The following section details the experimental context, designs and data used for empirical comparison within this paper. Section 4 provides a discussion of the results before concluding comments are provided. 2. Literature review In this section, we report the results of literature review of a subset of tier one transportation journals specifically looking at the types of experimental designs used. We then go onto to discuss the literature examining empirically the impacts of various experimental design types have upon behavioural outcomes. 2.1. A Review of the transportation literature Table 1 summarizes the literature using DCE methods published in a subset of tier one transportation journals during the period of January 2000 to August 2009. Only articles in which the type of experimental design used could be determined are listed (i.e., articles such as Hess et al., 2007, which report the use of DCEs but do not provide any details as to the experimental designs used are excluded from the table). A total of 64 research papers were identified yielding 61 unique DCE experimental designs. Of these 61 studies, 40 (66%) utilized an orthogonal design, 12 a D-efficient designs (20%), seven (11%) randomly assigned attribute levels shown to respondents and three (5%) used an adaptive design approach, alternating the levels shown to respondents based on the respondents previous answers. The number of alternatives, J, for each study reported range between two and six and several of these studies included a current or reference alternative from which the DCE alternatives are pivoted from (e.g., Hensher and Prioni, 2002). The number of attributes, K, per alternative varied from three to 46 and the number of levels per attribute from 2 to 30. Some studies, such as Leitham et al. (2000), employed designs in which different attributes of the experiment had different numbers of attribute levels. Also reported, when detailed in the original study, are the fractional factorial dimensions and the size of the experimental designs used. The size of the design represents the total number of choice sets, S, generated. The majority of studies reviewed employed only a subset of the total possible number of choice sets. In determining which choice sets to give to which respondents, most studies used an additional blocking column in the design (39, or 64% of designs), however, some randomly assigned choice sets to respondents (eight, or 13%). Three (5%) studies gave each respondent all choice sets that were generated as part of the design. It was not possible to determine how the choice tasks were allocated to respondents in 11 (18%) of the studies reviewed. The number of choice sets given to any particular respondent is shown in the eighth column of Table 1. The number of respondents, N, surveyed is shown in the third last column of Table 1. The sample sizes reported represent the number of complete surveys returned that were deemed useable for each study. Related to the sample size is the number

67

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79 Table 1 Review of recent stated preference studies and sample sizes in the literature. Reference

Design type

Labelled/ unlabeled

Choice task assignment

J

K

S

Shown

Design size

N

NS

Models estimated

Brewer and Hensher (2000) Brownstone et al. (2000) Leitham et al. (2000) Ortúzar et al. (2000) Wang et al. (2000) Hensher (2001a) Hensher (2001b) Hensher and King (2001) Saelensminde (2001) Bhat and Castelar (2002) Cherchi and Ortuzar (2002) Garrod et al. (2002) Hensher and Greene (2002) Hensher and Prioni (2002) Wang et al. (2002)

Orthogonal

Unlabeled

Random

3

10

27

3

310

20

60

MNL

Random

Unlabeled

N/S

3

14

N/S

1

414

7387

4747

ML model

4

Greene and Hensher (2003) Hensher and Greene (2003) –

Orthogonal

Unlabeled

Random

2

7

16

2

2 3

40

640

MNL

Orthogonal

Labelled

All

2

7

8

8

27

357

2774

Logit

Orthogonal Orthogonal Orthogonal Orthogonal

Unlabeled Unlabeled Unlabeled Labelled

Blocked Blocked Blocked N/S

2 2 + rp 2 + rp 5

14 6 6 9

81 64 64 27

9 16 16 3

23  311 42  61 412 39

335 114 198 536

3015 1824 3168 1608

MNL MMNL MNL/MMNL NL

Orthogonal

Unlabeled

Random

2

3

16

9

43

508

4572

Logit scaling

Orthogonal

Labelled

Blocked

6

4

32

8

N/S

136

1068

MNL/NL

Orthogonal

Labelled

Blocked

3

4

27

8–9

34

524

1396

NL (SP-RP)

3

2

Orthogonal

Unlabeled

Random

2 + no

5

72

8

2 3

414

3312

MNL

Orthogonal

Labelled

N/S

5

3

N/S

4

315

210

840

NL

Orthogonal

Unlabeled

Blocked

2 + rp

13

81

3

313

N/S

3489

MNL

5

4

Orthogonal

Labelled

Blocked

4

6

32

16

4

274

4384

Orthogonal

Blocked

3 + rp

6

32

16

46

274

4384

Ordered Probit MNL/LC/ MMNL MMNL

Blocked

2 + rp

6

64

16

412

143

2288

MMNL

5

Orthogonal

Unlabeled

–

Orthogonal

Jovicic and Hansen (2003)

Orthogonal

a: Unlabeled b: Unlabeled c: Unlabeled Unlabeled

Hensher (2004)

D-efficient

Unlabeled

Zhang et al. (2004) Cantillo and Ortuzar (2005) Caussade et al. (2005) de Palma and Picard (2005) Saleh and Farrell (2005) Anderson et al. (2006) Bastin et al. (2006) Bhat and Sardesai (2006)

3

Orthogonal

NA

1

9

27

27

3 3

194

5238

6

3

Blocked

2 + rp

8

N/S

16

4 2

60

960

MMNL

N/S

2

3–5

N/S

N/S

N/S

19,989

NL (SP-RP)

Blocked

3–5

3–6

12– 30 81 27

6–15

Varying

418/ 589/ 453 427

4593

MNL/HEV

9 9

3 33

1400 342

3015 3071

MNL MNL/MMNL

12– 30 N/S

6–15

Varying

403

8020

N/S

N/S

4137

N/S

Orthogonal Orthogonal

Unlabeled Unlabeled

Blocked Blocked

2 2

14 3

D-efficient

Unlabeled

Blocked

3–5

3–6

Random

Unlabeled

N/A

N/S

N/S

14

Orthogonal

Labelled

Random

3

8

27

7

3

632

4424

MNL/Hetero L Ordered Probit MNL

D-efficient

Unlabeled

Blocked

2

7

35

7

35  42

298

2063

MMNL

Orthogonal Orthogonal

Labelled Labelled

N/S Blocked

4 3=

N/S 64

3 4

36  21 211  43

871 319

2602 1276

MMNL MMNL

D-efficient

Unlabeled

Blocked

3/5

7 5 from 14 3

N/S

N/S

N/S

N/S

782

MNL

Orthogonal

Labelled

Blocked

2

7

27

9

74

338

3042

NL (SP-RP)

Orthogonal

Labelled

Blocked

2

5

27

9

21  34

97

871

NL (SP-RP)

4

8

Cantillo et al. (2006) Cherchi and Ortuzar (2006) Espino et al. (2006) Greene et al. (2006) Hensher (2006)

D-efficient

Labelled

Blocked

3–5

46

60

10

4

184

1840

MMNL

D-efficient

Unlabeled

Blocked

3–5

3–6

6–15

Varying

427

4593

MNL/HEV

Hollander (2006) Loo et al. (2006)

Random Orthogonal

Unlabeled Unlabeled

N/A NS

2 2

7 13

12– 30 N/A N/S

9 2

N/A 21  313

244 483

2196 815

MNL MNL

*

46

(continued on next page)

68

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

Table 1 (continued) Reference

Design type

Labelled/ unlabeled

Choice task assignment

J

K

S

Shown

Design size

N

NS

Models estimated

Rizzi and Ortuzar (2006) Scott and Axhausen (2006) Scott and Axhausen (2006) Washbrook et al. (2006) Espino et al. (2007) Fosgerau (2007)

Orthogonal

Unlabeled

Blocked

2

3

27

9

33

Fowkes (2007) Hensher and Rose (2007) Hunt and Abraham (2007) Pucket et al. (2007a) Pucket et al. (2007b) Tilahun et al. (2007) Ahern and Tapley (2008) Beuthe and Bouffioux (2008) Hensher (2008a) Hensher (2008b) Hensher et al. (2008) Teichert et al. (2008) Train and Wilson (2008) Tseng and Verhoef (2008) Dagsvik and Liu (2009) Hensher et al. (2009) Hess and Rose (2009) Lee and Cho (2009) McDonnell et al. (2009) Puckett and Hensher (2009) Sener et al. (2009) Rouwendal et al. (2010)

12

342

3078

Binary logit

12

163

1304

Probit

Orthogonal

Labelled

Blocked

4

6

N/S

8

2

Orthogonal

Labelled

Blocked

4

24

72

8

213  311

N/S

1034

MNL

Orthogonal

Labelled

Blocked

3

4

32

8

34

548

4384

MNL

5

3

Orthogonal

Labelled

Blocked

2

5

27

9

3

97

871

NL (SP RP)

Random

Unlabeled

N/A

2

2

N/A

8

N/A

N/S

15,451

Adaptive

Unlabeled

N/A

4

9

N/A

10

N/A

49

N/S

46

Defficient* Random

Labelled

Blocked

3–5

46

60

10

4

184

1840

WTP space MNL Weighted Regression MMNL

Unlabeled

Random

2

5

N/A

1

Random

1128

1128

MNL

Defficient– Defficient– Adaptive

Unlabeled

Blocked

2 + rp

7

40

4/8

57

102

1248

MMNL

7

Unlabeled

Blocked

2 + rp

7

40

4/8

5

102

1248

MMNL

Unlabeled

N/A

2

2

N/A

5

N/A

167

835

GLMM

Orthogonal

Labelled

Random

2

5

16

5/10

22  43

189

1375

Orthogonal

Unlabeled

All

1

5

25

25

56

N/S

N/S

Rank ordered logit Ordered logit

D-efficient Defficient  D-efficient

Unlabeled Unlabeled

Blocked Blocked

2 + rp 2 + rp

5 5

32 32

16 16

45 4^5

222 205

3552 3280

MMNL MMNL

Unlabeled

Blocked

4

6

32

4

23  43

3232

MMNL

Orthogonal

Unlabeled

Blocked

2

7

128

8

27

209 pairs 5829

46,632

LCM

3

Random

Labelled

N/A

2

3

N/A

3

6

181

543

MMNL

Orthogonal

Labelled

N/S

4

15

N/S

11

413  22

1005

12,265

MMNL

Random

Labelled

All

3

8

15

15

100

1491

Ordered GEV

D-efficient

Unlabeled

Blocked

2

12

60

10

213

2130

MMNL

Defficient  Orthogonal

Unlabeled

Blocked

2 + rp

5

32

16

22  31  52  71  111  151 31  55  62  101  201  301 45

205

3280

MMNL

Unlabeled

Blocked

Ranked

5

216

20

21  33  41

492

9840

Orthogonal

Unlabeled

Random

2 + rp

7

32

18

22  32  53

310

5442

Linear regression MMNL

7

D-efficient

Unlabeled

Blocked

2 + rp

7

40

4

5

108

432

MMNL

Orthogonal Orthogonal

Unlabeled Unlabeled

Blocked Blocked

3 2

6 3

N/S N/S

4 N/S

21  38  52 N/S

1621 1055

6484 N/S

MMNL MMNL

* –   , , indicate that the same data set has been used in these studies, N/S = not stated, N/A = not applicable. Table notes: Journals examined are limited to (in alphabetical order) Journal of Transport Economics and Policy, Transportation, Transportation Research Part A, Transportation Research Part B, and Transportation Science. These transportation journals have been selected based on their impact factors, and their relevancy for the study in this paper. Some studies used a subset of observations from the total sample population. Numbers reported are those used in the estimation of models within the article. In many papers reviewed, the authors were unable to determine the dimensions of the experimental design or information about the sample size and number of observations captured. Where we were unable to determine these facts, we have marked the appropriate cell in the table as ‘not stated’.

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

69

of observations, NS, defined here as the number of choice sets used in the model estimation process. The number of observations ranged from a low of 60 (Brewer and Hensher, 2000) to a high of 46,632 (Teichert et al., 2008). The median number of observations over all the studies in which the number of observations in the study can be readily determined is 2688. A conclusion that can be drawn from our literature review is that much improvement can be made in terms of what is reported within the stated choice literature. 2.2. A review of the empirical literature exploring the impact of experimental designs upon modelling outcomes Only a small number of studies have empirically examined the differences between data sets collected using different experimental designs. Toner et al. (1998) compared the boundary value approach to a new approach with so-called ‘magic P’ values which represent the analytically derived optimal choice probabilities in binary choice experiments. These magic P values are to some extent consistent with D-efficiency. They find that the t-values of the parameter estimates can be increased by at least 50% in theory, although in practice they found improvements of on average 25%. Toner et al. (1999) compared five different designs, both theoretically and empirically, to estimate a binary choice model with four attributes. Designs compared included a standard orthogonal design, an orthogonal design with a reasonable coverage of boundary values, a magic P design, a constrained magic P design in which the attribute levels have to fall within some boundaries, and a constrained magic P design with attribute levels that have been rounded off. As they show in the paper, in a theoretical setting, for the (unconstrained) magic P design they found the largest gains in t-ratios compared to the standard orthogonal design, while the boundary value showed (on average) no improvement. However, in an empirical setting, these results could not be replicated. Indeed, no improvements in t-ratios were found in practice and even worse results than the standard orthogonal design were found. This finding is not consistent with their previous finding in Toner et al. (1998), however significant differences in the relative scales were found, with the two orthogonal designs yielding much larger scales than the efficient (magic P) designs, which may partly explain the inconsistencies between the two studies. Viney et al. (2005) empirically investigated three different experimental designs, namely an orthogonal main effects design, a utility-balanced design, and a random design. The designs consist of two alternatives, and three, four-level, alternative-specific attributes. They conclude that the three experimental designs did not impact on the underlying parameter estimates, however, the utility-balanced design (in which a respondent is expected not to have a preference for either alternative) yielded more random variability in the responses. Tudela and Rebolledo (2006) considered a simple binary choice experiment with three attributes. They compared a classical boundary value design with a design that is optimized for the variances of the parameter estimates. Comparing mixed logit estimates with only 16 respondents (144 choices) showed significant improvements in the t-ratios when using such an optimal design. Louviere et al. (2008) in a single study examined 44 different experimental designs varying systematically the various design dimensions as well as the level of statistical efficiency of the designs. It was found that the more statistically efficient the design, the greater the error variance resident within the data. This relationship was found to exist independent of the overall dimensions of the design. Hess et al. (2008) compared the results of three different experimental designs, including an orthogonal design with randomized choice set assignment, an orthogonal design with an orthogonal blocking column and an efficient design. They found that the efficient design performed only marginally better than the orthogonal design with blocking, but that the design with random assignment of choice tasks to respondents performed significantly worse than both the efficient designs and the orthogonal design with blocking. As such, they concluded that the blocking of the experiment was far more important than the underlying experimental design itself. Whilst contributing to our understanding of the impacts differing experimental design types might have upon modelling outcomes, the above mentioned studies fail to examine whether the two main theoretical advantages commonly associated with efficient designs actually translate into practice. That is, (D)-efficient designs have been theoretically promoted within the literature (see the references in the introduction) as producing more reliable parameter estimates with smaller sample sizes than other forms of designs, such as orthogonal designs. Since this claim has been primarily investigated using theoretical examples, we seek to examine this claim here from an empirical point of view, more rigorous and in a more realistic setting than the typical binary choice experiment with few attributes. 3. Empirical study: data collection 3.1. Empirical context The study presented in this paper deals specifically with air travel choice. Typically, when travelling to (larger) cities, it is possible to fly to a main airport close to the city itself. However, flying into main airports is becoming more expensive, both for the individual traveller and the airlines themselves. For this reason, there has been a growth in the use of regional airports that are located in the proximity of major cities. Such airports typically are cheaper to fly to (due to lower taxes) but require greater travel distanced to either access or egress. In the current study, we examine the trade-offs mainly between ticket

70

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

price and travel time. One aim of the study is to capture the behaviour of respondents by presenting choices that closely resemble real-life choices. For this reason, we selected a particular trip, namely a holiday trip from Amsterdam to Barcelona, a popular holiday destination with a significant frequency of trips. The origin airport for the study is Schiphol Airport in Amsterdam, while the destination airport is either the main Barcelona Airport or the regional Girona Airport. The latter airport is much further away from Barcelona city, being 100 km northeast from Barcelona, compared to 13 km west for Barcelona Airport. Currently, several charter airlines offer cheap tickets to Girona Airport, such that the traveller has the option to pay less for the air component of travel, but requires a longer travel time to Barcelona city by bus. Travellers choose a ticket among others based on ticket price, departure time, number of stops or transfers, and airline company (one can have a preference for a certain airline based on for example, experienced comfort or air miles membership card). In the current study, we also take destination airport and travel time and cost to the city centre into account. 3.2. The discrete choice experiment Respondents were asked to complete an online DCE involving the choice of air ticket from amongst a set of five possible tickets. In the experiment, each of the five alternative tickets were described by the name of the airline offering the ticket, the ticket price, the departure time, a possible transfer time (at maximum only 1 stop) and the destination airport. To capture egress times and costs, the various tickets also displayed the time required to travel from the destination airport to the city of Barcelona as well as the price. Given that the flight time from Amsterdam to Barcelona airport and Girona airport are roughly the same, this was fixed at 2 h and not varied as part of the experiment. It is important to make a distinction between what is presented to the respondent, and what model is estimated. In the experiment we show the number of stops (0 or 1) and the transfer time (1 or 2 h), while in our model we estimate a single parameter for transfer time with attribute values 0, 60, and 120 min (all time attributes have been expressed in minutes for consistency). This is a compromise between limiting the number of parameters while keeping the choice experiment as realistic as possible. Note that multiple models could be estimated from the choice experiment (as is the case with basically all experiments), but we have chosen one of the simplest that we also optimized on in our design process. Each respondent was exposed to six choice tasks in which the attributes shown were varied according to some underlying experimental design (see Section 3.3). The levels that each attribute could take as part of the experiment are shown in Table 2. Note that depending on the destination airport, different egress time and cost levels were selected. It should be noted that all attribute levels are consistent with a single trip, and that for example for a round trip double the price is to be considered. Fig. 2 shows an example choice task given to respondents (two alternatives are cut off in the screenshot). Within each choice task, respondents were able to sort the presented alternatives by airline, price, travel time, etc., and also select which attributes to show or omit. There is also an option to show the total travel time (flight time plus transfer time plus egress time) and total price (flight price plus egress price). This flexibility was included so as to enhance the feeling of dealing with an actual online travel agent rather than a traditional DCE. Of course, before the choice situations are presented, the scenario is sketched (a holiday trip from Amsterdam to Barcelona), and in the end also questions about the respondent (gender, income, etc.) are asked. 3.3. The experimental design procedure Three different experimental designs have been generated; one orthogonal design and two D-efficient designs. The smallest (simultaneous) orthogonal design that could be located with all attribute levels uncorrelated had 108 choice situations. Given that it was determined that presenting each respondent with 108 choice situations was not feasible, an orthogonal blocking column was generated such that 18 blocks were created, each containing six different choice situations. Alongside the orthogonal design, a D-efficient design was constructed with 108 choice situations which was also blocked into 18 blocks containing six choice tasks each. Finally, a second D-efficient design was constructed with 18 choice situations which was subsequently blocked into three blocks. In the final survey, each respondent faced six choice situations from one of these three designs. The block allocation to the respondents was implemented in such a way that an even distribution over the different designs was established. The actual experimental designs used are available from the authors upon request.

Table 2 The attribute levels used as part of the DCE.

a

Airline

Ticket price

Departure time

Transfer time

Egress price

Egress time

Air France KLM Iberia Vueling Transavia Easy jet

€50 €75 €100

6:00 12:00 18:00

0 h (non-stop) 1 h (1 stop) 2 h (1 stop)

€1a or €9 €3a or €12 €5a or €15

20 mina or 1 h 30 mina or 1hr20 40 mina or 1hr40

If destination is Barcelona, other values are for destination Girona.

71

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

Fig. 2. Example choice task.

In order to construct D-efficient designs, the analyst is required to assume prior parameter estimates in a Bayesian-like fashion. Researchers need not assume precise prior parameter values but rather, may assume prior parameter distributions that are expected to contain within their range, the true population parameter. In taking this approach, the resulting Bayesian efficient design is then optimized over a range of possible parameter values, without the analyst having to know the precise population value in advance (see e.g., Sándor and Wedel, 2001; Kessels et al., 2006, 2008). For the current study, the two D-efficient designs were generated using parameter priors obtained from a small pilot study consisting of 36 respondents yielding a total of 216 choice observations. The pilot experiment used the aforementioned orthogonal design such that each block was replicated exactly twice. The parameter estimates of the pilot study have been estimated according to an MNL model where the attribute ‘Airline’ has been dummy-coded with base ‘EasyJet’, and departure time has been dummy-coded with base ‘6 pm’. Table 3 shows the model results for the pilot study. The prior parameters used for design generation were drawn from Bayesian normal distributions b N(l, r2) where the mean l was assumed to be the estimation value of the parameter from the pilot study and the standard deviation r the standard error of the parameter. For non-significant parameter estimates, the prior parameters were assumed to be zero with the corresponding standard error as value for the standard deviation (e.g. the prior for egress price is assumed to be normally distributed with mean 0 and standard deviation 0.03).

Table 3 Pilot study parameter estimates and efficient design priors. Attribute

Parameter

t-value

Prior (mean)

Prior (stdev)

Air France KLM Iberia Vueling Transavia Ticket price Departure time (6 am) Departure time (12 pm) Transfer time Egress price Egress time

0.516 0.194 0.218 0.125 0.547 0.040 0.215 0.614 0.006 0.027 0.018

1.596 0.661 0.729 0.426 1.765 8.444 0.974 2.883 3.298 0.904 3.265

0 0 0 0 0 0.040 0 0.614 0.006 0 0.018

0.323 0.293 0.299 0.293 0.310 0.005 0.221 0.213 0.002 0.030 0.006

q2 = 0.190, LL = 266.

72

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

3.4. Main field phase sample Respondents were sampled by a Dutch market research company, TeamVier, who sent out emails to people in their panel database. In total 618 respondents completed the survey, such that the total number of observations (chosen alternatives) is 3708. Of this sample, 206 (1236 observations) respondents completed choice tasks from the orthogonal design, 208 (1248 observations) from the efficient design with 108 choice tasks, and 204 (1224 observations) from the smaller efficient design. The age groups 26–35, 36–45, and 60+ are more or less equally represented in the sample, while age group 45–59 is the largest group and 18–25 is the smallest group. Slightly more males than females are in the sample. Most people have a full time job, the second largest group has a part time job, another large group has no job, and a small group stated to be a student. A quite uniform distribution of income is observed, from €0 to €70,000 per year. 4. Empirical study: results 4.1. Overall model results Before examining the impact of the different experimental designs on sample size, several models were estimated both on the full pooled data from all three experimental designs (i.e., using all 3708 choice observations), as well as on the separate design specific data sets. Table 4 presents the MNL model results from the estimation on the pooled data set. As well as estimating MNL models, the ‘nested logit trick’ was used to test for scale differences between each of the data sets. The result of this model are not reported however as the model collapsed back to the MNL model, suggesting that the different designs do not induce statistically significant different error variances. It should be noted that this finding does not contradict that of Louviere et al. (2008) however as in that paper, they explored not only different designs, but designs with differing design dimensions. In the current study, the design dimensions were kept constant with only the experimental design varying. Table 4 lists the parameter estimates on data from each design separately and pooled. Looking at the airline parameters (relative to ‘EasyJet’) of the pooled data, the order of preference for airlines appears to be (from most preferred to least preferred): Air France, EasyJet, Transavia, Vueling, KLM, and Iberia. As expected, both price parameters are negative, and so are the transfer time and egress time parameters. It is interesting to look at the parameter ratios, indicating WTP for less transfer time or less egress time. The WTP for 1 h less transfer time is approximately €32.14 (based on the flight price) compared to a WTP for 1 h less egress time is €19.29 (based on the flight price) or €15.43 (based on the egress price). Therefore, roughly speaking one is willing to pay about €10.29 (single trip) for flying to Barcelona instead of Girona airport, which is about 40 min further away. Note that the parameters in Table 4 are different from the parameters used as priors for the generation of the efficient designs in Table 3. Efficient designs are efficient under the assumption that the priors are correct, hence it is likely that some efficiency is lost in the data, although the data from a Bayesian efficient design remains relatively efficient over a range of parameter prior values. In terms of testing the hypothesis put forward by Hess et al. (2008) with relation to blocking representing the most important criteria of experimental design, given that all designs in the current study were blocked, we are unable to test at the level of estimating models on the full design specific data sets, the impact of blocking upon the design specific results. We are able to test this impact however in Section 4.4 where we use bootstrapping at various sample sizes, and compare the

Table 4 MNL model results (pooled data). Attribute

Parameter

t-ratio

Lower 95%

Upper 95%

Air France KLM Iberia Vueling Transavia Ticket price Departure time (6 am) Departure time (12 pm) Transfer time Egress price Egress time

0.377 0.232 0.535 0.008 0.137 0.028 0.283 0.460 0.015 0.035 0.009

6.019 3.456 7.629 0.120 2.046 25.115 6.221 9.860 32.228 5.956 8.459

0.254 0.364 0.672 0.132 0.268 0.030 0.194 0.368 0.016 0.047 0.011

0.500 0.101 0.398 0.117 0.006 0.026 0.372 0.551 0.014 0.024 0.007

Model fits LL (ASC only model) LL (b)

q2 Respondents Observations

5879.526 4774.862 0.188 618 3708

73

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

results after (i) maintaining equal representation of respondents over the blocks and (ii) bootstrapping without reference to the blocks that respondents were assigned. 4.2. Design specific model results Table 5 presents the MNL model results for each of the designs. The statistically significant parameter estimates have the same sign across the three design types and are more or less of the same magnitude. However, there are some notable differences. For example, the parameter estimates for transfer time and ticket price, the two parameters with the highest t-ratios, are quite different between the orthogonal and the two efficient designs, and their ratio even more so. Although asymptotically any design type should lead to the same parameter estimates, in limited samples the parameter estimates can clearly be different. While one would expect the parameter estimates to be unbiased (in large samples), regardless of the design type used, the design could have an influence. When a design would consist of many dominant alternatives, the parameters are likely to be larger (in an absolute sense), as the error variance will be smaller for such choice situations. Hence, in a stated choice experiment the parameters may become biased when many dominant alternatives are included in the experimental design. As the orthogonal design is generated without any assumptions on parameter priors, one cannot avoid dominant alternatives here (unless one removes them by manual inspection), whereas efficient designs mostly rule out choice situations with dominant alternatives (as such a choice situation would be very inefficient). Inspection of the orthogonal design indeed shows that it contains several choice situations with clearly dominant alternatives, whereas the efficient designs contain very few (efficient design with S = 108) or none (efficient design with S = 18). In Bliemer and Rose (2010a) evidence is found that bad priors may lead to dominant alternatives in the design, biasing the parameter estimates, at least in small sample sizes. This finding does not contradict results in Louviere et al. (2008), in which they found that efficient designs, typically containing ‘more difficult’ choice tasks, introduce more error variance (leading to smaller parameter values). In our case, we conjecture that dominant choice alternatives, often found more in inefficient designs, lead to smaller error variances. These error variances may therefore play an important role and may bias the parameter estimates. Since the error variance plays a role at the level of a choice task, not just the whole design, the impact upon each of the parameter estimates is somewhat unpredictable. More research into the role of the error variance is required on this topic and to confirm our conjecture. 4.3. Comparison of design outcomes against model results In generating the design, we assumed some parameter priors obtained from a small pilot study, see Table 3. The efficient designs were generated under these assumptions. The estimated parameter values are presented in Table 4 (and Table 5 for each design separately). There are some obvious differences between the assumed means of the priors and the estimated parameters, such that the efficient designs likely loose some efficiency. However, we have tried to minimize this loss in efficiency by adopting Bayesian efficient design principles in which we included the uncertainty about these priors (given by the standard deviations in Table 3), as explained in Section 3.3.

Table 5 MNL model results for different designs. Attribute

Air France KLM Iberia Vueling Transavia Ticket price Departure time (6 am) Departure time (12 pm) Transfer time Egress price Egress time

Design 1: orthogonal, S = 108

Design 2: efficient, S = 108

Design 3: efficient, S = 18

Par.

t-ratio

Lower 95%

Upper 95%

Par.

t-ratio

Lower 95%

Upper 95%

Par.

0.422 0.231 0.488 0.285 0.024 0.035 0.300

3.423 1.883 3.756 2.345 0.190 17.780 3.311

0.180 0.471 0.742 0.047 0.227 0.038 0.123

0.664 0.010 0.233 0.524 0.276 0.031 0.478

0.319 0.402 0.650 0.141 0.301 0.023 0.410

2.994 3.265 5.445 1.324 2.635 10.808 5.387

0.110 0.643 0.884 0.349 0.524 0.027 0.261

0.527 0.161 0.416 0.068 0.077 0.019 0.560

0.457 0.112 0.417 0.131 0.052 0.021 0.171

0.627

7.181

0.456

0.798

0.348

4.181

0.185

0.511

0.011 0.041 0.012

13.999 3.311 5.205

0.012 0.065 0.016

0.009 0.017 0.007

0.015 0.038 0.006

19.410 3.894 3.102

0.017 0.057 0.010

0.014 0.019 0.002

t-ratio

Lower 95%

Upper 95%

4.401 1.017 3.523 1.173 0.469 9.649 2.285

0.253 0.328 0.649 0.351 0.270 0.026 0.024

0.660 0.104 0.185 0.088 0.166 0.017 0.318

0.303

3.786

0.146

0.460

0.018 0.032 0.006

22.350 3.330 3.342

0.020 0.051 0.01

0.017 0.013 0.003

Model fits LL (ASC only model) LL (b)

q2 Respondents Observations

1953.577 1557.904 0.203 206 1236

1940.001 1607.444 0.171 204 1248

1983.633 1550.317 0.218 208 1224

74

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

In the design generation process, the predicted AVC matrices are used to assess their efficiency. In model estimation the variance–covariance matrices are an output. These variance–covariance matrices determine the standard errors and the tratios. In this section we would like to compare the predicted t-ratios in the design generation process (before the survey) with the t-ratios computed at model estimation (after the survey). The model t-ratios are listed in Table 5. The t-ratios for each design can be computed by taking the square roots of the diagonals of the AVC matrices (as predicted in the design process) for the given sample sizes (206, 204, and 208 respondents for the three designs, respectively). In order to be able to make the comparison, the parameter estimates from the pooled data set are used priors. Fig. 3 depicts the comparison between the predicted and observed (in estimation) t-ratios for all parameters in each of the three datasets (hence, 11  3 = 33 points in the figure). Overall, the prediction is quite good, where predicted low t-ratios are observed as low, and high predicted t-ratios are observed high. A very interesting observation is, that the observed t-ratios seem to be slightly smaller than the predicted t-ratios. Hence, in this case study the predicted t-ratios can be seen as a lower bound, where the bound is general quite tight, although some outliers exist. Therefore, we conclude that the theoretical design principles are indeed very useful and quite accurate in making predictions about standard errors and t-ratios without conducting any survey. Hence, it is possible to identify parameters in advance that will likely not be statistically significant in estimation. Also, sample sizes can be derived as discussed in Bliemer and Rose (2005, 2009b). Note that the predicted t-ratios will deviate more if the assumed priors deviate more from the actual parameter estimates. Therefore, it is wise to invest in a good pilot study in order to obtain a good prediction of the t-ratios.

4.4. Comparison of the efficiencies of the design specific data sets The theoretical justification for using efficient as opposed to orthogonal designs is that the former are expected to produce smaller standard errors at a given sample size, or conversely require smaller sample sizes to produce larger t-ratios. To date, and to the best of our knowledge, this has only been shown to occur in simulated data sets. In this section, we examine whether the theoretical advantages of generating efficient designs actually translate to real life empirical data sets. We will analyze our three data sets that resulted from the three underlying experimental designs by (i) looking at the aggregate results using all respondents and (ii) bootstrapping on each data set in order to randomly select respondents in the dataset for different sample sizes. Using the aggregate results, we can compute the D-error for each of the data sets, which is the determinant of the variance–covariance matrix, normalized to the number of parameters. In order to compare these D-errors, we have further normalized each D-error to a single respondent. The lower this D-error, the more efficient the data set. The D-errors for the original design, D-efficient design 1, and D-efficient design 2, are given by 0.1253, 0.1035, and 0.1017, respectively. The two efficient designs generated almost equally efficient data sets. Hence, a large design with 108 choice situations need not outperform a small design with only 18 choice situations. Since the number of choice situations in the orthogonal design could not be decreased (as we were not able to find any such orthogonal design), this is a clear advantage of the efficient designs in which the number of choice situations can be kept small. Comparing the D-errors, two efficient designs clearly

25

observed t-ratio

20

15

10

5

0 0

5

10 15 predicted t -ratio

20

25

Fig. 3. Predicted (before survey) and observed (after survey) t-ratios (red = orthogonal design, blue = efficient design 1 (S = 108), purple = efficient design 2 (S = 18). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

75

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

outperform the orthogonal design in terms of efficiency. This should translate into smaller standard errors for the two efficient designs, which will be investigated next. We use bootstrapping to examine the performance of the three designs over different sample sizes. Simultaneously, we also test the impact that blocking has upon design performance by performing two sets of bootstrapping on each. Firstly, we bootstrap in such a way that each block in each design is replicated an equal number of times such that the properties of the design will be maintained through to model estimation. Next, we perform bootstrapping in such a way that respondents are randomly selected, independent of the design block to which they were exposed. In this way, each bootstrap iteration will likely sample respondents unequally from each block, thus impacting upon the design properties in terms of how they are expected to translate through to model estimation. In each case, we perform 250 bootstrap iterations using the same utility specification that was assumed previously.

0.5

0.5

0.45

0.5

0.45

Air France

0.45

KLM

0.4

0.4

0.4

0.35

0.35

0.35

0.3

0.3

0.3

0.25

0.25

0.25

0.2

0.2

0.2

0.15

0.15

0.15

0.1

0.1

0.1

0.05

0.05

0.05

0

0

20

40

60

80

100

120

140

160

180

0.5

0

0

40

60

80

100

120

140

160

180

0.5

0.45

0.4

0.35

0.35

0.3

0.3

0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

20

40

60

80

100

120

140

160

180

0

0

0

20

40

60

80

100

120

140

160

180

140

160

180

140

160

180

0.015

0.45

Vueling

0.4

0

20

Iberia

Transavia

Ticket Price 0.01

0.005

0

20

40

60

80

100

120

140

160

180

0

0

20

40

60

80

100

120

-3

8

Departure 6am

0.3

Departure 12pm

0.3

0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

0

x 10

Transfer Time

7 6 5 4 3

0

20

40

60

80

100

120

140

160

180

2 1

0

20

40

60

80

100

120

140

160

180

0

0

20

40

60

80

100

120

-3

x 10

0.05 0.045

8

Egress Price

0.04

Egress Time

Orthogonal design (S=108)

7

0.035

6

0.03

Efficient design 1 (S=108)

5 0.025 4 0.02 3

0.015

1

0.005 0 0

Efficient design 2 (S=18)

2

0.01

20

40

60

80

100

120

140

160

180

0 0

20

40

60

80

100

120

140

160

180

Fig. 4. Standard errors for different designs per sample size.

76

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79 Table 6 Comparison of sample sizes. Attribute

Air France KLM Iberia Ticket price Departure time (6 am) Departure time (12 pm) Transfer time Egress price Egress time a

Orthogonal design (S = 108)

D-efficient design 1 (S = 108)

D-efficient design 2 (S = 18)

Sample size

Increasea

Sample size

Increasea

Sample size

85 221 47 4 82 29 3 98 47

+24 +49 +8 1 +26 +4

63 168 39 5 57 26 3 61 37

+2 4

61 172 39 5 56 25 3 61 32

+37 +15

+1 +1

+5

Extra sample size required, compared to D-efficient design 2.

First, we examine the impact upon the standard errors allowing for equal representation within each of the blocks. Across all sample sizes, at the 95% confidence level, seven of the 11 standard errors for the orthogonal design were found to be statistically different when comparing the results of maintaining equal block representation against unequal representation, only three were statistically different for the first efficient design, whilst all 11 were statistically different for the second efficient design. This suggests that blocking may have a statistically significant impact upon the statistical efficiency of the parameter estimates, although this impact appears to be small in magnitude. Therefore, in the analysis following we will just consider the results in which each block is represented an equal number of times, noting that the results for unequal representations of the blocks are similar. Fig. 4 graphs the average standard error of each parameter over a range of sample sizes based on the bootstrap simulations. The graphs show that the two efficient designs produce very similar standard errors, which are in smaller than the standard errors produced by the orthogonal design. Only in two situations does the orthogonal design produce smaller standard errors, namely for the ticket price parameter and (to a lesser extent) the transfer time parameter. This is no coincidence. In the graphs also the levels of standard error required to obtain a statistically significant parameter estimate (t = 1.96) are plotted by horizontal solid black lines, based on the pooled parameter estimates. The ticket price and transfer time parameters can easily be estimated with a very small sample size. Efficient designs are optimized such that they trade off all standard errors simultaneous, such that all parameters can be estimated with relatively low standard errors. It is clear that the ticket price and transfer time parameters already have low standard errors, hence the efficient designs mainly aim to minimize the standard errors of the other parameters, accepting slightly higher standard errors for ticket price and transfer time. The graphs for the two efficient designs are very similar, again finding that an efficient design with only a small number of choice situations is just as good as efficient designs with a large number of choice situations. Not only is it just as good, in larger sample sizes it even (slightly) outperforms the larger efficient design, while in smaller sample sizes the larger design seems to yield smaller standard errors. This is an interesting observation, which may be explained by the fact that for small sample sizes (with only few respondents) the variation in the choice situations may play a dominant role, while for large sample sizes the (few) best choice tasks, present in the smaller design, yield best results. The improvement in standard errors when using an efficient design may seem small, but standard errors that are a factor p higher require a factor p2 higher sample size. For example, if a standard error is 10% higher (factor 1.1), then a 21% higher sample size is required. Table 6 presents the sample sizes (required for statistical significant parameter estimates at the 5% level) for the nine statistically significant parameter estimates corresponding to the graphs in Fig. 4, where the appropriate sample sizes correspond to the intersection of the graphs with the horizontal line for t = 1.96. The values in Table 6 are absolute deviations from the sample sizes required for the overall best performing design, D-efficient design 2. The first efficient design performs almost equally well. The orthogonal design requires for all but one parameter significantly higher sample sizes, up to 49 respondents more for the same estimation reliability. 5. Discussion and conclusions In this paper, we have provided a systematic and detailed study comparing empirically various claims made by researchers as to the benefits of using efficient experimental designs for studies involving DCEs. In it, we have provided an overview of the various types of experimental designs that have been used in practice for the past decade based on a literature review of several tier one transportation journals. Based on this literature review, it is clear that the use of efficient designs is growing within the literature, although their use still remains in the minority. In this study we showed that the (asymptotic) variance–covariance matrices computed in the design generation process (before the survey) closely corresponds to the variance–covariance matrix in estimation (after the survey), such that for example t-ratios can be predicted with a quite high level of accuracy for each parameter before the experiment is taken to field. However, the prediction quality depends on the accuracy of the parameter priors needed to generate the efficient designs, making pilot studies even more essential.

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

77

An important finding in this research is that we could indeed empirically produce lower standard errors with efficient designs compared to an orthogonal design as suggest by the theory. These lower standard errors translate into significant decreases in sample size for achieving the same level of statistical significance in estimation. Also, we found that efficient designs with a large number of choice situations need not be more efficient per choice situation than an efficient design with a small number of choice situation, even the contrary may be true. When we look at the parameter estimates themselves, different designs yield somewhat different parameter values and the question rises of the design can bias the parameter estimates. It is not entirely clear why this is the case, but we conjectured that it may lie in the fact that dominant alternatives in a stated choice study bias the parameter estimates. While the orthogonal design contains the most choice situations with dominant alternatives, they may be the most biased. An alternative possible cause might lie in the way that orthogonal designs are constructed. These designs are constructed so that each pairwise attribute combination appears an equal number of times (or as close to possible) over the design. Thus for example, the 6 am, 12 pm and 6 pm departure time levels each appear 12 times with the 0, 60 and 120 min transfer time attribute levels. This relationship need not, indeed does not, hold for the efficient designs. As such, it is possible that in a stated choice experiment, the parameters may become biased when using efficient experimental designs in practice. At present, this remains conjecture, and urgent research is required to confirm or rule out this as a possibility. The current paper suffers from a number of limitations, the most predominate of which is our use of MNL models. As has been well know for some time, the MNL model fails to account for the correlation of preferences within individuals across choice tasks. To account for this phenomenon, more advanced models such as MMNL or error components models are required. Whilst we acknowledge this limitation here, we note that previous research has found that designs generated specifically for MNL models, tend to perform reasonably well when more advanced models are used (see e.g., Rose et al., 2009; Bliemer and Rose, 2010b). We note however that this research also tends to rely on model simulations and hence empirical work is required to confirm whether this finding also translates into practice. Overall, we were unable to substantiate the findings of Louviere et al. (2008) in terms of efficient designs tend to induce greater error variance, although we have also conjectured here that might indeed be the case. We hypothesize that this might be due to orthogonal designs producing far more dominated alternatives than efficient designs, although again, this requires further research to confirm. Our findings do however tend to support those of Hess et al. (2008) in that blocking of the design appears to be of some importance. Our study differs to that of Hess et al. (2008) in which we do not randomly assign choice tasks to respondents but rather maintain respondents within blocks. We further distinguish our research to that of Hess et al. (2008) by comparing model results of maintaining between block representativeness to a random assignment of respondents independent of blocks, nevertheless, we reach similar conclusions in terms of the importance of blocking and maintaining equal sampling over blocks in DCEs. In conclusion, our research empirically supports some of the claims made by those researching in the area of efficient designs. Despite the above acknowledged shortcomings, we have found that these designs do indeed appear to produce more reliable estimates. Whilst we recommend more studies similar to the one described here, encompassing greater numbers of types of designs be carried out, we find no reason at this point to reject the use of efficient designs, and indeed we recommend their use in studies where smaller sample sizes are likely to be a reality. Acknowledgments The authors would like to thank Reinier Beelaerts van Blokland for his work on the data collection and the preliminary analyses, and Yun Zhang for implementing the internet survey. References Ahern, A.A., Tapley, N., 2008. The use of stated preference techniques to model modal choices on interurban trips in Ireland. Transportation Research Part A 42 (1), 15–27. Anderson, C.M., Das, C., Tyrrel, T.J., 2006. Parking preferences among tourists in Newport, Rhode Island. Transportation Research Part A 40 (4), 334–353. Bastin, F., Cirillo, C., Toint, P.L., 2006. Application of an adaptive Monte Carlo algorithm to mixed logit estimation. Transportation Research Part B 40 (7), 577–593. Beuthe, M., Bouffioux, C., 2008. Analysing qualitative attributes of freight transport from stated orders of preference experiment. Transportation 42 (1), 105– 128. Bhat, C.R., Castelar, S., 2002. A unified mixed logit framework for modeling revealed and stated preferences: formulation and application to congestion pricing analysis in the San Francisco Bay area. Transportation Research Part B 36 (7), 593–616. Bhat, C.R., Sardesai, R., 2006. The impact of stop-making and travel time reliability on commute mode choice. Transportation Research Part B 40 (9), 709– 730. Bliemer, M.C.J., Rose, J.M., 2005. Efficiency and Sample Size Requirements for Stated Choice Studies, Working Paper: ITLS-WP-05-08. Bliemer, M.C.J., Rose, J.M., 2009a. Designing stated choice experiments: state-of-the-art. In: Kitamura, R., Yoshii, T., Yamamoto, T. (Eds.), The Expanding Sphere of Travel Behaviour Research. Emerald, UK, pp. 499–537. Bliemer, M.C., Rose, J.M., 2009b. Efficiency and Sample Size Requirements for Stated Choice Experiments. Transportation Research Board Annual Meeting, Washington, DC, January. Bliemer, M.C.J., Rose, J.M., Hensher, D.A., 2009. Efficient stated choice experiments for estimating nested logit models. Transportation Research Part B 43 (1), 1935. Bliemer, M.C.J., Rose, J.M., 2010a. Serial choice conjoint analysis for estimating discrete choice models. In: Hess, S., Daly, A. (Eds.), Choice Modelling: The State-of-the-art and State-of-the-Practice. Emerald, Bingley, UK, pp. 139–161. Bliemer, M.C.J., Rose, J.M., 2010b. Construction of experimental designs for mixed logit model allowing for correlation across choice observations. Transportation Research Part B 44 (6), 720–734.

78

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

Brewer, A.M., Hensher, D.A., 2000. Distributed work and travel behaviour: the dynamics of interactive agency choices between employers and employees. Transportation 27 (1), 117–148. Brownstone, D., Bunch, D.S., Train, K., 2000. Joint mixed logit models of stated and revealed preferences for alternative-fuel vehicles. Transportation Research Part B 34 (5), 315–338. Bunch, D.S., Louviere, J.J., Anderson, D.A., 1996. A Comparison of Experimental Design Strategies for Multinomial Logit Models: The Case of Generic Attributes. Working Paper, Graduate School of Management, University of California at Davis. Burgess, L., Street, D.J., 2005. Optimal designs for choice experiments with asymmetric attributes. Journal of Statistical Planning and Inference 134, 288–301. Cantillo, V., Ortuzar, J.deD., 2005. A semi-compensatory discrete choice model with explicit attribute thresholds of perception. Transportation Research Part B 39 (8), 641–657. Cantillo, V., Heydecker, B., Ortuzar, J de D., 2006. A discrete choice model incorporating thresholds for perception in attribute values. Transportation Research Part B 40 (9), 807–825. Caussade, S., Ortuzar, J de D., Rizzi, L.I., Hensher, D.A., 2005. Assessing the influence of design dimensions on stated choice experiment estimates. Transportation Research Part B 39 (8), 621–640. Cherchi, E., Ortuzar, J de D., 2002. Mixed RP/SP models incorporating interaction effects: modelling new suburban train services in Cagliari. Transportation 29 (4), 371–395. Cherchi, E., Ortuzar, J de D., 2006. On fitting mode specific constants in the presence of new options in RP/SP models. Transportation Research Part A 40 (1), 1–18. Dagsvik, J.K., Liu, G., 2009. A framework for analyzing rank-ordered data with application to automobile demand. Transportation Research Part A 43 (1), 1– 12. de Palma, A., Picard, N., 2005. Route choice decision under travel time uncertainty. Transportation Research Part A 39 (4), 295–324. Espino, R., Ortuzar, J de D., Roman, C., 2007. Understanding suburban travel demand: flexible modelling with revealed and stated choice data. Transportation Research Part A 41 (10), 899–912. Espino, R., Román, C., Ortúzar, J de D., 2006. Analysing demand for suburban trips: a mixed RP/SP model with latent variables and interaction effects. Transportation 33 (3), 241–261. Ferrini, S., Scarpa, R., 2007. Designs with a-priori information for nonmarket valuation with choice-experiments: a Monte Carlo study. Journal of Environmental Economics and Management 53, 342–363. Fosgerau, M., 2007. Using nonparametrics to specify a model to measure the value of travel time. Transportation Research Part A 41 (9), 842–856. Fowkes, T., 2007. The design and interpretation of freight stated preference experiments seeking to elicit behavioural valuations of journey attributes. Transportation Research Part B 41 (9), 966–980. Fowkes, A.S., Wardman, M.R., 1988. The design of stated preference travel choice experiments with particular regard to inter-personal taste variations. Journal of Transport Economics and Policy 22, 27–44. Fowkes, A.S., Wardman, M., Holden, D.G.P., 1993. Non-orthogonal stated preference design. In: Proceedings of the PTRC Summer Annual Meeting, pp. 91–97. Garrod, G.D., Scarpa, R., Willis, K.G., 2002. Estimating the benefits of traffic calming on through routes: a choice experiment approach. Transportation 36 (2), 211–232. Greene, W.H., Hensher, D.A., Rose, J.M., 2006. Accounting for heterogeneity in the variance of unobserved effects in mixed logit models. Transportation Research Part B 40 (1), 75–92. Greene, W.H., Hensher, D.A., 2003. A latent class model for discrete choice analysis: contrasts with mixed logit. Transportation Research Part B 37 (8), 681– 698. Hensher, D.A., 2001a. Measurement of the valuation of travel time savings. Transportation 35 (1), 71–98. Hensher, D.A., 2001b. The valuation of commuter travel time savings for car drivers, evaluating alternative model specifications. Transportation 28 (2), 101– 118. Hensher, D.A., 2004. Identifying the influence of stated choice design dimensionality on willingness to pay for travel time savings. Transportation 38 (3), 425–446. Hensher, D.A., 2006. Towards a practical method to establish comparable values of travel time savings from stated choice experiments with differing design dimensions. Transportation Research Part A 42 (10), 829–840. Hensher, D.A., 2008a. Influence of vehicle occupancy on the valuation of car driver’s travel time savings: identifying important behavioural segments. Transportation Research Part A 42 (1), 67–76. Hensher, D.A., 2008b. Joint estimation of process and outcome in choice experiments and implications for willingness to pay. Transportation 42 (2), 297– 322. Hensher, D.A., Greene, W.H., 2002. Specification and estimation of the nested logit model: alternative normalisations. Transportation Research Part B 36 (1), 1–17. Hensher, D.A., Greene, W.H., 2003. The mixed logit model: the state of practice. Transportation 30 (2), 133–176. Hensher, D.A., King, J., 2001. Parking demand and responsiveness to supply, pricing and location in the Sydney central business district. Transportation Research Part A 35 (3), 177–196. Hensher, D.A., Prioni, P., 2002. A service quality index for area-wide contract performance assessment. Transportation 36 (1), 93–114. Hensher, D.A., Rose, J.M., 2007. Development of commuter and non-commuter mode choice models for the assessment of new public transport infrastructure projects: a case study. Transportation Research Part A 40 (1), 75–92. Hensher, D.A., Rose, J.M., Black, I., 2008. Interactive agency choice in automobile purchase decisions: the role of negotiation in determining equilibrium choice outcomes. Transportation 42 (2), 269–296. Hensher, D.A., Rose, J.M., Ortúzar, J de D., Rizzi, L.I., 2009. Estimating the willingness to pay and value of risk reduction for car occupants in the road environment. Transportation Research Part A 43 (7), 692–707. Hess, S., Daly, A., Rohr, C., Hyman, G., 2007. On the development of time period and mode choice models for use in large scale modelling forecasting systems. Transportation Research Part A 41 (9), 802–826. Hess, S., Rose, J.M., 2009. Allowing for intra-respondent variations in coefficients estimated on repeated choice data. Transportation Research Part B 43 (6), 708–719. Hess, S., Smith, C., Falzarano, S., Stubits, J., 2008. Measuring the effects of different experimental designs and survey administration methods using an Atlanta managed lanes stated preference survey. Transportation Research Record 2049, 144–152. Hollander, Y., 2006. Direct versus indirect models for the effects of unreliability. Transportation Research Part A 42 (9), 699–711. Huber, J., Zwerina, K., 1996. The importance of utility balance and efficient choice designs. Journal of Marketing Research 33 (August), 307–317. Hunt, J.D., Abraham, J.E., 2007. Influences on bicycle use. Transportation 34 (4), 453–470. Jovicic, G., Hansen, C.O., 2003. A passenger travel demand model for Copenhagen. Transportation Research Part A 37 (4), 333–349. Kanninen, B.J., 2002. Optimal design for multinomial choice experiments. Journal of Marketing Research 39 (May), 214–217. Kessels, R., Goos, P., Vandebroek, M., 2006. A comparison of criteria to design efficient choice experiments. Journal of Marketing Research 43 (3), 409–419. Kessels, R., Jones, B., Goos, P., Vandebroek, M., 2008. Recommendations on the use of Bayesian optimal designs for choice experiments. Quality and Reliability Engineering International 24, 737–744. Lee, J., Cho, Y., 2009. Demand forecasting of diesel passenger car considering consumer preference and government regulation in South Korea. Transportation Research Part A 43 (4), 420–429. Leitham, S., McQuaid, R.W., Nelson, J.D., 2000. The influence of transport on industrial location choice: a stated preference experiment. Transportation Research Part A 34 (7), 515–535.

M.C.J. Bliemer, J.M. Rose / Transportation Research Part A 45 (2011) 63–79

79

Loo, B.P.Y., Wong, S.C., Hau, T.D., 2006. Introducing alternative fuel vehicles in Hong Kong: views from the public light bus industry. Transportation 33 (6), 605–619. Louviere, J.J., Hensher, D.A., Swait, J.D., 2000. Stated Choice Methods: Analysis and Application. Cambridge University Press, Cambridge. Louviere, J.J., Islam, T., Wasi, N., Street, D., Burgess, L., 2008. Designing discrete choice experiments: do optimal designs come at a price? Journal of Consumer Research 35 (2), 360–375. Louviere, J.J., Woodworth, G., 1983. Design and analysis of simulated consumer choice or allocation experiments: an approach based on aggregate data. Journal of Marketing Research 20, 350–367. McDonnell, S., Ferriera, S., Convery, F., 2009. Bus priority provision and willingness to pay differentials resulting from modal choice and residential location: evidence from a stated choice survey. Transportation 43 (2), 213–234. McFadden, D., 1974. Conditional logit analysis of qualitative choice behaviour. In: Zarembka, P. (Ed.), Frontiers of Econometrics. Academic Press, New York, pp. 05–142. Ortúzar, J de D., Iacobelli, A., Valeze, C., 2000. Estimating demand for a cycle-way network. Transportation Research Part A 34 (5), 353–373. Pucket, S.M., Hensher, D.A., Rose, J.M., Collins, A., 2007a. Agency decision making in freight distribution chains: establishing a parsimonious empirical framework from alternative behavioural structures. Transportation Research Part B 41 (9), 924–949. Pucket, S.M., Hensher, D.A., Rose, J.M., Collins, A., 2007b. Design and development of a stated choice experiment for interdependent agents: accounting for interactions between buyers and sellers of urban freight services. Transportation 34 (4), 429–451. Puckett, S.M., Hensher, D.A., 2009. Revealing the extent of process heterogeneity in choice analysis: an empirical assessment. Transportation Research Part A 43 (4), 429–451. Rizzi, L.I., Ortuzar, J de D., 2006. Road safety valuation under a stated choice framework. Transportation 40 (1), 69–94. Rose, J.M., Bliemer, M.C.J., 2008. Stated preference experimental design strategies. In: Hensher, D.A., Button, K.J. (Eds.), Handbook of Transport Modelling. Elsevier, Oxford, pp. 151–180 (Chapter 8). Rose, J.M., Bliemer, M.C.J., 2009. Constructing efficient stated choice designs. Transport Reviews 29 (5), 587–617. Rose, J.M., Bliemer, M.C.J., Hensher, D.A., Collins, A., 2008. Designing efficient stated choice experiments in the presence of reference alternatives. Transportation Research Part B 42, 395–406. Rose, J.M., Scarpa, R., Bliemer, M.C.J., 2009. Incorporating model uncertainty into the generation of efficient stated choice experiments: a model averaging approach. In: International Choice Modelling Conference March 30–April 1, Yorkshire, UK. Rouwendal, J., de Blaeij, A., Rietveld, P., Verhoef, E., 2010. The information content of a stated choice experiment: a new method and its application to the value of a statistical life. Transportation Research Part B 44 (1), 136–151. Saelensminde, K., 2001. Inconsistent choices in stated choice data; use of the logit scaling approach to handle resulting variance increases. Transportation 28 (3), 269–296. Saleh, W., Farrell, S., 2005. Implications of congestion charging for departure time choice: work and non-work schedule flexibility. Transportation Research Part A 39 (7–9), 773–791. Sándor, Z., Wedel, M., 2001. Designing conjoint choice experiments using managers’ prior beliefs. Journal of Marketing Research 38 (November), 430–444. Sándor, Z., Wedel, M., 2002. Profile construction in experimental choice designs for mixed logit models. Marketing Science 21 (4), 455–475. Sándor, Z., Wedel, M., 2005. Heterogeneous conjoint choice designs. Journal of Marketing Research 42, 210–218. Scarpa, R., Rose, J.M., 2008. Designs efficiency for non-market valuation with choice modelling: how to measure it, what to report and why. Australian Journal of Agricultural and Resource Economics 52 (3), 253–282. Scott, D.M., Axhausen, K.W., 2006. Household mobility tool ownership: modeling interactions between cars and season tickets. Transportation 33 (4), 311– 328. Sener, I.N., Eluru, N., Bhat, C.R., 2009. An analysis of bicycle route choice preferences in Texas, US. Transportation 36 (5), 511–539. Street, D.J., Burgess, L., 2004. Optimal and near-optimal pairs for the estimation of effects in 2-level choice experiments. Journal of Statistical Planning and Inference 118, 185–199. Teichert, T., Shehu, E., von Wartburg, I., 2008. Customer segmentation revisited: the case of the airline industry. Transportation Research Part A 42 (1), 227– 242. Tilahun, N.Y., Levinson, D.M., Krizek, K.J., 2007. Trails, lanes, or traffic: valuing bicycle facilities with an adaptive stated preference survey. Transportation Research Part A 41 (4), 287–301. Toner, J.P., Clark, S.D., Grant-Muller, S.M., Fowkes, A.S., 1998. Anything you can do, we can do better: a provocative introduction to a new approach to stated preference design. In: WCTR Proceedings, Antwerp 3, pp. 107–120. Toner, J.P., Wardman, M., Whelan, G., 1999. Testing recent advances in stated preference design. In: Proceedings of the European Transport Conference, Cambridge, UK. Train, K., 2009. Discrete Choice Methods with Simulation, second ed. Cambridge University Press, Cambridge. Train, K., Wilson, W.W., 2008. Estimation on stated-preference experiments constructed from revealed-preference choices. Transportation Research Part B 42 (3), 191–203. Tseng, Y.Y., Verhoef, E.T., 2008. Value of time by time of day: a stated-preference study. Transportation Research Part B 42 (7–8), 607–618. Tudela, A., Rebolledo, G., 2006. Optimal design of stated preference experiments when using mixed logit models. In: Proceedings of the European Transport Conference, The Netherlands. Viney, R., Savage, E., Louviere, J., 2005. Empirical investigation of experimental design properties of discrete choice experiments in health care. Health Economics 14 (4), 349–362. Wang, B., Hensher, D.A., Ton, T., 2002. Safety in the road environment: a driver behavioural response perspective. Transportation 29 (3), 253–270. Wang, D., Borgers, A., Oppewal, H., Timmermans, H., 2000. A stated choice approach to developing multi-faceted models of activity behavior. Transportation Research Part A 34 (8), 625–643. Washbrook, K., Wolfgang, H., Jaccard, M., 2006. Estimating commuter mode choice: a discrete choice analysis of the impact of road pricing and parking charges. Transportation 33 (6), 621–639. Zhang, J., Timmermans, H., Borgers, A., Wang, D., 2004. Modeling traveler choice behavior using the concepts of relative utility and relative interest. Transportation Research Part B 38 (3), 215–234.