- Email: [email protected]
Obtaining more information from conjoint experiments by best–worst choices
Computational Statistics and Data Analysis 54 (2010) 1426–1433
Contents lists available at ScienceDirect
Computational Statistics and Data Analysis journal homepage: www.elsevier.com/locate/csda
Obtaining more information from conjoint experiments by best–worst choices Bart Vermeulen a,∗ , Peter Goos b,c , Martina Vandebroek a,d a
Katholieke Universiteit Leuven, Faculty of Business and Economics, Naamsestraat 69, 3000 Leuven, Belgium
b
Universiteit Antwerpen, Faculty of Applied Economics, Prinsstraat 13, 2000 Antwerpen, Belgium
c
Erasmus Universiteit Rotterdam, Erasmus School of Economics, Postbus 1738, 3000, DR Rotterdam, The Netherlands
d
Leuven Statistics Research Centre, Naamsestraat 69, 3000 Leuven, Belgium
article
info
Article history: Received 19 May 2009 Received in revised form 2 January 2010 Accepted 2 January 2010 Available online 18 January 2010 Keywords: Bayesian optimal design Best–worst choices Maximum-difference model Conjoint analysis D-optimality
abstract Conjoint choice experiments elicit individuals’ preferences for the attributes of a good by asking respondents to indicate repeatedly their most preferred alternative in a number of choice sets. However, conjoint choice experiments can be used to obtain more information than that revealed by the individuals’ single best choices. A way to obtain extra information is by means of best–worst choice experiments in which respondents are asked to indicate not only their most preferred alternative but also their least preferred one in each choice set. To create D-optimal designs for these experiments, an expression for the Fisher information matrix for the maximum-difference model is developed. Semi-Bayesian D-optimal best–worst choice designs are derived and compared with commonly used design strategies in marketing in terms of the D-optimality criterion and prediction accuracy. Finally, it is shown that best–worst choice experiments yield considerably more information than choice experiments. © 2010 Elsevier B.V. All rights reserved.
1. Introduction When using conjoint choice experiments to elicit individuals’ preferences for the characteristics of a good or a service, respondents are asked to indicate repeatedly their preferred alternative in a number of choice sets consisting of different profiles, each of which describes a good or a service by means of its underlying attributes. Analyzing these choice data with a conditional logit model informs the researcher which attributes are important in the individuals’ purchase decision. However, conjoint experiments can easily be used to obtain more information than that given by the respondents’ single best choices. Two frequently encountered ways to increase the information content of conjoint experiments are rating and ranking the alternatives in each choice set. In rating experiments, the respondents are asked to rate on a scale the different profiles of each choice set. However, the scores assigned to the profiles are highly subjective and therefore not comparable across respondents (Boyle et al., 2001). In ranking experiments, the respondents are asked to rank all or a number of the profiles in each choice set in decreasing order of preference. The main disadvantage of these types of experiments is that the reliability of the rankings decreases with every ranking step. Ben-Akiva et al. (1992) state that more than four ranking steps possibly yields unreliable data because of the respondents’ inability to distinguish between the alternatives left in the choice set after this number of steps. Nevertheless, ranking and rating experiments offer the researcher considerably more information than experiments which only consider the respondents’ single best choices (Boyle et al., 2001; Vermeulen et al., 2007).
∗
Corresponding author. Tel.: +32 16 32 69 63; fax: +32 16 32 66 24. E-mail addresses: [email protected] (B. Vermeulen), [email protected] (P. Goos), [email protected] (M. Vandebroek). 0167-9473/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.csda.2010.01.002
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1427
Best–worst choices are another way to obtain more information from conjoint experiments. In these experiments, the respondents are asked to indicate the most and least preferred alternative in each choice set. Obtaining this extra information avoids the necessity to use a large number of choice sets in a conjoint choice experiment, causing declining attention of the respondent and so, yielding less reliable data. In a number of applications (e.g. Louviere et al. (2004) and Lancsar et al. (2007)), the best–worst choice task is repeated until all alternatives are ranked. However, the experiments considered in this paper are best–worst choice experiments in which the respondents are asked to indicate only once their most and least preferred alternative in each choice set. An example of this type of conjoint task can be found in Mueller et al. (2009) where respondents had to choose their most and least favorite wine in a choice set of size four. Each wine varied in terms of the alcohol level, oak and yeast flavour. An orthogonal design with 14 choice sets was developed and each respondent had to evaluate two choice sets such that seven respondents were required to cover the whole design. In total, 112 respondents were recruited to participate in the experiment. Other examples where respondents indicate the best and worst option in each choice set only once are found in Hein et al. (2008) and Dejaeger et al. (2008) which illustrates the increasing popularity of best–worst choice tasks in nutrition and food research and product development. Best–worst choice experiments take advantage of individuals’ propensity to identify extreme options more consistently. That is why best–worst choice experiments are easier for people to complete than ranking and rating tasks (Marley and Louviere, 2005). To our knowledge, there are few papers discussing the disadvantages and costs of best–worst choice experiments in terms of e.g. cognitive burden, time needed to complete the experiment or fatigue. Cohen and Orme (2004) found that more time is needed to complete a best–worst choice experiment than a rating experiment. The reason for this is not formally established although the authors state that less involvement and thoughts are required for a rating experiment. In a follow-up study on best–worst choice experiments described in Dejaeger et al. (2008), respondents state that this type of experiment is easier than a rating experiment. Nevertheless, further research is needed to examine the consequences of best–worst choice experiments in terms of simplicity of the choice tasks and the cognitive burden. Setting up a conjoint experiment is a difficult task because the number of possible alternatives increases exponentially with the number of attributes and their numbers of levels in the study. Following Sándor and Wedel (2001, 2005) and Kessels et al. (2006, 2009), we use the optimal experimental design theory to select the profiles to be used in the experiment and to bundle these in choice sets. In this paper, we show how the optimal experimental design theory can be used to set up best–worst choice experiments, and we compare the resulting designs with various alternatives from the literature in terms of the D-optimality criterion and prediction accuracy. The remainder of this paper is structured as follows. The next section is devoted to the maximum-difference model, which is a logit model developed to analyze the best–worst choices of the respondents. Next, we develop an expression for the Fisher information matrix for the maximum-difference model and explain how to generate an optimal design for best–worst choice experiments yielding the most precisely estimated utility coefficients. Then, we compare the resulting design in terms of the newly developed best–worst optimality criterion with several other design strategies commonly used in the marketing literature. Next, we perform a simulation study to assess all designs in terms of prediction accuracy. Finally, we quantify the amount of extra information obtained from best–worst choice experiments and compare it with the amount obtained from a partial rank-order conjoint choice experiment requiring the respondents to indicate the two most preferred alternatives in a choice set. 2. The maximum-difference model Consider an experiment in which the respondents have to indicate their most and least preferred alternative out of J possibilities in choice set k. Marley and Louviere (2005) recently formalized a number of probabilistic models for best–worst choice experiments differing in the way the best and worst choices are made. In this paper, we follow the approach where it is assumed that the respondent decides simultaneously and independently which alternative is the best one and which alternative is the worst one. The probability of choosing alternative j as the best option in choice set k is given by pBkj =
exp(x0kj β) J P
,
(1)
exp(x0ki β)
i=1
while the probability of choosing that alternative as the worst option equals pWkj =
exp(−x0kj β) J P
,
(2)
exp(−xki β) 0
i =1
where the q-dimensional vector xkj represents the attribute levels of alternative j in choice set k and the q-dimensional vector β contains the utility coefficients reflecting the importance of the q attributes of the good. The vector β is assumed to be equal for all respondents. As the choices of the best and worst option are made simultaneously and independently, they might be equal. In that particular case, the respondent reconsiders his or her choice until the two options are distinct. Then, the joint probability of
1428
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
choosing alternative j as the best and alternative j0 as the worst option in choice set k is given by pBWkjj0 =
J ∞ X X s=0
=
!s
i =1
J ∞ X X s=0
!s pBkj pWkj0 ,
pBki pWki
i =1
= pBkj pWkj0
∞ X
1−
,
,
J
i=1 i0 =1,i0 6=i
pBki pWki0
i=1 i0 =1,i6=i0
pBkj pWkj0 J P P
!s
J J X X
s =0
=
p(j is best and j0 is worst),
p(i is best and worst)
(3)
pBki pWki0
which is equal to pBWkjj0 =
exp (xkj − xkj0 )0 β J J P P i=1 i0 =1,i0 6=i
exp ((xki −
.
(4)
) β)
xki0 0
A shortcoming of the maximum-difference model is that it does not have consistent margins as
P
j0
pBWkjj0 6= pBkj and
P
j pBWkjj0 6= pWkj0 . Nevertheless, the model leads to only slightly biased estimates of the utility coefficients (Marley and Louviere, 2005). The log-likelihood function for K choice sets and N respondents can be written as
ln (Lk (β)) =
J J N X K X X X
ynkii0 ln pBWkii0 ,
n=1 k=1 i=1 i0 =1,i0 6=i
(5)
where ynkii0 is a dummy variable which equals one if respondent n chooses alternative i and i0 in choice set k as the most and least preferred alternative, respectively, and zero otherwise. Maximizing (5) yields the maximum likelihood estimates of the utility coefficients β. 3. Designs for the maximum-difference model In this section, we first develop an expression for the Fisher information matrix for the maximum-difference model. This allows us to derive the D-optimality criterion for this model and to define the corresponding D-error. Next, we give further details on how to create semi-Bayesian D-optimal designs for best–worst choice experiments. The benchmark designs, which will be compared with the D-optimal best–worst choice design, are presented in Section 3.2. 3.1. Semi-Bayesian D-optimal designs for best–worst choice experiments A D-optimal design minimizes the generalized variance of the utility coefficients estimates. Given that the determinant of the asymptotic variance–covariance matrix, measuring the generalized variance, is inversely proportional to the determinant of the Fisher information matrix, D-optimal designs maximize the information on the unknown utility coefficients. Consequently, to construct D-optimal designs, an expression for the Fisher information matrix for the maximum-difference model is required. Starting from expression (5), it can be shown that this information matrix takes the following form 0
IBW (X , β) = X (PBW − pBW p0BW )X ,
(6)
with
X = x¯ 112
...
x¯ 11J
...
x¯ kjj0
...
0
x¯ K ,J ,J −1 ,
a J (J − 1)K -dimensional vector consisting of q-dimensional vectors x¯ kjj0 containing the difference between the q attribute levels of profile j and j0 in choice set k,
pBW = pBW112
...
pBW11J
...
pBWkjj0
...
pBWK ,J ,J −1,
0
,
a J (J − 1)K -dimensional vector consisting of the probabilities pBWkjj0 of indicating alternative j and j0 as the most and least preferred alternative, respectively, in choice set k, and PBW a diagonal matrix with the elements of pBW on its main diagonal.
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1429
The performance of a design in terms of the D-optimality criterion is measured by the D-error, which is defined as D-error = det(IBW (X , β))
− 1q
.
(7)
A design which minimizes the D-error is called D-optimal. As can be seen in expression (7), the D-error depends on the value of the utility coefficients in β which we wish to estimate by the data from the experiment. As a result, to find an optimal design for a best–worst choice experiment, prior knowledge about the unknown model coefficients is required. This problem is solved here by using a semi-Bayesian approach in which a prior distribution π (β) is used to indicate which are the likely values of the utility coefficients (Sándor and Wedel, 2001, 2005; Kessels et al., 2006). The expected D-error over the prior distribution is then given by Db -error = Eβ [det(IBW (X , β))
− 1q
Z ]=
det(IBW (X , β))
− 1q
π (β)dβ.
(8)
The integral in expression (8) cannot be computed analytically and will be approximated by taking 1000 pseudo-random draws from the prior distribution π (β) and by averaging the D-error over these draws. The design minimizing the average D-error over these draws is the semi-Bayesian D-optimal best–worst choice design. To construct a semi-Bayesian D-optimal best–worst choice design, the coordinate exchange algorithm was applied: this algorithm exchanges the levels of all attributes of the profiles with all the possible values and chooses the level corresponding to the best value of the Db -error. We refer the interested reader to Meyer and Nachtsheim (1995) for more details about the coordinate exchange algorithm and to Kessels et al. (2009) for an application of it to conjoint choice experiments. We report results for optimal designs for an experiment consisting of nine choice sets of size four and involving three three-level attributes and two two-level attributes which are all effects-type coded. In case of a two-level attribute, the first level is coded as +1 and the second level is coded as −1. In case of a three-level attribute, the first level is coded as [ 1 0 ], the second level as [ 0 1 ] and the third level as [ −1 −1 ]. This means that the same coding as in Kessels et al. (2008) is used. Since we have three three-level attributes and two two-level attributes, the total number of unknown parameters in our model equals eight. As we develop semi-Bayesian D-optimal designs, we need to specify a prior distribution on the utility coefficients. We assume that these coefficients [β1,1 β1,2 β2,1 β2,2 β3,1 β3,2 β4,1 β5,1 ] come from an 8-dimensional normal distribution with mean [−1.5 0 −1.5 0 −1.5 0 −1.5 −1.5]. The first six elements of the mean vector correspond to the utility coefficients of the three three-level attributes, while the last two elements correspond to the two two-level attributes. The prior mean implies that the utility increases with the attribute levels and that the respondents exhibit a high response accuracy. Additionally, we assume that the prior variance for every coefficient equals 0.5. Finally, we suppose that the correlation between any two parameters is zero except when the parameters correspond to the same attribute. In that case, we assume a correlation of −0.5 between these parameters. The correlation between the coefficients associated with the same attribute remediates the problem raised in the discussion in Kessels et al. (2008). It ensures that the parameter of the reference level can be estimated with the same precision as the parameters of the other levels (see Kessels et al. (2008) for more details). In addition to searching a semi-Bayesian D-optimal best–worst choice design, we create a locally D-optimal best–worst choice design with the attribute coefficients set to zero. By doing this, we assume that the respondents are indifferent between the levels of the attributes and so, experience no additional utility in case of higher attribute levels. We refer to this design as the utility-neutral best–worst choice design. 3.2. Benchmark designs We compare the semi-Bayesian D-optimal best–worst choice design and the utility-neutral best–worst choice design with the semi-Bayesian D-optimal choice design, the utility-neutral choice design and three standard designs commonly used in marketing research. To create semi-Bayesian D-optimal choice designs, we followed the approach of Sándor and Wedel (2001, 2005) and Kessels et al. (2006). Analogous to utility-neutral best–worst choice designs, utility-neutral choice designs correspond to locally D-optimal choice designs with the utility coefficients set to zero. The three standard designs were all generated using Sawtooth Software. The first standard design is a (nearly) orthogonal design obtained by the design option ‘complete enumeration’. This option considers all possible profiles and constructs a design exhibiting a minimum attribute level overlap between the profiles in a set and approximating an orthogonal design as closely as possible. The second standard design we consider is a random design created by randomly choosing attribute levels for all the alternatives in the choice sets. This might result in a strong attribute level overlap within one choice set. Finally, the third option, denoted as ‘balanced attribute level overlap’, is the middle course between the two former options because it allows a limited attribute level overlap within one choice set. 4. Evaluation of the designs We first assess the designs in terms of the D-error. Then, we evaluate the different designs in terms of their ability to predict the choice probabilities of choosing the best alternative accurately.
1430
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
Fig. 1. D-efficiencies of the semi-Bayesian D-optimal best–worst choice design relative to the benchmark designs. Values larger than one indicate improvements over the benchmark designs.
4.1. D-error To evaluate the performance of the D-optimal best–worst choice design XBW , we computed its D-efficiency relative to the benchmark designs for the 1000 parameter vectors β used to approximate the prior distribution. That relative D-efficiency is defined as rel. D − eff. =
det(IBW (Xbench. , β))−1/q det(IBW (XBW , β))−1/q
,
(9)
where Xbench denotes one of the benchmark designs. The relative D-efficiency is larger than one whenever the D-error of the semi-Bayesian D-optimal best–worst choice design is smaller than that of the benchmark design, i.e. whenever the semi-Bayesian D-optimal best–worst choice design performs better than the benchmark design in terms of the D-optimality criterion. For each of the six benchmark designs, we ranked the 1000 relative D-efficiencies from small to large and plotted them against their rank. The six plots we obtained in this fashion are shown in Fig. 1. The plots clearly visualize in what fraction of the parameter space the semi-Bayesian D-optimal best–worst choice design outperforms the benchmark design. The top two plots in Fig. 1 show the performance of the semi-Bayesian D-optimal best–worst choice design relative to the random and the balanced overlap design. The middle two plots compare the semi-Bayesian D-optimal best–worst choice design to the semi-Bayesian D-optimal choice design and the utility-neutral D-optimal best–worst choice design. Finally, the bottom two plots in Fig. 1 show the efficiency of the semi-Bayesian D-optimal best–worst choice design relative to the utility-neutral D-optimal choice design and the orthogonal design. The relative D-efficiencies lie well above one in five of the six plots, indicating that for nearly every parameter vector β the semi-Bayesian D-optimal best–worst choice design outperforms the corresponding benchmark designs. It is only for the comparison with the semi-Bayesian D-optimal choice design that the relative efficiencies do not deviate much from one, which suggests that there is not much difference between the semi-Bayesian D-optimal best–worst choice design and the semi-Bayesian D-optimal choice design when the goal is merely to estimate the model parameters.
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1431
Fig. 2. ASEP values for the semi-Bayesian D-optimal best–worst choice design compared to those for six benchmark designs.
Finally, it should be pointed out that the top-left plot in Fig. 1 was created using only one random design. That plot is representative for other randomly generated designs we tried in our study. 4.2. Prediction accuracy Based on 1000 data sets of simulated observations for 50 respondents, we estimated the utility coefficients of the maximum-difference model. Using the coefficient estimates, the choice probabilities for each alternative in nine randomly generated choice sets were computed. Comparing these predicted probabilities with the probabilities based on the ‘‘true’’ utility coefficients (used to simulate the data) allowed us to evaluate the predictive performance of the designs. We quantified the prediction error using the average squared error of prediction ASEP (β) =
1
1000 X
1000 r =1
(p(βˆ r ) − p(β))0 (p(βˆ r ) − p(β)),
(10)
ˆ r and β are vectors containing the estimated and true parameter values, and p(βˆ r ) and p(β) contain the choice where β probabilities for each of the four alternatives in the nine randomly generated choice sets. Obviously, small ASEP values are preferred over large ones. Importantly, we did these computations for the semi-Bayesian D-optimal best–worst choice design and for each of the six benchmark designs, and for 100 different values of the ‘‘true’’ parameter vector β. This resulted in seven sets of 100 ASEP values, which allows us to compare the predictive performance of the semi-Bayesian D-optimal best–worst design and the six benchmark designs. This is done in Fig. 2, which has the same structure as Fig. 1. Each scatter plot in Fig. 2 compares one of the benchmark designs to the semi-Bayesian D-optimal best–worst design. The ASEP values of the benchmark are displayed on the vertical axis, whereas those of the semi-Bayesian D-optimal best–worst design are shown on the horizontal axis. In this kind of plot, the points are symmetrically spread around the bisector when the two designs compared have a similar performance. When the benchmark design is inferior, then the majority of the points lies above the bisector. In five of the six plots in Fig. 2, the vast majority of the points lies above the bisector, so that the semi-Bayesian D-optimal best–worst design is the better design in these comparisons. The predictions obtained from the semi-Bayesian D-optimal
1432
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
Table 1 Db -error of the different designs in the case of a best choice experiment, a partial rank-order conjoint experiment and a best–worst choice experiment with different sizes for all nine choice sets in the case of a multivariate normal prior distribution with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75]. Design
4 alternatives Best choice
5 alternatives Part. rank.
6 alternatives
Best–worst
Best choice
Part. rank.
Best–worst
Best choice
Part. rank.
Best–worst
0.1217 0.1688 0.1328 0.1070
0.2184 0.3096 0.2775 0.2368
0.1253 0.1555 0.1160 0.1298
0.1128 0.1495 0.1254 0.1037
0.2115 0.2973 0.2787 0.2255
0.1195 0.1509 0.1104 0.1229
0.1144 0.1483 0.1218 0.1007
0.1575 0.2044 0.1730 0.1513
0.2953 0.3798 0.3667 0.3101
0.1575 0.1834 0.1437 0.1596
0.1443 0.1799 0.1664 0.1399
0.2780 0.3584 0.353 0.3002
0.1453 0.1736 0.1336 0.1521
0.1349 0.1718 0.1602 0.1319
Distribution π1 (β) with var(βi ) = 0.04 Choice Orth. Part. Rank. Best–worst
0.2318 0.3825 0.3040 0.2521
0.1396 0.1851 0.1196 0.1415
Distribution π2 (β) with var(βi ) = 0.5 Choice Orth. Part. Rank. Best–worst
0.3254 0.4745 0.3805 0.3470
0.1744 0.2232 0.1531 0.1775
choice design, however, are comparable to those from the semi-Bayesian D-optimal best–worst design. Thus, also in terms of predictive performance, these two designs are close competitors. We also considered the case where the utility coefficients used to develop the designs and to generate the data come from an 8-dimensional normal distribution with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75] and a variance–covariance structure identical to that we used before. The results for this scenario indicate a smaller difference between the benchmark designs and the semi-Bayesian D-optimal best–worst choice design in terms of estimation and prediction accuracy. However, for reasons of brevity of the paper, these results are not displayed. 5. Extra information obtained from an additional choice In this section, we investigate how much additional information can be obtained compared to a conjoint choice experiment from an additional choice in each choice set, i.e. a second best choice in a partial rank-order conjoint experiment or the least preferred choice in a best–worst choice experiment. To do so, we compare the Db -error, which measures the amount of information, of a nearly orthogonal, a choice, a ranking and a best–worst choice design used for a best choice, a best–worst choice and a partial rank-order conjoint experiment. The difference between the Db -error of a design used for a best choice experiment and the Db -error of this design for a best–worst choice experiment gives an idea about the information obtained from the additional choice of the least preferred alternative in each choice set. The information obtained from the additional choice of the second best option is given by the difference between the Db -error of a design for a best choice experiment and the Db -error of a design for a partial rank-order conjoint experiment. Moreover, comparing the Db -error of a best–worst choice design and the Db -error of a ranking design used for their corresponding experiments allows us to determine which experiment yields most information. Notice that the choice, the best–worst choice and the ranking designs are generated in this section in a semi-Bayesian fashion using the D-optimality criterion. We refer the reader to Sándor and Wedel (2001, 2005) and Kessels et al. (2006) for the generation of choice designs and to Vermeulen et al. (2007) for the development of ranking designs. In this section, we consider experiments consisting of nine choice sets of size four, five and six, each time involving three three-level and two two-level attributes. To generate semi-Bayesian D-optimal designs to be compared, we assumed that the utility coefficients come from two 8-dimensional normal distributions with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75]. The first normal distribution we used had variance 0.04 for each model coefficient, and the correlation pattern we discussed before. The second normal distribution we used had variance 0.5 for each coefficient. We name these prior distributions π1 (β) and π2 (β). Table 1 displays the Db -errors of the four designs used for best choice, partial rank-ordered conjoint and best–worst choice experiments. The Db -error was calculated assuming the prior that was used to optimize the designs. The numbers in bold represent the Db -errors of the optimal designs for a particular type of experiment. Table 1 demonstrates that the Db -error of the designs decreases with 45% to 60% by having an extra choice in each choice set. This means that asking respondents to make an extra choice, either a second best or a worst choice, yields a substantial amount of additional information compared to an experiment with only single best choices. Furthermore, Table 1 shows that the Db -error of the best–worst choice design is similar to the Db -error of the ranking design when they are used in their corresponding experiments. This indicates that when respondents are asked to choose two alternatives in each choice set, indicating the most and least preferred alternative yields more or less the same amount of information on the respondent’s preferences as the choice of the best and the second best alternative. Remark that choosing the best and worst option in a choice set consisting of three alternatives corresponds with ranking all alternatives of this choice set. Table 1 reveals that the Db -error of the ranking design used for a rank-order conjoint experiment is smaller than the Db -error of the orthogonal design used for this type of experiment. A similar observation can be made for the Db -error of
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1433
the best–worst choice design used for a best–worst choice experiment which is smaller than the Db -error of the orthogonal design. This means that it is worthwhile to develop tailor-made designs for best–worst choice and partial rank-order conjoint experiments rather than using an orthogonal design as is often done in practice (see, e.g., Louviere et al. (2004), Ahn et al. (2006) and Baarsma (2003)). Notice that a semi-Bayesian D-optimal choice design yields comparable D-errors to the semiBayesian D-optimal designs for rank-order conjoint and best–worst choice experiments when used in these corresponding experiments. Finally, it can be seen that for all experiments considered, the Db -error of the choice design is smaller than that of an orthogonal design. This demonstrates that it is always better to a use a choice design than an orthogonal design. Notice as well that the best–worst choice design used in a best choice experiment yields a smaller Db -error than the ranking design. This means that when a respondent fails to give a second choice, the best–worst choice design results in more information than a ranking design. 6. Conclusion In best–worst choice experiments, the respondents are asked to indicate their most and least preferred alternative in a number of choice sets to elicit individuals’ preferences for the attributes of a good. In this way, the researcher is able to obtain more information than that unveiled by the respondents’ single best choices. This paper tackles the design issue by showing how to generate semi-Bayesian D-optimal best–worst choice designs yielding the highest efficiency in estimating the utility coefficients. For that purpose, we develop an expression for the Fisher information matrix for the maximumdifference model which is used to analyze the data coming from such an experiment. We compare the semi-Bayesian D-optimal best–worst choice design in terms of the D-optimality criterion and prediction accuracy with a semi-Bayesian D-optimal choice design and three standard designs, i.e. an orthogonal design, a random design and a design exhibiting limited attribute level overlap within one choice set. The results of this study reveal that a best–worst choice design leads to the most accurately estimated utility coefficients and predictions, but the semi-Bayesian D-optimal choice design is a very close competitor. Best–worst choice experiments give the researcher considerably more information about individuals’ preferences than traditional best choice experiments. Moreover, it was shown that indicating the least preferred alternative in each choice set gives more or less the same amount of information as indicating the second best option. Taking into account the fact that a best–worst choice task is easier to complete for respondents than a rating or ranking task, the use of best–worst choice experiments is certainly a valuable option to elicit individuals’ preferences. Acknowledgement Bart Vermeulen is funded by the project G.0611.05 of the Fund for Scientific Research Flanders. References Ahn, J., Lee, J., Lee, J.-D., Kim, T.-Y., 2006. An analysis of consumer preferences among wireless LAN and mobile internet services. Electronics and Telecommunications Research Institute 28, 205–215. Baarsma, B., 2003. The valuation of the Ijmeer nature reserve using conjoint analysis. Environmental and Resource Economics 25, 343–356. Ben-Akiva, M., Morikawa, T., Shiroishi, F., 1992. Analysis of the reliability of preference ranking data. Journal of Business Research 37, 149–164. Boyle, K., Holmes, T., Teisl, M., Roe, B., 2001. A comparison of conjoint analysis response formats. American Journal of Agricultural Economics 83, 441–454. Cohen, S., Orme, B., 2004. What’s your preference? Marketing Research 16, 32–37. Dejaeger, S., Jorgensen, A., Aaslyung, M., Bredie, W., 2008. Best-worst scaling: An introduction and initial comparison with monadic rating for preference elicitation with food products. Food Quality and Preference 19, 579–588. Hein, K., Dejaeger, S., Carr, B., Delahunty, C., 2008. Comparison of five common acceptance and preference methods. Food Quality and Preference 19, 651–661. Kessels, R., Goos, P., Vandebroek, M., 2006. A comparison of criteria to design efficient choice experiments. Journal of Marketing Research 43 (3), 409–419. Kessels, R., Jones, B., Goos, P., Vandebroek, M., 2008. Recommendations on the use of Bayesian optimal designs for choice experiments. Quality and Reliability Engineering International 24, 737–744. Kessels, R., Jones, B., Goos, P., Vandebroek, M., 2009. An efficient algorithm for constructing Bayesian optimal choice designs. Journal of Business and Economic Statistics 27, 279–291. Lancsar, E., Louviere, J., Flynn, T., 2007. Several methods to investigate relative attribute impact in stated preference experiments. Social Science and Medicine 64, 1738–1753. Louviere, J., Burgess, L., Street, D., Marley, A., 2004. Modelling the choices of single individuals by combining efficient choice experiment designs with extra preference information. Centre for the Study of Choice (CenSoC), Working Paper No. 04-005. Marley, A., Louviere, J., 2005. Some probabilistic models of best, worst and best–worst choices. Journal of Mathematical Psychology 49, 464–480. Meyer, R., Nachtsheim, C., 1995. The coordinate-exchange algorithm for constructing exact optimal experimental designs. Technometrics 37, 60–69. Mueller, S., Francis, I., Lockshin, L., 2009. Comparison of best–worst and hedonic scaling for the measurement of consumer wine preferences. Australian Journal of Grape and Wine Research 15, 205–215. Sándor, Z., Wedel, M., 2001. Designing conjoint choice experiments using managers’ prior beliefs. Journal of Marketing Research 38, 430–444. Sándor, Z., Wedel, M., 2005. Heterogeneous conjoint choice designs. Journal of Marketing Research 42, 210–218. Vermeulen, B., Goos, P., Vandebroek, M., 2007. Rank-order conjoint experiments: Efficiency and design. Research Report KBI_0734, Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, p. 30.
Contents lists available at ScienceDirect
Computational Statistics and Data Analysis journal homepage: www.elsevier.com/locate/csda
Obtaining more information from conjoint experiments by best–worst choices Bart Vermeulen a,∗ , Peter Goos b,c , Martina Vandebroek a,d a
Katholieke Universiteit Leuven, Faculty of Business and Economics, Naamsestraat 69, 3000 Leuven, Belgium
b
Universiteit Antwerpen, Faculty of Applied Economics, Prinsstraat 13, 2000 Antwerpen, Belgium
c
Erasmus Universiteit Rotterdam, Erasmus School of Economics, Postbus 1738, 3000, DR Rotterdam, The Netherlands
d
Leuven Statistics Research Centre, Naamsestraat 69, 3000 Leuven, Belgium
article
info
Article history: Received 19 May 2009 Received in revised form 2 January 2010 Accepted 2 January 2010 Available online 18 January 2010 Keywords: Bayesian optimal design Best–worst choices Maximum-difference model Conjoint analysis D-optimality
abstract Conjoint choice experiments elicit individuals’ preferences for the attributes of a good by asking respondents to indicate repeatedly their most preferred alternative in a number of choice sets. However, conjoint choice experiments can be used to obtain more information than that revealed by the individuals’ single best choices. A way to obtain extra information is by means of best–worst choice experiments in which respondents are asked to indicate not only their most preferred alternative but also their least preferred one in each choice set. To create D-optimal designs for these experiments, an expression for the Fisher information matrix for the maximum-difference model is developed. Semi-Bayesian D-optimal best–worst choice designs are derived and compared with commonly used design strategies in marketing in terms of the D-optimality criterion and prediction accuracy. Finally, it is shown that best–worst choice experiments yield considerably more information than choice experiments. © 2010 Elsevier B.V. All rights reserved.
1. Introduction When using conjoint choice experiments to elicit individuals’ preferences for the characteristics of a good or a service, respondents are asked to indicate repeatedly their preferred alternative in a number of choice sets consisting of different profiles, each of which describes a good or a service by means of its underlying attributes. Analyzing these choice data with a conditional logit model informs the researcher which attributes are important in the individuals’ purchase decision. However, conjoint experiments can easily be used to obtain more information than that given by the respondents’ single best choices. Two frequently encountered ways to increase the information content of conjoint experiments are rating and ranking the alternatives in each choice set. In rating experiments, the respondents are asked to rate on a scale the different profiles of each choice set. However, the scores assigned to the profiles are highly subjective and therefore not comparable across respondents (Boyle et al., 2001). In ranking experiments, the respondents are asked to rank all or a number of the profiles in each choice set in decreasing order of preference. The main disadvantage of these types of experiments is that the reliability of the rankings decreases with every ranking step. Ben-Akiva et al. (1992) state that more than four ranking steps possibly yields unreliable data because of the respondents’ inability to distinguish between the alternatives left in the choice set after this number of steps. Nevertheless, ranking and rating experiments offer the researcher considerably more information than experiments which only consider the respondents’ single best choices (Boyle et al., 2001; Vermeulen et al., 2007).
∗
Corresponding author. Tel.: +32 16 32 69 63; fax: +32 16 32 66 24. E-mail addresses: [email protected] (B. Vermeulen), [email protected] (P. Goos), [email protected] (M. Vandebroek). 0167-9473/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.csda.2010.01.002
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1427
Best–worst choices are another way to obtain more information from conjoint experiments. In these experiments, the respondents are asked to indicate the most and least preferred alternative in each choice set. Obtaining this extra information avoids the necessity to use a large number of choice sets in a conjoint choice experiment, causing declining attention of the respondent and so, yielding less reliable data. In a number of applications (e.g. Louviere et al. (2004) and Lancsar et al. (2007)), the best–worst choice task is repeated until all alternatives are ranked. However, the experiments considered in this paper are best–worst choice experiments in which the respondents are asked to indicate only once their most and least preferred alternative in each choice set. An example of this type of conjoint task can be found in Mueller et al. (2009) where respondents had to choose their most and least favorite wine in a choice set of size four. Each wine varied in terms of the alcohol level, oak and yeast flavour. An orthogonal design with 14 choice sets was developed and each respondent had to evaluate two choice sets such that seven respondents were required to cover the whole design. In total, 112 respondents were recruited to participate in the experiment. Other examples where respondents indicate the best and worst option in each choice set only once are found in Hein et al. (2008) and Dejaeger et al. (2008) which illustrates the increasing popularity of best–worst choice tasks in nutrition and food research and product development. Best–worst choice experiments take advantage of individuals’ propensity to identify extreme options more consistently. That is why best–worst choice experiments are easier for people to complete than ranking and rating tasks (Marley and Louviere, 2005). To our knowledge, there are few papers discussing the disadvantages and costs of best–worst choice experiments in terms of e.g. cognitive burden, time needed to complete the experiment or fatigue. Cohen and Orme (2004) found that more time is needed to complete a best–worst choice experiment than a rating experiment. The reason for this is not formally established although the authors state that less involvement and thoughts are required for a rating experiment. In a follow-up study on best–worst choice experiments described in Dejaeger et al. (2008), respondents state that this type of experiment is easier than a rating experiment. Nevertheless, further research is needed to examine the consequences of best–worst choice experiments in terms of simplicity of the choice tasks and the cognitive burden. Setting up a conjoint experiment is a difficult task because the number of possible alternatives increases exponentially with the number of attributes and their numbers of levels in the study. Following Sándor and Wedel (2001, 2005) and Kessels et al. (2006, 2009), we use the optimal experimental design theory to select the profiles to be used in the experiment and to bundle these in choice sets. In this paper, we show how the optimal experimental design theory can be used to set up best–worst choice experiments, and we compare the resulting designs with various alternatives from the literature in terms of the D-optimality criterion and prediction accuracy. The remainder of this paper is structured as follows. The next section is devoted to the maximum-difference model, which is a logit model developed to analyze the best–worst choices of the respondents. Next, we develop an expression for the Fisher information matrix for the maximum-difference model and explain how to generate an optimal design for best–worst choice experiments yielding the most precisely estimated utility coefficients. Then, we compare the resulting design in terms of the newly developed best–worst optimality criterion with several other design strategies commonly used in the marketing literature. Next, we perform a simulation study to assess all designs in terms of prediction accuracy. Finally, we quantify the amount of extra information obtained from best–worst choice experiments and compare it with the amount obtained from a partial rank-order conjoint choice experiment requiring the respondents to indicate the two most preferred alternatives in a choice set. 2. The maximum-difference model Consider an experiment in which the respondents have to indicate their most and least preferred alternative out of J possibilities in choice set k. Marley and Louviere (2005) recently formalized a number of probabilistic models for best–worst choice experiments differing in the way the best and worst choices are made. In this paper, we follow the approach where it is assumed that the respondent decides simultaneously and independently which alternative is the best one and which alternative is the worst one. The probability of choosing alternative j as the best option in choice set k is given by pBkj =
exp(x0kj β) J P
,
(1)
exp(x0ki β)
i=1
while the probability of choosing that alternative as the worst option equals pWkj =
exp(−x0kj β) J P
,
(2)
exp(−xki β) 0
i =1
where the q-dimensional vector xkj represents the attribute levels of alternative j in choice set k and the q-dimensional vector β contains the utility coefficients reflecting the importance of the q attributes of the good. The vector β is assumed to be equal for all respondents. As the choices of the best and worst option are made simultaneously and independently, they might be equal. In that particular case, the respondent reconsiders his or her choice until the two options are distinct. Then, the joint probability of
1428
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
choosing alternative j as the best and alternative j0 as the worst option in choice set k is given by pBWkjj0 =
J ∞ X X s=0
=
!s
i =1
J ∞ X X s=0
!s pBkj pWkj0 ,
pBki pWki
i =1
= pBkj pWkj0
∞ X
1−
,
,
J
i=1 i0 =1,i0 6=i
pBki pWki0
i=1 i0 =1,i6=i0
pBkj pWkj0 J P P
!s
J J X X
s =0
=
p(j is best and j0 is worst),
p(i is best and worst)
(3)
pBki pWki0
which is equal to pBWkjj0 =
exp (xkj − xkj0 )0 β J J P P i=1 i0 =1,i0 6=i
exp ((xki −
.
(4)
) β)
xki0 0
A shortcoming of the maximum-difference model is that it does not have consistent margins as
P
j0
pBWkjj0 6= pBkj and
P
j pBWkjj0 6= pWkj0 . Nevertheless, the model leads to only slightly biased estimates of the utility coefficients (Marley and Louviere, 2005). The log-likelihood function for K choice sets and N respondents can be written as
ln (Lk (β)) =
J J N X K X X X
ynkii0 ln pBWkii0 ,
n=1 k=1 i=1 i0 =1,i0 6=i
(5)
where ynkii0 is a dummy variable which equals one if respondent n chooses alternative i and i0 in choice set k as the most and least preferred alternative, respectively, and zero otherwise. Maximizing (5) yields the maximum likelihood estimates of the utility coefficients β. 3. Designs for the maximum-difference model In this section, we first develop an expression for the Fisher information matrix for the maximum-difference model. This allows us to derive the D-optimality criterion for this model and to define the corresponding D-error. Next, we give further details on how to create semi-Bayesian D-optimal designs for best–worst choice experiments. The benchmark designs, which will be compared with the D-optimal best–worst choice design, are presented in Section 3.2. 3.1. Semi-Bayesian D-optimal designs for best–worst choice experiments A D-optimal design minimizes the generalized variance of the utility coefficients estimates. Given that the determinant of the asymptotic variance–covariance matrix, measuring the generalized variance, is inversely proportional to the determinant of the Fisher information matrix, D-optimal designs maximize the information on the unknown utility coefficients. Consequently, to construct D-optimal designs, an expression for the Fisher information matrix for the maximum-difference model is required. Starting from expression (5), it can be shown that this information matrix takes the following form 0
IBW (X , β) = X (PBW − pBW p0BW )X ,
(6)
with
X = x¯ 112
...
x¯ 11J
...
x¯ kjj0
...
0
x¯ K ,J ,J −1 ,
a J (J − 1)K -dimensional vector consisting of q-dimensional vectors x¯ kjj0 containing the difference between the q attribute levels of profile j and j0 in choice set k,
pBW = pBW112
...
pBW11J
...
pBWkjj0
...
pBWK ,J ,J −1,
0
,
a J (J − 1)K -dimensional vector consisting of the probabilities pBWkjj0 of indicating alternative j and j0 as the most and least preferred alternative, respectively, in choice set k, and PBW a diagonal matrix with the elements of pBW on its main diagonal.
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1429
The performance of a design in terms of the D-optimality criterion is measured by the D-error, which is defined as D-error = det(IBW (X , β))
− 1q
.
(7)
A design which minimizes the D-error is called D-optimal. As can be seen in expression (7), the D-error depends on the value of the utility coefficients in β which we wish to estimate by the data from the experiment. As a result, to find an optimal design for a best–worst choice experiment, prior knowledge about the unknown model coefficients is required. This problem is solved here by using a semi-Bayesian approach in which a prior distribution π (β) is used to indicate which are the likely values of the utility coefficients (Sándor and Wedel, 2001, 2005; Kessels et al., 2006). The expected D-error over the prior distribution is then given by Db -error = Eβ [det(IBW (X , β))
− 1q
Z ]=
det(IBW (X , β))
− 1q
π (β)dβ.
(8)
The integral in expression (8) cannot be computed analytically and will be approximated by taking 1000 pseudo-random draws from the prior distribution π (β) and by averaging the D-error over these draws. The design minimizing the average D-error over these draws is the semi-Bayesian D-optimal best–worst choice design. To construct a semi-Bayesian D-optimal best–worst choice design, the coordinate exchange algorithm was applied: this algorithm exchanges the levels of all attributes of the profiles with all the possible values and chooses the level corresponding to the best value of the Db -error. We refer the interested reader to Meyer and Nachtsheim (1995) for more details about the coordinate exchange algorithm and to Kessels et al. (2009) for an application of it to conjoint choice experiments. We report results for optimal designs for an experiment consisting of nine choice sets of size four and involving three three-level attributes and two two-level attributes which are all effects-type coded. In case of a two-level attribute, the first level is coded as +1 and the second level is coded as −1. In case of a three-level attribute, the first level is coded as [ 1 0 ], the second level as [ 0 1 ] and the third level as [ −1 −1 ]. This means that the same coding as in Kessels et al. (2008) is used. Since we have three three-level attributes and two two-level attributes, the total number of unknown parameters in our model equals eight. As we develop semi-Bayesian D-optimal designs, we need to specify a prior distribution on the utility coefficients. We assume that these coefficients [β1,1 β1,2 β2,1 β2,2 β3,1 β3,2 β4,1 β5,1 ] come from an 8-dimensional normal distribution with mean [−1.5 0 −1.5 0 −1.5 0 −1.5 −1.5]. The first six elements of the mean vector correspond to the utility coefficients of the three three-level attributes, while the last two elements correspond to the two two-level attributes. The prior mean implies that the utility increases with the attribute levels and that the respondents exhibit a high response accuracy. Additionally, we assume that the prior variance for every coefficient equals 0.5. Finally, we suppose that the correlation between any two parameters is zero except when the parameters correspond to the same attribute. In that case, we assume a correlation of −0.5 between these parameters. The correlation between the coefficients associated with the same attribute remediates the problem raised in the discussion in Kessels et al. (2008). It ensures that the parameter of the reference level can be estimated with the same precision as the parameters of the other levels (see Kessels et al. (2008) for more details). In addition to searching a semi-Bayesian D-optimal best–worst choice design, we create a locally D-optimal best–worst choice design with the attribute coefficients set to zero. By doing this, we assume that the respondents are indifferent between the levels of the attributes and so, experience no additional utility in case of higher attribute levels. We refer to this design as the utility-neutral best–worst choice design. 3.2. Benchmark designs We compare the semi-Bayesian D-optimal best–worst choice design and the utility-neutral best–worst choice design with the semi-Bayesian D-optimal choice design, the utility-neutral choice design and three standard designs commonly used in marketing research. To create semi-Bayesian D-optimal choice designs, we followed the approach of Sándor and Wedel (2001, 2005) and Kessels et al. (2006). Analogous to utility-neutral best–worst choice designs, utility-neutral choice designs correspond to locally D-optimal choice designs with the utility coefficients set to zero. The three standard designs were all generated using Sawtooth Software. The first standard design is a (nearly) orthogonal design obtained by the design option ‘complete enumeration’. This option considers all possible profiles and constructs a design exhibiting a minimum attribute level overlap between the profiles in a set and approximating an orthogonal design as closely as possible. The second standard design we consider is a random design created by randomly choosing attribute levels for all the alternatives in the choice sets. This might result in a strong attribute level overlap within one choice set. Finally, the third option, denoted as ‘balanced attribute level overlap’, is the middle course between the two former options because it allows a limited attribute level overlap within one choice set. 4. Evaluation of the designs We first assess the designs in terms of the D-error. Then, we evaluate the different designs in terms of their ability to predict the choice probabilities of choosing the best alternative accurately.
1430
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
Fig. 1. D-efficiencies of the semi-Bayesian D-optimal best–worst choice design relative to the benchmark designs. Values larger than one indicate improvements over the benchmark designs.
4.1. D-error To evaluate the performance of the D-optimal best–worst choice design XBW , we computed its D-efficiency relative to the benchmark designs for the 1000 parameter vectors β used to approximate the prior distribution. That relative D-efficiency is defined as rel. D − eff. =
det(IBW (Xbench. , β))−1/q det(IBW (XBW , β))−1/q
,
(9)
where Xbench denotes one of the benchmark designs. The relative D-efficiency is larger than one whenever the D-error of the semi-Bayesian D-optimal best–worst choice design is smaller than that of the benchmark design, i.e. whenever the semi-Bayesian D-optimal best–worst choice design performs better than the benchmark design in terms of the D-optimality criterion. For each of the six benchmark designs, we ranked the 1000 relative D-efficiencies from small to large and plotted them against their rank. The six plots we obtained in this fashion are shown in Fig. 1. The plots clearly visualize in what fraction of the parameter space the semi-Bayesian D-optimal best–worst choice design outperforms the benchmark design. The top two plots in Fig. 1 show the performance of the semi-Bayesian D-optimal best–worst choice design relative to the random and the balanced overlap design. The middle two plots compare the semi-Bayesian D-optimal best–worst choice design to the semi-Bayesian D-optimal choice design and the utility-neutral D-optimal best–worst choice design. Finally, the bottom two plots in Fig. 1 show the efficiency of the semi-Bayesian D-optimal best–worst choice design relative to the utility-neutral D-optimal choice design and the orthogonal design. The relative D-efficiencies lie well above one in five of the six plots, indicating that for nearly every parameter vector β the semi-Bayesian D-optimal best–worst choice design outperforms the corresponding benchmark designs. It is only for the comparison with the semi-Bayesian D-optimal choice design that the relative efficiencies do not deviate much from one, which suggests that there is not much difference between the semi-Bayesian D-optimal best–worst choice design and the semi-Bayesian D-optimal choice design when the goal is merely to estimate the model parameters.
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1431
Fig. 2. ASEP values for the semi-Bayesian D-optimal best–worst choice design compared to those for six benchmark designs.
Finally, it should be pointed out that the top-left plot in Fig. 1 was created using only one random design. That plot is representative for other randomly generated designs we tried in our study. 4.2. Prediction accuracy Based on 1000 data sets of simulated observations for 50 respondents, we estimated the utility coefficients of the maximum-difference model. Using the coefficient estimates, the choice probabilities for each alternative in nine randomly generated choice sets were computed. Comparing these predicted probabilities with the probabilities based on the ‘‘true’’ utility coefficients (used to simulate the data) allowed us to evaluate the predictive performance of the designs. We quantified the prediction error using the average squared error of prediction ASEP (β) =
1
1000 X
1000 r =1
(p(βˆ r ) − p(β))0 (p(βˆ r ) − p(β)),
(10)
ˆ r and β are vectors containing the estimated and true parameter values, and p(βˆ r ) and p(β) contain the choice where β probabilities for each of the four alternatives in the nine randomly generated choice sets. Obviously, small ASEP values are preferred over large ones. Importantly, we did these computations for the semi-Bayesian D-optimal best–worst choice design and for each of the six benchmark designs, and for 100 different values of the ‘‘true’’ parameter vector β. This resulted in seven sets of 100 ASEP values, which allows us to compare the predictive performance of the semi-Bayesian D-optimal best–worst design and the six benchmark designs. This is done in Fig. 2, which has the same structure as Fig. 1. Each scatter plot in Fig. 2 compares one of the benchmark designs to the semi-Bayesian D-optimal best–worst design. The ASEP values of the benchmark are displayed on the vertical axis, whereas those of the semi-Bayesian D-optimal best–worst design are shown on the horizontal axis. In this kind of plot, the points are symmetrically spread around the bisector when the two designs compared have a similar performance. When the benchmark design is inferior, then the majority of the points lies above the bisector. In five of the six plots in Fig. 2, the vast majority of the points lies above the bisector, so that the semi-Bayesian D-optimal best–worst design is the better design in these comparisons. The predictions obtained from the semi-Bayesian D-optimal
1432
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
Table 1 Db -error of the different designs in the case of a best choice experiment, a partial rank-order conjoint experiment and a best–worst choice experiment with different sizes for all nine choice sets in the case of a multivariate normal prior distribution with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75]. Design
4 alternatives Best choice
5 alternatives Part. rank.
6 alternatives
Best–worst
Best choice
Part. rank.
Best–worst
Best choice
Part. rank.
Best–worst
0.1217 0.1688 0.1328 0.1070
0.2184 0.3096 0.2775 0.2368
0.1253 0.1555 0.1160 0.1298
0.1128 0.1495 0.1254 0.1037
0.2115 0.2973 0.2787 0.2255
0.1195 0.1509 0.1104 0.1229
0.1144 0.1483 0.1218 0.1007
0.1575 0.2044 0.1730 0.1513
0.2953 0.3798 0.3667 0.3101
0.1575 0.1834 0.1437 0.1596
0.1443 0.1799 0.1664 0.1399
0.2780 0.3584 0.353 0.3002
0.1453 0.1736 0.1336 0.1521
0.1349 0.1718 0.1602 0.1319
Distribution π1 (β) with var(βi ) = 0.04 Choice Orth. Part. Rank. Best–worst
0.2318 0.3825 0.3040 0.2521
0.1396 0.1851 0.1196 0.1415
Distribution π2 (β) with var(βi ) = 0.5 Choice Orth. Part. Rank. Best–worst
0.3254 0.4745 0.3805 0.3470
0.1744 0.2232 0.1531 0.1775
choice design, however, are comparable to those from the semi-Bayesian D-optimal best–worst design. Thus, also in terms of predictive performance, these two designs are close competitors. We also considered the case where the utility coefficients used to develop the designs and to generate the data come from an 8-dimensional normal distribution with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75] and a variance–covariance structure identical to that we used before. The results for this scenario indicate a smaller difference between the benchmark designs and the semi-Bayesian D-optimal best–worst choice design in terms of estimation and prediction accuracy. However, for reasons of brevity of the paper, these results are not displayed. 5. Extra information obtained from an additional choice In this section, we investigate how much additional information can be obtained compared to a conjoint choice experiment from an additional choice in each choice set, i.e. a second best choice in a partial rank-order conjoint experiment or the least preferred choice in a best–worst choice experiment. To do so, we compare the Db -error, which measures the amount of information, of a nearly orthogonal, a choice, a ranking and a best–worst choice design used for a best choice, a best–worst choice and a partial rank-order conjoint experiment. The difference between the Db -error of a design used for a best choice experiment and the Db -error of this design for a best–worst choice experiment gives an idea about the information obtained from the additional choice of the least preferred alternative in each choice set. The information obtained from the additional choice of the second best option is given by the difference between the Db -error of a design for a best choice experiment and the Db -error of a design for a partial rank-order conjoint experiment. Moreover, comparing the Db -error of a best–worst choice design and the Db -error of a ranking design used for their corresponding experiments allows us to determine which experiment yields most information. Notice that the choice, the best–worst choice and the ranking designs are generated in this section in a semi-Bayesian fashion using the D-optimality criterion. We refer the reader to Sándor and Wedel (2001, 2005) and Kessels et al. (2006) for the generation of choice designs and to Vermeulen et al. (2007) for the development of ranking designs. In this section, we consider experiments consisting of nine choice sets of size four, five and six, each time involving three three-level and two two-level attributes. To generate semi-Bayesian D-optimal designs to be compared, we assumed that the utility coefficients come from two 8-dimensional normal distributions with mean [−0.75 0 −0.75 0 −0.75 0 −0.75 −0.75]. The first normal distribution we used had variance 0.04 for each model coefficient, and the correlation pattern we discussed before. The second normal distribution we used had variance 0.5 for each coefficient. We name these prior distributions π1 (β) and π2 (β). Table 1 displays the Db -errors of the four designs used for best choice, partial rank-ordered conjoint and best–worst choice experiments. The Db -error was calculated assuming the prior that was used to optimize the designs. The numbers in bold represent the Db -errors of the optimal designs for a particular type of experiment. Table 1 demonstrates that the Db -error of the designs decreases with 45% to 60% by having an extra choice in each choice set. This means that asking respondents to make an extra choice, either a second best or a worst choice, yields a substantial amount of additional information compared to an experiment with only single best choices. Furthermore, Table 1 shows that the Db -error of the best–worst choice design is similar to the Db -error of the ranking design when they are used in their corresponding experiments. This indicates that when respondents are asked to choose two alternatives in each choice set, indicating the most and least preferred alternative yields more or less the same amount of information on the respondent’s preferences as the choice of the best and the second best alternative. Remark that choosing the best and worst option in a choice set consisting of three alternatives corresponds with ranking all alternatives of this choice set. Table 1 reveals that the Db -error of the ranking design used for a rank-order conjoint experiment is smaller than the Db -error of the orthogonal design used for this type of experiment. A similar observation can be made for the Db -error of
B. Vermeulen et al. / Computational Statistics and Data Analysis 54 (2010) 1426–1433
1433
the best–worst choice design used for a best–worst choice experiment which is smaller than the Db -error of the orthogonal design. This means that it is worthwhile to develop tailor-made designs for best–worst choice and partial rank-order conjoint experiments rather than using an orthogonal design as is often done in practice (see, e.g., Louviere et al. (2004), Ahn et al. (2006) and Baarsma (2003)). Notice that a semi-Bayesian D-optimal choice design yields comparable D-errors to the semiBayesian D-optimal designs for rank-order conjoint and best–worst choice experiments when used in these corresponding experiments. Finally, it can be seen that for all experiments considered, the Db -error of the choice design is smaller than that of an orthogonal design. This demonstrates that it is always better to a use a choice design than an orthogonal design. Notice as well that the best–worst choice design used in a best choice experiment yields a smaller Db -error than the ranking design. This means that when a respondent fails to give a second choice, the best–worst choice design results in more information than a ranking design. 6. Conclusion In best–worst choice experiments, the respondents are asked to indicate their most and least preferred alternative in a number of choice sets to elicit individuals’ preferences for the attributes of a good. In this way, the researcher is able to obtain more information than that unveiled by the respondents’ single best choices. This paper tackles the design issue by showing how to generate semi-Bayesian D-optimal best–worst choice designs yielding the highest efficiency in estimating the utility coefficients. For that purpose, we develop an expression for the Fisher information matrix for the maximumdifference model which is used to analyze the data coming from such an experiment. We compare the semi-Bayesian D-optimal best–worst choice design in terms of the D-optimality criterion and prediction accuracy with a semi-Bayesian D-optimal choice design and three standard designs, i.e. an orthogonal design, a random design and a design exhibiting limited attribute level overlap within one choice set. The results of this study reveal that a best–worst choice design leads to the most accurately estimated utility coefficients and predictions, but the semi-Bayesian D-optimal choice design is a very close competitor. Best–worst choice experiments give the researcher considerably more information about individuals’ preferences than traditional best choice experiments. Moreover, it was shown that indicating the least preferred alternative in each choice set gives more or less the same amount of information as indicating the second best option. Taking into account the fact that a best–worst choice task is easier to complete for respondents than a rating or ranking task, the use of best–worst choice experiments is certainly a valuable option to elicit individuals’ preferences. Acknowledgement Bart Vermeulen is funded by the project G.0611.05 of the Fund for Scientific Research Flanders. References Ahn, J., Lee, J., Lee, J.-D., Kim, T.-Y., 2006. An analysis of consumer preferences among wireless LAN and mobile internet services. Electronics and Telecommunications Research Institute 28, 205–215. Baarsma, B., 2003. The valuation of the Ijmeer nature reserve using conjoint analysis. Environmental and Resource Economics 25, 343–356. Ben-Akiva, M., Morikawa, T., Shiroishi, F., 1992. Analysis of the reliability of preference ranking data. Journal of Business Research 37, 149–164. Boyle, K., Holmes, T., Teisl, M., Roe, B., 2001. A comparison of conjoint analysis response formats. American Journal of Agricultural Economics 83, 441–454. Cohen, S., Orme, B., 2004. What’s your preference? Marketing Research 16, 32–37. Dejaeger, S., Jorgensen, A., Aaslyung, M., Bredie, W., 2008. Best-worst scaling: An introduction and initial comparison with monadic rating for preference elicitation with food products. Food Quality and Preference 19, 579–588. Hein, K., Dejaeger, S., Carr, B., Delahunty, C., 2008. Comparison of five common acceptance and preference methods. Food Quality and Preference 19, 651–661. Kessels, R., Goos, P., Vandebroek, M., 2006. A comparison of criteria to design efficient choice experiments. Journal of Marketing Research 43 (3), 409–419. Kessels, R., Jones, B., Goos, P., Vandebroek, M., 2008. Recommendations on the use of Bayesian optimal designs for choice experiments. Quality and Reliability Engineering International 24, 737–744. Kessels, R., Jones, B., Goos, P., Vandebroek, M., 2009. An efficient algorithm for constructing Bayesian optimal choice designs. Journal of Business and Economic Statistics 27, 279–291. Lancsar, E., Louviere, J., Flynn, T., 2007. Several methods to investigate relative attribute impact in stated preference experiments. Social Science and Medicine 64, 1738–1753. Louviere, J., Burgess, L., Street, D., Marley, A., 2004. Modelling the choices of single individuals by combining efficient choice experiment designs with extra preference information. Centre for the Study of Choice (CenSoC), Working Paper No. 04-005. Marley, A., Louviere, J., 2005. Some probabilistic models of best, worst and best–worst choices. Journal of Mathematical Psychology 49, 464–480. Meyer, R., Nachtsheim, C., 1995. The coordinate-exchange algorithm for constructing exact optimal experimental designs. Technometrics 37, 60–69. Mueller, S., Francis, I., Lockshin, L., 2009. Comparison of best–worst and hedonic scaling for the measurement of consumer wine preferences. Australian Journal of Grape and Wine Research 15, 205–215. Sándor, Z., Wedel, M., 2001. Designing conjoint choice experiments using managers’ prior beliefs. Journal of Marketing Research 38, 430–444. Sándor, Z., Wedel, M., 2005. Heterogeneous conjoint choice designs. Journal of Marketing Research 42, 210–218. Vermeulen, B., Goos, P., Vandebroek, M., 2007. Rank-order conjoint experiments: Efficiency and design. Research Report KBI_0734, Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, p. 30.