- Email: [email protected]
Reliability (or “lack thereof”) of on-line preference revelation: A controlled experimental analysis
Decision Support Systems 56 (2013) 270–274
Contents lists available at ScienceDirect
Decision Support Systems journal homepage: www.elsevier.com/locate/dss
Reliability (or “lack thereof”) of on-line preference revelation: A controlled experimental analysis☆,☆☆ Li Chen a, James R. Marsden b, Zhongju Zhang b,⁎ a b
Department of Management, School of Business and Economics, Fayetteville State University, 1200 Murchison Road, Fayetteville, NC 28301, United States Department of Operations and Information Management, School of Business, University of Connecticut, 2100 Hillside Road, Storrs, CT 06269, United States
a r t i c l e
i n f o
Article history: Received 13 May 2013 Accepted 13 June 2013 Available online 21 June 2013 Keywords: Preference revelation Information market Information revelation Information aggregation Controlled experiment Survey accuracy Survey reliability
a b s t r a c t Networks permitting anonymous contributions continue to expand and flourish. In some networks, the reliability of a contribution is not of particular importance. In other settings, however, the development of a network is driven by specific purposes which make the reliability of information exchanged of significant importance. One such situation involves the use of information markets for aggregating individuals' preferences on new or emerging technologies. At this point, there remains skepticism concerning the reliability of the preference revelations in such markets and thus the resulting preference aggregations and rankings of emerging technologies. In this paper, we study the reliability of on-line preference revelation using a series of controlled laboratory experiments. Our analysis includes individuals' pre- and post-experiment rankings of technologies, individual trading and accumulation activities during an electronics market experiment, the final experimental market outcomes, and a ranking of the same technologies by a panel of experts from a Fortune 5 company. In addition, as a final step, we allowed each participant to actually select and keep a unit of one of the technologies at zero price (free). That is, we were able to observe each participant's actual final true preference from the set of technologies. © 2013 Elsevier B.V. All rights reserved.
1. Introduction Networks permitting anonymous contributions continue to expand and flourish. In some networks, the reliability of a contribution is not of particular importance. Banter in a social media network may include tongue-in-cheek kidding or creation of a pseudo persona for participation in a second-life. In other settings, however, the development of a network is driven by specific purposes which make the reliability of information exchanged of significant importance. One such situation involves the implementation of networks for engaging individuals in real business applications such as the evaluation of new or emerging technologies [1], new product concepts [3,4,6], and idea generation [8]. Information preference markets (see discussion in [1]) have gained growing recognition as a relatively efficient process for gathering and aggregating information from a large number of dispersed individuals. However, there remains skepticism concerning the reliability of ☆ This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial-No Derivative Works License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and source are credited. ☆☆ The authors are grateful for the research support and funding provided by the Treibick Family Endowed Chair for CITI, the Gladstein Endowed MIS Research Lab, and the Connecticut Information Technology Institute. ⁎ Corresponding author. Tel.: +1 8604860660. E-mail addresses: [email protected] (L. Chen), [email protected] (J.R. Marsden), [email protected] (Z. Zhang). 0167-9236/$ – see front matter © 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.dss.2013.06.010
preference revelations (and thus the usefulness of the preference aggregations and decision rankings) gathered in such on-line markets. Chen et al. [2] argued that the assumption that traders are “truth-telling” might not be reasonable when an information market is relatively small. Van de Walle and Turoff [14] observed in their experiment that a large number of traders chose to follow market leaders instead of acting based on their private information. To date, however, we find no direct exploration of the accuracy of preference revelation, no testing of whether individuals really reveal their true preferences in preference markets. Further, we find no direct analysis of the consistency of individual preference revelations before, during, and/or after participation in preference markets. Here we report the structuring, operationalization, and outcomes of a series of controlled laboratory experiments addressing these issues. The ability to carefully structure the settings and observe all individual activities and market occurrences makes laboratory experimentation an attractive process to address the questions of interest (see [5,7,9,10]). Specifically, through controlled laboratory experiments, we seek to address each of the following research questions: (1) are individuals' actual revealed preferences consistent with initial reported preference, portfolio accumulation activities during the course of the experimental market, and/or post-experiment reported preference? (2) are individual true revealed preferences more closely aligned with peer activities during the experimental market and the resulting
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
final market outcomes than with the individual's stated preferences pre- and/or post-experiment? and, (3) is there any relationship between the number of market participants who accurately reveal their preference and the consistency between market outcomes and rankings from a panel of experts? The third question requires a bit of explanation. Chen et al. [1] explained in detail that their Fortune 5 partner company was not dis-satisfied with the results from the use of panels of experts. The issues for the company were the costs and the lack of scalability of panels of experts. Here we are not arguing whether or not panels of experts provide the best outputs. Rather, our experience suggests that the outcomes from panels of experts are valued by firms (Chen et al.'s [1] partner firm and others) and representative of an interesting comparison. The remainder of the paper is as follows. In Section 2, we briefly review related literature in preference markets and experimental economics. Section 3 is devoted to the development and discussion of the research questions analyzed in this note. In Section 4, we outline the design of our experimental market and detail the experimental techniques used to capture the participants' preference revelations over the course of the experimental process. Section 5 presents the findings of our controlled experiments. Section 6 summarizes our findings and suggests future research directions. 2. Literature review We use the term “on-line preference markets” to refer to virtual markets established to collect participants' preference on virtual stocks representing items of interest. Alternative historical methods used to capture such information include surveys, focus groups, critical success factors, and scoring models. Compared to such alternatives, on-line preference markets have the following advantages: (1) flexibility, (2) scalability, and (3) ability to incorporate and capture dynamic interactions among participants [4]. Gruca et al. [6] used such markets to forecast new product success, finding results similar to those of surveys but with reduced variability of results. LaComb et al. [8] utilized preference markets in generating and evaluating new product ideas. They reported increased idea generation and an expanded number of participants compared to conventional techniques. In a recent study of on-line preference market, Chen et al. [1] investigated the use of such markets for evaluating and ranking emerging technologies. The authors developed and implemented an on-line market for a major corporation with participants being employees with technology-related job responsibilities. Based on consistency between the preference market rankings and rankings from a company-selected panel of experts, the company decided to invest significant new funds in further development and use of on-line preference markets in the area of evaluating emerging technologies. As noted earlier, we are not arguing that panels of experts necessarily yield the best outcomes. Rather, our experience suggests that the outcomes from panels of experts are valued by firms (Chen et al.'s [1] partner firm and others) and representative of an interesting comparison. While on-line preference markets have begun to get traction in the business community, many questions remain. In the arena of prediction markets for predicting near future events, Wolfers and Zitzewitz [15] point out that earning through better prediction provides incentives for traders to reveal their true information. This is consistent with the experimental economics literature which emphasizes using significant performance based monetary incentive. In on-line preference markets, however, the outcomes are an aggregated ranking of preferences and performance based payoffs are not easily measurable. Developing performance-based reward metrics linked directly to observable outcomes is a thorny issue. In the area of emerging technologies, for example, the actual outcome of interest (commercial success or failure) cannot actually be determined in a short period of time. Any new product or emerging technology may be years away from
271
final implementation. The actual technologies or product evaluated is likely to be subsequently modified or morphed. Further, only a few, if any, of the products or technologies will move ahead to implementation. Thus, we are unable to evaluate those not selected and only able to evaluate those selected, if at all, sometime in the future. In preference markets, we actually have two measurement issues of concern. First, as Chen et al. [1] discussed in the area of emerging technologies, there is the question of whether the preferences observed and aggregated are sufficiently closely aligned with a panel of experts so that a company might use the scalable and more flexible preference markets to substitute for expense, non-scalable, and often inflexible panels of experts. Second, there is the important question of whether participants actually reveal their true preferences. While Chen et al. [1] focused on the former, in this paper we focus on the latter. Before detailing our experimental market design and operationalization in Section 4, we devote Section 3 to the presentation and development of a series of preference rankings in order to address the research questions discussed earlier. 3. Development of rankings for comparison As noted, earlier work by Chen et al. [1] detailed the structure of on-line preference market and presented the results and analysis of a full scale on-line preference market for emerging technologies. The market was developed and implemented for a Fortune 5 company. Commenting on the success of the implemented market, the authors noted, “Perhaps most importantly, our partner company viewed the study results as sufficient to warrant expanded investigation and use of on-line preference market in their early-stage emerging technology screening.” In that work, the authors' focus was at the market level and included analysis of price convergence under various conditions, overall trading activity (including overall activity of specific types of trades), and consistency of market outcomes with results from a panel of experts engaged by the partner company. The present analysis is focused at the individual level. As detailed in the next section, our research process, including controlled experimentation, provides a series of rankings from each participant on a set of four technologies. In addition to these individual rankings, we also have rankings from the experimental market outcomes (where “shares of stock” in the four technologies were traded) and those from a panel of experts that evaluated the four technologies. The panel of experts was enlisted from a technology evaluation community within the Fortune 5 company, providing a benchmark similar to that utilized in [1]. The set of available rankings are summarized below, where the index label, such as I or II corresponds to a time ordering when the rankings occur within the experimental process (the expert panel ranking has no time order since it is outside the experimental process — we arbitrarily assign this ranking the last value or “VI”): Individual rankings: I) the participant's rank ordering of technologies on a preexperiment survey (PreES); II) the participant's bids and transactions during the experiment (IMT); IV) the participant's rank ordering of technologies on a postexperiment survey (PostES); and, in a novel twist for this analysis, V) the participant's actual choice of a unit of one of the technologies (Actual). Experimental market rankings III) rankings determined by the market's final share prices of the four technologies (MP) Expert panel rankings VI) rankings from a panel of experts enlisted from a technology evaluation community within the Fortune 5 company (Expert)
272
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
Thus we have a series of indicators of each individual's preferences on a proffered set of technologies. We note that the fourth ranking by individuals is only a single ranking, that is, the actual first preference of each individual participant, i.e., the actual choice of one of the four technologies that an individual could choose and receive for free. The various rankings – information gathered before, during, and following the experiment – along with the information gathered from the panel of experts enabled a variety of consistency analyses. Fig. 1 depicts the six alternative information sets gathered. Rankings I–V are clearly time ordered. Ranking set VI (panel of experts) is not absolutely required to have the information gathered prior to the experiment, but that was the case in our work. The next section details our experimental process and how each of the rankings was operationalized. This is followed by a summary of experiment results and analysis of the outcomes. 4. The experiment process 4.1. Experiment design Controlled laboratory experiments were used to gather the data necessary to analyze our research questions. This approach has been thoroughly discussed and demonstrated, especially in the context of various market analyses (see [5,7,9,11–13]). Using the open-source software platform jMarkets (http://jmarkets.ssel.caltech.edu/), we constructed an initial experimental preference market design and trading process. A series of pilot experiments were conducted using a group of volunteer participants (Ph.D. students, student workers, and university staff), to check for any issues with system operation, participant understanding or use of the system, or other issues that might arise. Following the first several pilot runs, modifications were made to aid participant understanding (mainly through enhancements in depth and quality of explanatory materials provided to participants) and in screen design (mainly minor changes in location and detail of information presented on a participant's screen). Subsequent trial runs using a mix of earlier participants along with new volunteer participants resulted in a second set of modifications. A third and final run by several pilot study participants was completed to make sure issues had been addressed. Once the electronic market was constructed and pilot tested, we moved to the specifics of our experimental design. Based on past experiments, reports, and recommendations from experimental economics,
together with our research budget constraint, we decided on experiments with eight participants in each experiment. Anyone who had participated in pilot testing was excluded from participating in an actual experiment session. Given typical show-up payments ($10 per subject), overall reward-based session payments (averaging $10 per subject per session as explained below) to participants, and the cost of purchasing the technology products (slightly less than $20 per subject), our budget enabled eight sessions with eight participants in each session. The experiment and experimental process were reviewed and approved by the university's internal human subject review panel. We recruited 64 university students (24 graduate students and 40 undergraduate students). Of the 24 graduate students, 18 were from the School of Business, 1 from the Department of Computer Science and Engineering, and 5 from the Department of Economics. All the undergraduate students were from the School of Business. Each recruited participant was randomly assigned into one of the planned eight sessions. We verified that recruited subjects had no knowledge of or experience in preference markets. Instructions to participants were specific that they must agree to no communication with any other subject before or after their session. The preference market for each session involved four virtual stocks, each virtual stock representing a product that incorporated an emerging technology. We choose final product formats (with embedded emerging technology) because, in discussion with a variety of technologists and emerging technology specialists, the strong feeling was that the final product format made it easier for subjects to evaluate them. The products were chosen so that the incorporated emerging technologies were not competing technologies and the products were neither substitutes nor complements. The products we chose had only recently appeared and had not been widely seen in the marketplace. This choice was made to make it unlikely that participants would have existing experience with the selected products. In addition, we chose items of similar monetary value (approximately $20) to limit any effect on traders' evaluation linked to or influenced by the price/value of the item. 4.2. Experiment procedure Experiments were conducted in a controlled laboratory setting (the Gladstein Endowed MIS Research Lab). For each session, each
I. individual rankings from pre-experiment survey (PreES)
V. individual’s actual choice of a unit of one of the four technologies (Actual)
AVAILABLE TECHNOLOGY RANKINGS
II. individual trading activities during experiment (IMT)
VI. rankings from panel of experts (Expert) III. rankings from final market prices (MP) IV. individual rankings from post—experiment survey (PostES) Fig. 1. Six rankings gathered — set for which consistencies/inconsistencies are evaluated.
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
participant was assigned a unique ID. Individual participants were seated in front of a computer with no view of another participant's screen. We enforced a strict no-talking policy during the entire session. One author stayed in the lab during the course of each experiment. Once the eight participants for a session had been logged in, a short video (roughly five minutes) was presented to explain the trading mechanism they would utilize in the electronic preference market. The video was created as a means to ensure that participants in different sessions received consistent instructions and explanations. A hard copy of the same instruction was also provided to each participant. To further ensure the instructions were understood, after looking at the video and reading the detailed instruction participants were then given a short paper quiz consisting of five questions. When participants completed the short quiz, correct answers were reviewed with the participants and any questions or requests for clarification were answered. Details on the four emerging technology products were provided as web pages (one for each product) along with a hardcopy of product descriptions and details. We note that the instructions and explanations emphasized that the four products had been evaluated and rated by a group of emerging technology professionals at a major corporation. Participants were informed that they would earn performance rewards based on how their final stock portfolio matched up against the ratings of the set of emerging technology professionals (see details below). At this point of the session, we collected our first set of ranking information - each participant's pre-experiment preference rankings (PreES) of the four products. Subjects were asked to fill in a survey providing their preference orderings on the four products. Typically, the process involving watching the video, reading instructions, checking understanding, and completing the survey took approximately 15 min. Following completion of the pre-experiment survey, a brief (approximately 5 min) trial session was provided so participants could experience the preference market interfaces and trading mechanisms. The trial session incorporated random “phantom offers” so more trades would go through and help provide participants with an understanding of the processes. Following the trial session, the formal forty-minute experiment began. The trading mechanism followed a double auction framework that has been widely used in on-line preference markets [1,3,4]. Participants begin with an initial stock portfolio and a cash balance. At any point during the experiment, a participant with sufficient cash balance can submit a buy order. Similarly, with sufficient stock assets, a participant can submit a sell order. When another participant (with sufficient appropriate “assets” or “holdings”) accepts an existing buy or sell order, a transaction occurs. Transactions of partial volume are permitted.
4.3. Incentive details As noted above, there were two types of rewards in the experiment: a $10 payment for showing up and a dollar reward based on a participant's final portfolio value where the value is directly linked to the rating evaluations by a panel of emerging technology experts. To
273
determine the later rewards, we took the total reward payment available for each of the eight sessions ($80 per session) and divided this amount so that each share of stock in a given emerging technology product received a valuation based on the proportionate ranking by the professionals. Thus, in total, each participant received $10 for showing up and participating, a “final portfolio reward” equal to the sum of shares held of each stock times the weighted value for that stock (based on the expert panel ratings), and the final choice of one technology product to keep. Subjects were not aware of the last reward until after the completion of their market session. To encourage trading among traders, we utilized the rule that the any “cash” held was valued at zero in the final portfolio evaluation. The goal of rewarding participants with a reasonable show-up fee plus extra rewards related to their performance relative to expert ratings should help to induce serious market behavior (see the discussion in [7,11,12]). In the structure utilized, payments to participants ranged from $15 to $24 dollars. Following completion of an experiment, each participant was again provided with a survey form used to rank the four technology products (PostES). Following completion of the form and without any prior indication, each participant was allowed to select and keep one of the four technology products (Actual). Assuming rationality, this last selection step provides actual preference revelation of each individual's most preferred or top choice from the four technology products. In the next section, we analyze the “preference revelations” obtained from each of the participants, from the market, and from the expert panel. 5. Experiment results and analysis We compiled the six ranking sets indicated in Fig. 1: pre-experiment survey (PreES) for each individual, market transactions (IMT) for each individual, post-experiment survey (PostES) for each individual, actual technology product selection (Actual) for each individual, final market prices (MP) for each of the eight markets, and the expert panel ranking (Expert) for the four technology products. We consider each of our three research questions in turn. Table 1 presents the consistency rates between the first choice indicated at various stages and the actual choice or actual preference. (1) Do individuals show consistency in initial reported preference, portfolio accumulation (trading) activities during the course of the experimental market, post-experiment reported preference, and final actual demonstrated preference? The post-experiment survey has only a slightly higher consistency rate (73.44%) with the actual choice than does the pre-experiment (70.31%). That is, 70.3% of the participants revealed their true preference in the pre-survey rising to 73.4% in the post-survey which was taken after the individuals had been informed of the market outcomes. Both however, displayed greater consistency rates with actual choices than did trading activities (portfolio accumulations) whose consistency with the actual choice was 64.06%.
Table 1 Consistency rates between the first choices from five available rankings and the actual final individual choice (the actual individual revealed preference).
I. pre-market survey PreES II. market transactions IMT III. market outcomes MP IV. post-market survey PostES V. actual choice Actual VI. expert panel Expert
I. pre-market survey PreES
II. market transaction IMT
III. market outcomes MP
IV. post-market survey PostES
V. actual choice Actual
VI. expert panel Expert
100%
57.81% 100%
45.31% 60.94% 100%
59.38% 68.75% 59.38% 100%
70.31% 64.06% 53.13% 73.44% 100%
48.44% 50.00% 62.50% 54.69% 54.69% 100%
274
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
(2) Are individual true preferences impacted by peer activities during the experimental market and the resulting final market outcomes? The market outcome consistency rate with the post-market survey was 59.38%, an increase of 14.07% from the consistency rate of 45.31% between the pre-market survey and the market outcomes. This suggests some impact from peer activities during the experimental market. The market outcomes had a much lower consistency rate with actual choice (53.13%) than did the post-experiment survey with the actual choice (73.44%), but the market outcomes did have the highest consistency rate (62.5%) with the expert panel. Finally, the post-market survey and actual choices each had somewhat higher consistency rates (54.69%) with the expert panel than did the pre-market survey with the expert panel (48.44%). (3) Is there any relationship between the number of market participants who accurately reveal their preference and the consistency between market outcomes and rankings from a panel of experts? For the five groups whose market choice (MP) was consistent with the expert panel's top choice (Expert), 23 of 40 individuals or 57.5% had their Actual being the same as Expert. For the three groups whose market choice (MP) was inconsistent with the expert panel's top choice (Expert), 12 of 24 individuals or 50% had their Actual being the same as Expert. If a firm (such as the partner firm in [1]) has utilized panels of experts over time and their analysis indicates very positive outcomes, then consistency of the outcomes with those of the panel of experts is certainly a positive for the that firm. In general, the valuation of the performance of panels of experts is done only within a firm and we have no general or cross-firm detailed analysis available to us. Thus result (3) is likely of most interest to firms having had good results from the use of panels of experts.
6. Summary and Conclusions We summarize our experimental results as following: i) We found that just 70.3% of the participants revealed their true preference in the PreES and 73.4% did so in the PostES, a time point after the individuals had been informed by the market outcomes. Thus, even at a point immediately before individuals revealed their true preference by selecting one technology to keep (Actual), almost 27% misstated their true preference. ii) There was some market impact on individual stated preferences in the sense that consistency rates were higher between PostES and MP (59.38%) than between PreES and MP (45.31%). iii) The market outcomes did have the highest consistency rate (62.5%) with the expert panel. Further, only one of the markets failed to have the Expert top pick in the top two MP values. Of the results detailed in this research note, it is the first one that raises a general concern since it indicates that individuals alone cannot be counted on to accurately reveal their preferences. In structuring our analysis, we incorporated an opportunity for individuals to reveal their true preference by selecting and keeping as their own one of the four technologies. In the PreES, the failure rate for individuals to accurately match their Actual choice was just slightly below 30%. Even after participating in the electronic market and getting more time to think about the alternatives and observe information on the ratings (MP) from others, almost 27% still misstated the Actual preference in the PostES. The latter is even more striking when we recall that only a minute or so passed between completion of the PostES's and the Actual choice. These results suggest cause for concern when stated preferences rather than actual observed choice are used in analyses and decision making.
References [1] L. Chen, P. Goes, J.R. Marsden, J. Zhang, Design and use of on-line preference markets for evaluation of early stage technologies, Journal of Management Information Systems 26 (2009) 45–70. [2] Y. Chen, C. Chu, T. Mullen, Predicting uncertain outcomes using information markets: trader behavior and information aggregation, New Mathematics and Natural Computation 2 (2006) 281–297. [3] E. Dahan, A.J. Kim, A.W. Lo, T. Poggio, N. Chan, Securities trading of concepts (STOC), Journal of Marketing Research 48 (2011) 497–517. [4] E. Dahan, A. Soukhoroukova, M. Spann, New product development 2.0: preference markets. How scalable securities markets identify winning product concepts & attributes, Journal of Product Innovation Management 27 (2010) 937–954. [5] C.L. Gardner, J.R. Marsden, D.E. Pingry, The design and use of laboratory experiments for DSS evaluation, Decision Support Systems 9 (4) (1993) 369–379. [6] T. Gruca, J. Berg, M. Cipriano, The effect of electronic markets on forecasts of new product success, Information Systems Frontiers 5 (2003) 95–105. [7] E. Hoffman, J.R. Marsden, A. Whinston, Laboratory experiments and computer simulation: an introduction to the use of experimental and process model data in economic analysis, in: J. Kagel, L. Green (Eds.), Advances in Behavioral Economics, volume 2, Ablex Publishing, Norwood, NJ, 1990, pp. 1–31. [8] C.A. LaComb, J.A. Barnett, Q. Pan, The imagination market, Information Systems Frontiers 9 (2007) 245–256. [9] J.R. Marsden, R. Pakath, K. Wibowo, Decision making under time pressure with different information sources and performance-based financial incentives—part 1, Decision Support Systems 34 (1) (2002) 75–97. [10] J.R. Marsden, Y.T. Tung, The use of information system technology to develop tests on insider trading and asymmetric information, Management Science 45 (1999) 1025–1040. [11] V.L. Smith, Microeconomic systems as an experimental science, The American Economic Review 72 (1982) 923–955. [12] V.L. Smith, Experimental economics: induced value theory, The American Economic Review 66 (1976) 274–279. [13] M. Spann, B. Skiera, Internet-based virtual stock markets for business forecasting, Management Science 49 (2003) 1310–1326. [14] B. Van de Walle, M. Turoff, Fuzzy relations for the analysis of traders' preferences in an information market game, European Journal of Operational Research 195 (2009) 905–913. [15] J. Wolfers, E. Zitzewitz, Prediction markets, The Journal of Economic Perspectives 18 (2004) 107–126. Li Chen is an assistant professor of management in the Department of Management, School of Business and Economics at Fayetteville State University. He received his Ph.D. from the University of Connecticut. Dr. Chen's research interests include IT innovation, preference markets, sponsored online co-creation communities, business integration, and information goods. His research examines how companies use IT-based platforms in evaluating emerging technologies, optimal strategies for business integration, and the competitive landscape facing digital goods retailers. His research has been published in the Journal of Management Information Systems, Interfaces, IEEE Transactions on Engineering Management, and various conference proceedings, such as Americas Conference on Information Systems, the Workshop of Electronic Business, and the Annual Meeting of Decision Science Institute. James R. Marsden is a Board of Trustees Distinguished Professor and the Treibick Family Endowed Chair for CITI at the Department of Operations and Information Management (OPIM) at the University of Connecticut. He has been at UConn since 1993 as a Professor, serving fifteen years (1993–2008) as Head of OPIM. He helped develop both the Connecticut Information Technology Institute and the Treibick Electronic Commerce Initiative and served many years as Executive Director of both. Jim was part of the development team and severed as the UConn Director of edgelab, the unique research partnership between GE and UConn through its eighth year of operation. Dr. Marsden has a lengthy publication record in market innovation and analyses, economics of information, artificial intelligence, and production theory. His research work has appeared or is forthcoming in Management Science; Journal of Law and Economics; American Economic Review; Journal of Economic Theory; Journal of Political Economy; IEEE Transactions on Systems, Man, and Cybernetics; Statistical Science; Computer Integrated Manufacturing Systems; Decision Support Systems; Journal of Management Information Systems, Communications of the ACM, and numerous other academic research journals. He received his A.B. from the University of Illinois and his M.S. and Ph.D. from Purdue University. Also holding a J.D., Jim has been admitted to both the Kentucky and Connecticut Bar. Zhongju Zhang is an Associate Professor of Operations and Information Management in the School of Business, University of Connecticut. Zhang's main research interests include pricing issues and service quality of e-business; incentives and online user behaviors; social and organizational impacts of IT; and information systems resource management and scheduling. Zhang's research has been published in academic journals including Information Systems Research, INFORMS Journal on Computing, Journal of Management Information Systems, IEEE Transactions on Engineering Management, European Journal of Operational Research, Decision Sciences, Decision Support Systems, Communications of the ACM. Zhang is a guest associate editor for MIS Quarterly and serves on the editorial board of a number of academic journals including Journal of Database Management, Information Technology and Management, and Electronic Commerce Research and Applications. Zhang teaches both undergraduate and graduate level courses. He has lead and advised graduate students on numerous industry-strength, strategic decision problems involving application of advanced data analytics, quantitative methods, and marketing techniques.
Contents lists available at ScienceDirect
Decision Support Systems journal homepage: www.elsevier.com/locate/dss
Reliability (or “lack thereof”) of on-line preference revelation: A controlled experimental analysis☆,☆☆ Li Chen a, James R. Marsden b, Zhongju Zhang b,⁎ a b
Department of Management, School of Business and Economics, Fayetteville State University, 1200 Murchison Road, Fayetteville, NC 28301, United States Department of Operations and Information Management, School of Business, University of Connecticut, 2100 Hillside Road, Storrs, CT 06269, United States
a r t i c l e
i n f o
Article history: Received 13 May 2013 Accepted 13 June 2013 Available online 21 June 2013 Keywords: Preference revelation Information market Information revelation Information aggregation Controlled experiment Survey accuracy Survey reliability
a b s t r a c t Networks permitting anonymous contributions continue to expand and flourish. In some networks, the reliability of a contribution is not of particular importance. In other settings, however, the development of a network is driven by specific purposes which make the reliability of information exchanged of significant importance. One such situation involves the use of information markets for aggregating individuals' preferences on new or emerging technologies. At this point, there remains skepticism concerning the reliability of the preference revelations in such markets and thus the resulting preference aggregations and rankings of emerging technologies. In this paper, we study the reliability of on-line preference revelation using a series of controlled laboratory experiments. Our analysis includes individuals' pre- and post-experiment rankings of technologies, individual trading and accumulation activities during an electronics market experiment, the final experimental market outcomes, and a ranking of the same technologies by a panel of experts from a Fortune 5 company. In addition, as a final step, we allowed each participant to actually select and keep a unit of one of the technologies at zero price (free). That is, we were able to observe each participant's actual final true preference from the set of technologies. © 2013 Elsevier B.V. All rights reserved.
1. Introduction Networks permitting anonymous contributions continue to expand and flourish. In some networks, the reliability of a contribution is not of particular importance. Banter in a social media network may include tongue-in-cheek kidding or creation of a pseudo persona for participation in a second-life. In other settings, however, the development of a network is driven by specific purposes which make the reliability of information exchanged of significant importance. One such situation involves the implementation of networks for engaging individuals in real business applications such as the evaluation of new or emerging technologies [1], new product concepts [3,4,6], and idea generation [8]. Information preference markets (see discussion in [1]) have gained growing recognition as a relatively efficient process for gathering and aggregating information from a large number of dispersed individuals. However, there remains skepticism concerning the reliability of ☆ This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial-No Derivative Works License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and source are credited. ☆☆ The authors are grateful for the research support and funding provided by the Treibick Family Endowed Chair for CITI, the Gladstein Endowed MIS Research Lab, and the Connecticut Information Technology Institute. ⁎ Corresponding author. Tel.: +1 8604860660. E-mail addresses: [email protected] (L. Chen), [email protected] (J.R. Marsden), [email protected] (Z. Zhang). 0167-9236/$ – see front matter © 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.dss.2013.06.010
preference revelations (and thus the usefulness of the preference aggregations and decision rankings) gathered in such on-line markets. Chen et al. [2] argued that the assumption that traders are “truth-telling” might not be reasonable when an information market is relatively small. Van de Walle and Turoff [14] observed in their experiment that a large number of traders chose to follow market leaders instead of acting based on their private information. To date, however, we find no direct exploration of the accuracy of preference revelation, no testing of whether individuals really reveal their true preferences in preference markets. Further, we find no direct analysis of the consistency of individual preference revelations before, during, and/or after participation in preference markets. Here we report the structuring, operationalization, and outcomes of a series of controlled laboratory experiments addressing these issues. The ability to carefully structure the settings and observe all individual activities and market occurrences makes laboratory experimentation an attractive process to address the questions of interest (see [5,7,9,10]). Specifically, through controlled laboratory experiments, we seek to address each of the following research questions: (1) are individuals' actual revealed preferences consistent with initial reported preference, portfolio accumulation activities during the course of the experimental market, and/or post-experiment reported preference? (2) are individual true revealed preferences more closely aligned with peer activities during the experimental market and the resulting
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
final market outcomes than with the individual's stated preferences pre- and/or post-experiment? and, (3) is there any relationship between the number of market participants who accurately reveal their preference and the consistency between market outcomes and rankings from a panel of experts? The third question requires a bit of explanation. Chen et al. [1] explained in detail that their Fortune 5 partner company was not dis-satisfied with the results from the use of panels of experts. The issues for the company were the costs and the lack of scalability of panels of experts. Here we are not arguing whether or not panels of experts provide the best outputs. Rather, our experience suggests that the outcomes from panels of experts are valued by firms (Chen et al.'s [1] partner firm and others) and representative of an interesting comparison. The remainder of the paper is as follows. In Section 2, we briefly review related literature in preference markets and experimental economics. Section 3 is devoted to the development and discussion of the research questions analyzed in this note. In Section 4, we outline the design of our experimental market and detail the experimental techniques used to capture the participants' preference revelations over the course of the experimental process. Section 5 presents the findings of our controlled experiments. Section 6 summarizes our findings and suggests future research directions. 2. Literature review We use the term “on-line preference markets” to refer to virtual markets established to collect participants' preference on virtual stocks representing items of interest. Alternative historical methods used to capture such information include surveys, focus groups, critical success factors, and scoring models. Compared to such alternatives, on-line preference markets have the following advantages: (1) flexibility, (2) scalability, and (3) ability to incorporate and capture dynamic interactions among participants [4]. Gruca et al. [6] used such markets to forecast new product success, finding results similar to those of surveys but with reduced variability of results. LaComb et al. [8] utilized preference markets in generating and evaluating new product ideas. They reported increased idea generation and an expanded number of participants compared to conventional techniques. In a recent study of on-line preference market, Chen et al. [1] investigated the use of such markets for evaluating and ranking emerging technologies. The authors developed and implemented an on-line market for a major corporation with participants being employees with technology-related job responsibilities. Based on consistency between the preference market rankings and rankings from a company-selected panel of experts, the company decided to invest significant new funds in further development and use of on-line preference markets in the area of evaluating emerging technologies. As noted earlier, we are not arguing that panels of experts necessarily yield the best outcomes. Rather, our experience suggests that the outcomes from panels of experts are valued by firms (Chen et al.'s [1] partner firm and others) and representative of an interesting comparison. While on-line preference markets have begun to get traction in the business community, many questions remain. In the arena of prediction markets for predicting near future events, Wolfers and Zitzewitz [15] point out that earning through better prediction provides incentives for traders to reveal their true information. This is consistent with the experimental economics literature which emphasizes using significant performance based monetary incentive. In on-line preference markets, however, the outcomes are an aggregated ranking of preferences and performance based payoffs are not easily measurable. Developing performance-based reward metrics linked directly to observable outcomes is a thorny issue. In the area of emerging technologies, for example, the actual outcome of interest (commercial success or failure) cannot actually be determined in a short period of time. Any new product or emerging technology may be years away from
271
final implementation. The actual technologies or product evaluated is likely to be subsequently modified or morphed. Further, only a few, if any, of the products or technologies will move ahead to implementation. Thus, we are unable to evaluate those not selected and only able to evaluate those selected, if at all, sometime in the future. In preference markets, we actually have two measurement issues of concern. First, as Chen et al. [1] discussed in the area of emerging technologies, there is the question of whether the preferences observed and aggregated are sufficiently closely aligned with a panel of experts so that a company might use the scalable and more flexible preference markets to substitute for expense, non-scalable, and often inflexible panels of experts. Second, there is the important question of whether participants actually reveal their true preferences. While Chen et al. [1] focused on the former, in this paper we focus on the latter. Before detailing our experimental market design and operationalization in Section 4, we devote Section 3 to the presentation and development of a series of preference rankings in order to address the research questions discussed earlier. 3. Development of rankings for comparison As noted, earlier work by Chen et al. [1] detailed the structure of on-line preference market and presented the results and analysis of a full scale on-line preference market for emerging technologies. The market was developed and implemented for a Fortune 5 company. Commenting on the success of the implemented market, the authors noted, “Perhaps most importantly, our partner company viewed the study results as sufficient to warrant expanded investigation and use of on-line preference market in their early-stage emerging technology screening.” In that work, the authors' focus was at the market level and included analysis of price convergence under various conditions, overall trading activity (including overall activity of specific types of trades), and consistency of market outcomes with results from a panel of experts engaged by the partner company. The present analysis is focused at the individual level. As detailed in the next section, our research process, including controlled experimentation, provides a series of rankings from each participant on a set of four technologies. In addition to these individual rankings, we also have rankings from the experimental market outcomes (where “shares of stock” in the four technologies were traded) and those from a panel of experts that evaluated the four technologies. The panel of experts was enlisted from a technology evaluation community within the Fortune 5 company, providing a benchmark similar to that utilized in [1]. The set of available rankings are summarized below, where the index label, such as I or II corresponds to a time ordering when the rankings occur within the experimental process (the expert panel ranking has no time order since it is outside the experimental process — we arbitrarily assign this ranking the last value or “VI”): Individual rankings: I) the participant's rank ordering of technologies on a preexperiment survey (PreES); II) the participant's bids and transactions during the experiment (IMT); IV) the participant's rank ordering of technologies on a postexperiment survey (PostES); and, in a novel twist for this analysis, V) the participant's actual choice of a unit of one of the technologies (Actual). Experimental market rankings III) rankings determined by the market's final share prices of the four technologies (MP) Expert panel rankings VI) rankings from a panel of experts enlisted from a technology evaluation community within the Fortune 5 company (Expert)
272
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
Thus we have a series of indicators of each individual's preferences on a proffered set of technologies. We note that the fourth ranking by individuals is only a single ranking, that is, the actual first preference of each individual participant, i.e., the actual choice of one of the four technologies that an individual could choose and receive for free. The various rankings – information gathered before, during, and following the experiment – along with the information gathered from the panel of experts enabled a variety of consistency analyses. Fig. 1 depicts the six alternative information sets gathered. Rankings I–V are clearly time ordered. Ranking set VI (panel of experts) is not absolutely required to have the information gathered prior to the experiment, but that was the case in our work. The next section details our experimental process and how each of the rankings was operationalized. This is followed by a summary of experiment results and analysis of the outcomes. 4. The experiment process 4.1. Experiment design Controlled laboratory experiments were used to gather the data necessary to analyze our research questions. This approach has been thoroughly discussed and demonstrated, especially in the context of various market analyses (see [5,7,9,11–13]). Using the open-source software platform jMarkets (http://jmarkets.ssel.caltech.edu/), we constructed an initial experimental preference market design and trading process. A series of pilot experiments were conducted using a group of volunteer participants (Ph.D. students, student workers, and university staff), to check for any issues with system operation, participant understanding or use of the system, or other issues that might arise. Following the first several pilot runs, modifications were made to aid participant understanding (mainly through enhancements in depth and quality of explanatory materials provided to participants) and in screen design (mainly minor changes in location and detail of information presented on a participant's screen). Subsequent trial runs using a mix of earlier participants along with new volunteer participants resulted in a second set of modifications. A third and final run by several pilot study participants was completed to make sure issues had been addressed. Once the electronic market was constructed and pilot tested, we moved to the specifics of our experimental design. Based on past experiments, reports, and recommendations from experimental economics,
together with our research budget constraint, we decided on experiments with eight participants in each experiment. Anyone who had participated in pilot testing was excluded from participating in an actual experiment session. Given typical show-up payments ($10 per subject), overall reward-based session payments (averaging $10 per subject per session as explained below) to participants, and the cost of purchasing the technology products (slightly less than $20 per subject), our budget enabled eight sessions with eight participants in each session. The experiment and experimental process were reviewed and approved by the university's internal human subject review panel. We recruited 64 university students (24 graduate students and 40 undergraduate students). Of the 24 graduate students, 18 were from the School of Business, 1 from the Department of Computer Science and Engineering, and 5 from the Department of Economics. All the undergraduate students were from the School of Business. Each recruited participant was randomly assigned into one of the planned eight sessions. We verified that recruited subjects had no knowledge of or experience in preference markets. Instructions to participants were specific that they must agree to no communication with any other subject before or after their session. The preference market for each session involved four virtual stocks, each virtual stock representing a product that incorporated an emerging technology. We choose final product formats (with embedded emerging technology) because, in discussion with a variety of technologists and emerging technology specialists, the strong feeling was that the final product format made it easier for subjects to evaluate them. The products were chosen so that the incorporated emerging technologies were not competing technologies and the products were neither substitutes nor complements. The products we chose had only recently appeared and had not been widely seen in the marketplace. This choice was made to make it unlikely that participants would have existing experience with the selected products. In addition, we chose items of similar monetary value (approximately $20) to limit any effect on traders' evaluation linked to or influenced by the price/value of the item. 4.2. Experiment procedure Experiments were conducted in a controlled laboratory setting (the Gladstein Endowed MIS Research Lab). For each session, each
I. individual rankings from pre-experiment survey (PreES)
V. individual’s actual choice of a unit of one of the four technologies (Actual)
AVAILABLE TECHNOLOGY RANKINGS
II. individual trading activities during experiment (IMT)
VI. rankings from panel of experts (Expert) III. rankings from final market prices (MP) IV. individual rankings from post—experiment survey (PostES) Fig. 1. Six rankings gathered — set for which consistencies/inconsistencies are evaluated.
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
participant was assigned a unique ID. Individual participants were seated in front of a computer with no view of another participant's screen. We enforced a strict no-talking policy during the entire session. One author stayed in the lab during the course of each experiment. Once the eight participants for a session had been logged in, a short video (roughly five minutes) was presented to explain the trading mechanism they would utilize in the electronic preference market. The video was created as a means to ensure that participants in different sessions received consistent instructions and explanations. A hard copy of the same instruction was also provided to each participant. To further ensure the instructions were understood, after looking at the video and reading the detailed instruction participants were then given a short paper quiz consisting of five questions. When participants completed the short quiz, correct answers were reviewed with the participants and any questions or requests for clarification were answered. Details on the four emerging technology products were provided as web pages (one for each product) along with a hardcopy of product descriptions and details. We note that the instructions and explanations emphasized that the four products had been evaluated and rated by a group of emerging technology professionals at a major corporation. Participants were informed that they would earn performance rewards based on how their final stock portfolio matched up against the ratings of the set of emerging technology professionals (see details below). At this point of the session, we collected our first set of ranking information - each participant's pre-experiment preference rankings (PreES) of the four products. Subjects were asked to fill in a survey providing their preference orderings on the four products. Typically, the process involving watching the video, reading instructions, checking understanding, and completing the survey took approximately 15 min. Following completion of the pre-experiment survey, a brief (approximately 5 min) trial session was provided so participants could experience the preference market interfaces and trading mechanisms. The trial session incorporated random “phantom offers” so more trades would go through and help provide participants with an understanding of the processes. Following the trial session, the formal forty-minute experiment began. The trading mechanism followed a double auction framework that has been widely used in on-line preference markets [1,3,4]. Participants begin with an initial stock portfolio and a cash balance. At any point during the experiment, a participant with sufficient cash balance can submit a buy order. Similarly, with sufficient stock assets, a participant can submit a sell order. When another participant (with sufficient appropriate “assets” or “holdings”) accepts an existing buy or sell order, a transaction occurs. Transactions of partial volume are permitted.
4.3. Incentive details As noted above, there were two types of rewards in the experiment: a $10 payment for showing up and a dollar reward based on a participant's final portfolio value where the value is directly linked to the rating evaluations by a panel of emerging technology experts. To
273
determine the later rewards, we took the total reward payment available for each of the eight sessions ($80 per session) and divided this amount so that each share of stock in a given emerging technology product received a valuation based on the proportionate ranking by the professionals. Thus, in total, each participant received $10 for showing up and participating, a “final portfolio reward” equal to the sum of shares held of each stock times the weighted value for that stock (based on the expert panel ratings), and the final choice of one technology product to keep. Subjects were not aware of the last reward until after the completion of their market session. To encourage trading among traders, we utilized the rule that the any “cash” held was valued at zero in the final portfolio evaluation. The goal of rewarding participants with a reasonable show-up fee plus extra rewards related to their performance relative to expert ratings should help to induce serious market behavior (see the discussion in [7,11,12]). In the structure utilized, payments to participants ranged from $15 to $24 dollars. Following completion of an experiment, each participant was again provided with a survey form used to rank the four technology products (PostES). Following completion of the form and without any prior indication, each participant was allowed to select and keep one of the four technology products (Actual). Assuming rationality, this last selection step provides actual preference revelation of each individual's most preferred or top choice from the four technology products. In the next section, we analyze the “preference revelations” obtained from each of the participants, from the market, and from the expert panel. 5. Experiment results and analysis We compiled the six ranking sets indicated in Fig. 1: pre-experiment survey (PreES) for each individual, market transactions (IMT) for each individual, post-experiment survey (PostES) for each individual, actual technology product selection (Actual) for each individual, final market prices (MP) for each of the eight markets, and the expert panel ranking (Expert) for the four technology products. We consider each of our three research questions in turn. Table 1 presents the consistency rates between the first choice indicated at various stages and the actual choice or actual preference. (1) Do individuals show consistency in initial reported preference, portfolio accumulation (trading) activities during the course of the experimental market, post-experiment reported preference, and final actual demonstrated preference? The post-experiment survey has only a slightly higher consistency rate (73.44%) with the actual choice than does the pre-experiment (70.31%). That is, 70.3% of the participants revealed their true preference in the pre-survey rising to 73.4% in the post-survey which was taken after the individuals had been informed of the market outcomes. Both however, displayed greater consistency rates with actual choices than did trading activities (portfolio accumulations) whose consistency with the actual choice was 64.06%.
Table 1 Consistency rates between the first choices from five available rankings and the actual final individual choice (the actual individual revealed preference).
I. pre-market survey PreES II. market transactions IMT III. market outcomes MP IV. post-market survey PostES V. actual choice Actual VI. expert panel Expert
I. pre-market survey PreES
II. market transaction IMT
III. market outcomes MP
IV. post-market survey PostES
V. actual choice Actual
VI. expert panel Expert
100%
57.81% 100%
45.31% 60.94% 100%
59.38% 68.75% 59.38% 100%
70.31% 64.06% 53.13% 73.44% 100%
48.44% 50.00% 62.50% 54.69% 54.69% 100%
274
L. Chen et al. / Decision Support Systems 56 (2013) 270–274
(2) Are individual true preferences impacted by peer activities during the experimental market and the resulting final market outcomes? The market outcome consistency rate with the post-market survey was 59.38%, an increase of 14.07% from the consistency rate of 45.31% between the pre-market survey and the market outcomes. This suggests some impact from peer activities during the experimental market. The market outcomes had a much lower consistency rate with actual choice (53.13%) than did the post-experiment survey with the actual choice (73.44%), but the market outcomes did have the highest consistency rate (62.5%) with the expert panel. Finally, the post-market survey and actual choices each had somewhat higher consistency rates (54.69%) with the expert panel than did the pre-market survey with the expert panel (48.44%). (3) Is there any relationship between the number of market participants who accurately reveal their preference and the consistency between market outcomes and rankings from a panel of experts? For the five groups whose market choice (MP) was consistent with the expert panel's top choice (Expert), 23 of 40 individuals or 57.5% had their Actual being the same as Expert. For the three groups whose market choice (MP) was inconsistent with the expert panel's top choice (Expert), 12 of 24 individuals or 50% had their Actual being the same as Expert. If a firm (such as the partner firm in [1]) has utilized panels of experts over time and their analysis indicates very positive outcomes, then consistency of the outcomes with those of the panel of experts is certainly a positive for the that firm. In general, the valuation of the performance of panels of experts is done only within a firm and we have no general or cross-firm detailed analysis available to us. Thus result (3) is likely of most interest to firms having had good results from the use of panels of experts.
6. Summary and Conclusions We summarize our experimental results as following: i) We found that just 70.3% of the participants revealed their true preference in the PreES and 73.4% did so in the PostES, a time point after the individuals had been informed by the market outcomes. Thus, even at a point immediately before individuals revealed their true preference by selecting one technology to keep (Actual), almost 27% misstated their true preference. ii) There was some market impact on individual stated preferences in the sense that consistency rates were higher between PostES and MP (59.38%) than between PreES and MP (45.31%). iii) The market outcomes did have the highest consistency rate (62.5%) with the expert panel. Further, only one of the markets failed to have the Expert top pick in the top two MP values. Of the results detailed in this research note, it is the first one that raises a general concern since it indicates that individuals alone cannot be counted on to accurately reveal their preferences. In structuring our analysis, we incorporated an opportunity for individuals to reveal their true preference by selecting and keeping as their own one of the four technologies. In the PreES, the failure rate for individuals to accurately match their Actual choice was just slightly below 30%. Even after participating in the electronic market and getting more time to think about the alternatives and observe information on the ratings (MP) from others, almost 27% still misstated the Actual preference in the PostES. The latter is even more striking when we recall that only a minute or so passed between completion of the PostES's and the Actual choice. These results suggest cause for concern when stated preferences rather than actual observed choice are used in analyses and decision making.
References [1] L. Chen, P. Goes, J.R. Marsden, J. Zhang, Design and use of on-line preference markets for evaluation of early stage technologies, Journal of Management Information Systems 26 (2009) 45–70. [2] Y. Chen, C. Chu, T. Mullen, Predicting uncertain outcomes using information markets: trader behavior and information aggregation, New Mathematics and Natural Computation 2 (2006) 281–297. [3] E. Dahan, A.J. Kim, A.W. Lo, T. Poggio, N. Chan, Securities trading of concepts (STOC), Journal of Marketing Research 48 (2011) 497–517. [4] E. Dahan, A. Soukhoroukova, M. Spann, New product development 2.0: preference markets. How scalable securities markets identify winning product concepts & attributes, Journal of Product Innovation Management 27 (2010) 937–954. [5] C.L. Gardner, J.R. Marsden, D.E. Pingry, The design and use of laboratory experiments for DSS evaluation, Decision Support Systems 9 (4) (1993) 369–379. [6] T. Gruca, J. Berg, M. Cipriano, The effect of electronic markets on forecasts of new product success, Information Systems Frontiers 5 (2003) 95–105. [7] E. Hoffman, J.R. Marsden, A. Whinston, Laboratory experiments and computer simulation: an introduction to the use of experimental and process model data in economic analysis, in: J. Kagel, L. Green (Eds.), Advances in Behavioral Economics, volume 2, Ablex Publishing, Norwood, NJ, 1990, pp. 1–31. [8] C.A. LaComb, J.A. Barnett, Q. Pan, The imagination market, Information Systems Frontiers 9 (2007) 245–256. [9] J.R. Marsden, R. Pakath, K. Wibowo, Decision making under time pressure with different information sources and performance-based financial incentives—part 1, Decision Support Systems 34 (1) (2002) 75–97. [10] J.R. Marsden, Y.T. Tung, The use of information system technology to develop tests on insider trading and asymmetric information, Management Science 45 (1999) 1025–1040. [11] V.L. Smith, Microeconomic systems as an experimental science, The American Economic Review 72 (1982) 923–955. [12] V.L. Smith, Experimental economics: induced value theory, The American Economic Review 66 (1976) 274–279. [13] M. Spann, B. Skiera, Internet-based virtual stock markets for business forecasting, Management Science 49 (2003) 1310–1326. [14] B. Van de Walle, M. Turoff, Fuzzy relations for the analysis of traders' preferences in an information market game, European Journal of Operational Research 195 (2009) 905–913. [15] J. Wolfers, E. Zitzewitz, Prediction markets, The Journal of Economic Perspectives 18 (2004) 107–126. Li Chen is an assistant professor of management in the Department of Management, School of Business and Economics at Fayetteville State University. He received his Ph.D. from the University of Connecticut. Dr. Chen's research interests include IT innovation, preference markets, sponsored online co-creation communities, business integration, and information goods. His research examines how companies use IT-based platforms in evaluating emerging technologies, optimal strategies for business integration, and the competitive landscape facing digital goods retailers. His research has been published in the Journal of Management Information Systems, Interfaces, IEEE Transactions on Engineering Management, and various conference proceedings, such as Americas Conference on Information Systems, the Workshop of Electronic Business, and the Annual Meeting of Decision Science Institute. James R. Marsden is a Board of Trustees Distinguished Professor and the Treibick Family Endowed Chair for CITI at the Department of Operations and Information Management (OPIM) at the University of Connecticut. He has been at UConn since 1993 as a Professor, serving fifteen years (1993–2008) as Head of OPIM. He helped develop both the Connecticut Information Technology Institute and the Treibick Electronic Commerce Initiative and served many years as Executive Director of both. Jim was part of the development team and severed as the UConn Director of edgelab, the unique research partnership between GE and UConn through its eighth year of operation. Dr. Marsden has a lengthy publication record in market innovation and analyses, economics of information, artificial intelligence, and production theory. His research work has appeared or is forthcoming in Management Science; Journal of Law and Economics; American Economic Review; Journal of Economic Theory; Journal of Political Economy; IEEE Transactions on Systems, Man, and Cybernetics; Statistical Science; Computer Integrated Manufacturing Systems; Decision Support Systems; Journal of Management Information Systems, Communications of the ACM, and numerous other academic research journals. He received his A.B. from the University of Illinois and his M.S. and Ph.D. from Purdue University. Also holding a J.D., Jim has been admitted to both the Kentucky and Connecticut Bar. Zhongju Zhang is an Associate Professor of Operations and Information Management in the School of Business, University of Connecticut. Zhang's main research interests include pricing issues and service quality of e-business; incentives and online user behaviors; social and organizational impacts of IT; and information systems resource management and scheduling. Zhang's research has been published in academic journals including Information Systems Research, INFORMS Journal on Computing, Journal of Management Information Systems, IEEE Transactions on Engineering Management, European Journal of Operational Research, Decision Sciences, Decision Support Systems, Communications of the ACM. Zhang is a guest associate editor for MIS Quarterly and serves on the editorial board of a number of academic journals including Journal of Database Management, Information Technology and Management, and Electronic Commerce Research and Applications. Zhang teaches both undergraduate and graduate level courses. He has lead and advised graduate students on numerous industry-strength, strategic decision problems involving application of advanced data analytics, quantitative methods, and marketing techniques.