Strategic effort allocation in online innovation tournaments

Information & Management xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Information & Management journal homepage: www.elsevier.com/locate/im

Strategic effort allocation in online innovation tournaments ⁎

Indika Dissanayakea, , Jie Zhangb, Mahmut Yasarc, Sridhar P. Nerurb a b c

University of North Carolina Greensboro, PO Box 26170, Greensboro, NC 27402, United States The University of Texas at Arlington, 701 S West St, PO Box 19437 Arlington, TX 76019, United States The University of Texas at Arlington, 701 S West St, PO Box 19479 Arlington, TX 76019, United States

A R T I C L E I N F O

A B S T R A C T

Keywords: Innovation tournaments Crowdsourcing Social comparison Solver strategies Sniping

Online innovation tournaments, such as those hosted by crowdsourcing platforms (e.g., Kaggle), have been widely adopted by firms to evolve creative solutions to various problems. Solvers compete in these tournaments to earn rewards. In such competitive environments, it is imperative that solvers provide creative solutions with minimum effort. This article explores the factors that influence the solvers’ effort allocation decisions in a dynamic tournament setting. Specifically, comprehensive time variant data of teams that participated in crowdsourcing competitions on Kaggle were analyzed to gain insight into how solvers continually formulate strategies in light of performance feedback obtained through interim ranking. The results suggest that solvers strategically allocate their efforts throughout the contest to dynamically optimize their payoffs through balancing the probability of winning and the cost of expending effort. In particular, solvers tend to increase their efforts toward the end of tournaments or when they get closer to winning positions. Furthermore, our findings indicate that a last-minute surge in effort is more prevalent among high-skill solvers than in those with lower skill levels. In addition to providing insights that may help solvers develop strategies to improve their performance, the study has implications for the design of online crowdsourcing platforms, particularly in terms of incentivizing solvers to put forth their best effort.

1. Introduction An innovation tournament refers to “a process that uncovers exceptionally good opportunities by considering many raw opportunities at the outset and selecting the best to survive” ([1] p.80). With rapid advances in Information Technologies, companies have increasingly adopted online innovation tournaments and contests to complement their in-house innovation projects, primarily to reduce costs without compromising on quality [2]. Online platforms facilitate tournamentbased tasks in different areas, including software development, predictive analytics, scientific problem solving, and graphics and arts design [3]. For instance, the celebrated Netflix $1 M challenge attracted about 51,000 participants from 86 countries working, all vying to build a prediction model that would improve the accuracy of Netflix’s movie recommendation algorithm by 10% [4]. In yet another competition, more than 57,000 online gamers, most of whom had no prior experience in molecular biology, contributed to the identification of the structure of a particular protein within three weeks, thus solving a problem that had defied researchers at the University of Washington for years [5,6]. It is apparent that such platforms provide a cost-effective means to exploiting the “wisdom of the crowds,” thereby affording

⁎

companies novel insights and solutions that may not be forthcoming with in-house projects alone. The emerging literature on innovation tournaments has primarily focused on the optimal design of contests [7–10] or on the effects of individual characteristics and behaviors on contest outcomes [11–15], with the goal of maximizing payoffs. These studies provide evidence of the benefits of innovation contests, such as lower costs [13], lower risks [7], and higher quality of solutions [16], as well as affording multiple alternative solutions to challenging problems [9]. Prior studies (e.g., [17]) have carefully explored the impact of the characteristics and behaviors of solvers on the outcome of innovation tournaments under different competition conditions. Our paper extends these studies by examining how solvers strategically exploit the dynamics of these tournaments to wisely allocate efforts in an un-blind competition setting. In innovation tournaments, solvers need to improve their skills and/ or enhance their efforts to increase the likelihood of winning [9]. Although all the contestants expend effort to come up with a superior solution, it is only the best solution that is ultimately rewarded. Therefore, increasing effort in this setting can be costly, as a consequence of which contestants strategically decide how much effort to

Corresponding author. E-mail addresses: [email protected] (I. Dissanayake), [email protected] (J. Zhang), [email protected] (M. Yasar), [email protected] (S.P. Nerur).

http://dx.doi.org/10.1016/j.im.2017.09.006 Received 20 May 2016; Received in revised form 9 September 2017; Accepted 28 September 2017 0378-7206/ © 2017 Elsevier B.V. All rights reserved.

Please cite this article as: Dissanayake, I., Information & Management (2017), http://dx.doi.org/10.1016/j.im.2017.09.006

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

and all-pay auctions. Since these features simultaneously impact solvers’ effort allocation decisions in innovation tournaments, it may not be appropriate to explore them separately. Thus, we relied on theories from both the auction and tournament literature to study this effect. In this section, we first compare and contrast our study with prior work on innovation tournaments. Subsequently, we review pertinent literature and theories on interim ranking feedback (e.g., social comparison theory) and timing strategies (auction literature).

exert over the duration of the contest in order to enhance their chances of winning. That is, participants may not necessarily put forth their best efforts to achieve their highest potential but rather strive just enough to accomplish their goal of providing a solution that is good enough to outperform their opponents and win the reward. Understanding these strategies would not only be helpful to contestants but also provide insights to platform providers. Seekers, or platform providers, do not have direct control over the extent to which contestants exert effort to solve the problem at hand. However, an understanding of the factors that influence how much effort solvers expend would help platform providers design competitions in such a way that solvers are persuaded to put forth their best efforts, thus increasing the likelihood of an optimal winning solution. Our study used data from Kaggle.com, a large online predictive analytics tournament platform, to investigate how contestants formulate strategies for allocating efforts throughout a contest with a view toward improving their rankings and eventually securing a winning position. We identified two different strategies: timing of efforts (i.e., timing of submitting solutions) and interim rank impact. A key insight of our study is that solvers strategically delay their efforts until the end of the contest. Specifically, consistent with the findings in a complete information, sequential all-pay auction setting, our results showed that strong players strategically delay their efforts [18,19]. Furthermore, our finding that solvers strategize based on their interim rankings and subsequently exert more effort as they get closer to the winning position is consistent with the tenets of social comparison theory [20,21]. Thus, our study provides novel insight into the behavior of contestants as they respond to “game mechanics” such as interim rankings – a manifestation of leaderboards – and evolve strategies to strike the right balance between effort and performance. In summary, this paper makes significant contributions to the sparse but emerging literature on innovation tournaments. First, this study is among a select few that have examined the effects of interim performance feedback in dynamic innovation tournaments. It extends the application of social comparison theory to online tournament platforms by showing that feedback can intensify the competition among top rankers. Interestingly, this is also an affirmation of the claim by advocates of gamification that “gaming elements” such as leaderboards (i.e., interim rankings) can engage and motivate participants (e.g., [22]). Second, it extends the timing strategies that were widely examined in the online auction literature to investigate the timing of efforts in a dynamic innovation tournament setting. Third, it shows that these timing strategies are contingent on the expertise of solvers by examining the moderating impact of their skills. Finally, to the best of our knowledge, ours is the first study to utilize temporally varying data, such as interim rankings and efforts, to elucidate how participants with differing skills continually strategize to balance their effort with the level of performance they desire. The remainder of this paper is organized as follows. The next section reviews the literature related to innovation tournaments, interim ranking feedback, and timing strategies. It is followed by an articulation of our research model and the hypotheses that follow from it. Subsequently, we describe our data collection procedures and measures and then present our findings. Finally, we conclude with a discussion of the study’s theoretical and managerial implications as well as its limitations, followed by directions for future research.

2.1. Innovation tournaments A tournament is a “competition in which the outcome is determined by relative performance and the winner takes disproportionally larger award than the loser” ([24] p.578). It is also referred to as rank-ordered tournament since the performance is based on rank. Tournament theory has been applied in various contexts including academics, sports, sales, scientific work, and executive promotions [3]. In the online innovation tournament context, geographically distributed contestants compete with one another for monetary rewards. There is a growing body of literature on online innovation tournaments that primarily investigates the influence of contest characteristics (reward structure, problem characteristics, scope) and contestant characteristics (demographics, familiarity, skill) on the likelihood of finding a high-quality solution [11,25]. For example, Boudreau et al. [26] showed that, in general, increasing competition negatively impacts the performance of competitors, but induces a small group of competitors at the very top to exert more effort [3]. While adding more competitors reduces the incentive to solvers to exert more effort, it also increases the likelihood of finding an optimal solution [7]. Terwiesch and Xu [9] investigated how efficiencies in these competitions improved with changing award structure. Liu et al. [19] used a randomized field experiment in a crowdsourcing context to examine the effects of reward size and early high-quality submission on the number and quality of subsequent submissions. They found that the level of participation as well as the quality of submissions was positively associated with the size of the incentive. Furthermore, they demonstrated that experienced users were less likely to pursue tasks that already had high-quality solutions. While their study focused on decisions related to contest participation, our study draws attention to the underlying dynamics of the effort allocation process. Archak [11] showed that reputation systems influence strategic behaviors of solvers. Specifically, the study demonstrated that top-rated solvers used different strategies to successfully deter entry of their rivals in the same contest. Most of these studies are based on blind, one-shot competition settings, with a few notable exceptions [25,27]. That is, solvers cannot see how good the solutions submitted by their rivals are, and they also get only one chance to submit a solution. Thus, the contestants primarily rely on the problem specification provided by the contest organizer at the beginning of the contest period. Online “un-blind” innovation tournaments are becoming popular for finding creative solutions to diverse problems (e.g., Kaggle.com, logomyway.com, and taskcn.com). Moreover, some innovation tournaments allow solvers to make multiple solution submissions within the contest duration. Thus, over time solvers learn strategies to be successful in this “un-blind” and dynamic competition setting. However, to the best of our knowledge, the strategic behaviors of solvers have not attracted much attention in the innovation tournament literature. Among the few exceptions to this are [15,25,28,29]. Both Yang et al. [15] and Chen and Liu [29] showed that solvers who made their initial submission early or late within the contest duration have a higher chance of winning the competition, while Bockstedt et al. [28] showed that contestants who have a lower position in initial submission, or a higher level of separation between initial and last submission, are more likely to be successful. Yang et al. [15] argued that some solvers prefer to submit good solutions early in the contest to scare away other competitors as well as to receive early feedback from the seekers. In

2. Literature review This study is grounded in two distinct streams of research, namely Tournament- and Auction-related. In a general tournament setting (e.g., Sport tournaments such as weightlifting tournaments [23]), social comparison theory can be used to explain the impact of interim ranking on participants’ efforts allocation patterns, while in auctions, timing strategies are more apparent. However, innovation tournaments in an un-blind setting, such as this study, have features of both tournaments 2

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

used to explain the effect of open feedback about interim rank on participants’ behaviors.

contrast, other contestants may prefer to strategically wait until the end to submit their solutions, so that they can have access to more information. Al-Hasan et al. [30] also emphasized the importance of the timing of the entry. They showed that feedback in open innovation contests helps early entrants by providing an opportunity to significantly revise their submission, while it results in a significant amount of information spillover favoring later entrants. However, our study is distinctive in many ways: First, Yang et al. [15] and Chen and Liu [29] considered only the timing of the initial submission. Thus, their data were cross-sectional in nature, similar to one-shot/one-bid auctions. Bockstedt et al. [25] showed that participants’ strategies, such as timing of the first entry, number of entries, range of entries, and skewness of entries, influence the probability of winning a contest. Though Bockstedt et al. [25] and Bockstedt et al. [28] refer to repeated submissions, they considered only the difference between the first and last submissions. That is, they did not have any time variant component in their model. In our setting, solvers can make multiple submissions throughout the duration of the contest, which is akin to repeated/ multiple-bid auctions. We observed how the allocation of efforts changes during the entire period of the contest and not just at the beginning or the end of the contest. Our study is unique in that it accounts for the dynamics of effort allocation throughout the competition by considering the timing of every submission by every team. Second, our study further distinguishes itself from prior works (e.g., [15,25,28,29]) by considering the impact of ranking feedback as well as the moderating effect of the skill levels of contestants. Table 1, which compares and contrasts our work with previous studies on innovation tournaments, clearly shows the broader scope of our study. Some crowdsourcing platforms provide interim feedback that can be viewed by all participants. For example, leaderboards, an element inspired by games and one that is strongly advocated by proponents of gamification (e.g., [22]), display a rank-ordered listing of contestants. Feedback, in general, has a signaling effect on other solvers, potentially either motivating them to work harder to compete or persuading them to quit. Therefore, open feedback that can be viewed by all may influence solvers’ effort levels. Wooten [31] showed that in un-blind, repeated entry settings, substantial solution improvement by a contestant enhances the efforts of rivals and results in a higher submission rate. This, in turn, results in an improved contest outcome. The study also showed that contest-specific characteristics, such as higher prices, and participant-specific characteristics, such as prior performance and platform experience, lead to a greater number of incremental improvements. The next section discusses how social comparison theory may be

2.2. Interim ranking feedback According to social comparison theory, individuals are driven by a basic desire to improve their performance (“unidirectional drive upward”) and to minimize the difference between themselves and other persons (“targets”) [32]. Social comparison means “the tendency to self-evaluate by comparing ourselves to others” ([32] p. 634) and is the key to competitive behavior. Furthermore, scholars have shown that situational factors, such as incentive structure (e.g., winner takes all or multiple rewards), proximity to a standard (i.e., closer to the winning position or far away), and the number of competitors influence levels of social comparison [32]. All these situational factors are pertinent to the innovation tournaments discussed in this study. Furthermore, Garcia et al. [21] generalized their findings and stated that there is “a tendency for competition among commensurate rivals on a relevant dimension to intensify in the proximity of a meaningful standard” ([21] p. 970). In innovation tournaments, interim rank disclosure facilitates social comparisons among contestants (i.e., they evaluate their own performance by comparing themselves with others). These social comparisons could result in competitive arousal [33] and impact contestants’ behaviors. This is a widely investigated area in the management and psychological literature [34]. However, findings on the impact of ranking feedback on contestants’ efforts are inconclusive. Social comparison theory offers some insights that can potentially provide an explanation for these inconclusive findings. For instance, Hannan’s [35] experimental study showed that relative performance feedback under tournament compensation plan reduced the average performance, while it increased the average performance under individual performance compensation plan. According to social comparison theory, people generally exhibit an upward drive as they endeavor to improve their performance and outperform those they consider to be marginally better than themselves [36]. This, however, is contingent on the standard used for comparison. In the tournament compensation plan used by Hannan [35], only the top 10% were compensated. Thus, the standard was at the top. Analysis showed that even though the overall average performance went down, the mean performance of the top two deciles increased. This suggests that solvers closer to the standard act more competitively; thus, ranking feedback increases the effort of higher performing participants. In a classroom setting, Azmat and Iriberri [34] found that providing information about the class average to students has a positive effect on their final performance. In contrast, Barankay’s [37] experiment showed that, in the absence of a standard, interim rank feedback negatively impacts employees’ efforts. That is, participants in Barankay’s [37] study got paid irrespective of their performance or ranking. Thus, there was no standard. In yet another study, Casas-Arce and Martinez-Jerez [38] found that winners reduce their effort as the lead increases, while the trailing contestants reduce their efforts only when the gap between their current rank and the winning position is very large. The context of Casas-Arce and Martinez-Jerez [38]’s study was, however, different, for the top 50 participants were rewarded regardless of their true rankings. Thus, the goal of the solvers was to be among the first 50 without incurring a huge cost in terms of effort. As a consequence, there is a heightened intensity of competition among commensurate rivals whose rankings are closer to the standard (e.g., #50 vs. #51), whereas those whose rankings are far away from the standard (e.g., #1 vs. #2 or #1000 vs. #1001) tend to expend less effort. Ederer [39] has argued that interim performance feedback impacts workers’ incentives to exert effort. It helps workers effectively tailor their effort choices based on their abilities. That is, contestants with high abilities exert more effort, whereas contestants with lower abilities put forth less effort [39]. Ericksson [40] showed empirical evidence for positive peer effects of ranking feedback in a tournament. That is, frontrunners do not reduce

Table 1 Innovation Tournament Literature. Time Series Data

Yang et al. [15] Chen & Liu [29] Bockstedt et al. [28] Bockstedt et al. [25] Al-Hasan et al. [30] Liu et al. [19] Terwiesch & Xu [9] Boudreau et. al. [25] Boudreu et al. [26] Wooten [31] Archak [11] Boudreau et al. [3] Our study

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

Explanatory Variables

Timing Strategies

Skill Moderation

Interim Rank

☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

Note: represents “applied” or “considered.” ☐ represents “not applied” or “not considered”.

3

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

Jian et al. [27], is similar to sequential auction. However, Jian et al. [27] considered a one-shot interaction, while our setup involved repeated entries. Moreover, they did not explore participants’ strategic behaviors with respect to timing and interim ranks. Also, many studies that have adopted the all-pay auction framework have not considered heterogeneity of contestant skills. The notable exceptions are Liu et al. [8] and Konrad and Leininger [18]. Liu et al. [8] argued that players exert their best effort when their rivals have similar skill levels. Konrad and Leininger [18] is the closest analytical model to this study. In an allpay auction setting with complete information, they have shown that contestants’ choice of timing can be endogenized and the strongest player strategically decides to enter late. The ensuing section articulates a research model and formulates hypotheses resulting therefrom.

their efforts and underdogs do not leave the tournament. In a weightlifting tournament setting, Genakos and Paliero [23] have shown that revealing information on the ranking of contestants increases risk-taking behavior of contestants who are just behind the interim leaders. Although risk-taking behavior is somewhat similar to effort allocation in innovation tournaments, effort allocation or the risktaking in a weightlifting tournament occurs at fixed intervals and participants cannot decide on when to allocate the efforts. In our context, solvers can simultaneously decide not only how much effort to put forth but also when to allocate the effort. Thus, our study enriches this literature by introducing timing strategies. The next section explains how we draw on the conceptual underpinnings of the auction literature to explain timing strategies (i.e., timing of allocation of efforts) that emerge in open innovation tournaments characterized by a repeated entry setting.

3. Research model 2.3. Timing strategies (auction literature) In an online innovation tournament with an un-blind setting, solvers strategically alter their efforts to enhance their chances of winning, while reducing the cost of effort. This study investigated solvers’ strategies using weekly performance data from an online innovation tournament platform for data analysts, Kaggle.com. Based on a review of prior literature as well as findings from competitions, we identified two such strategies: timing strategies and strategies related to interim ranks. Drawing from the theoretical foundations of all-pay auctions with complete information and social comparison theory, we hypothesize that the contest time elapsed and the solvers’ rankings influence their effort allocation decision. Unlike offline contests, most of these online contests allow participants to compete dynamically. Online innovation tournaments considered in this study show some characteristics of all-pay auctions with complete information. In our context, solvers can submit multiple solutions throughout the competition. Every time a contestant submits a solution, his/her ranking and solution quality are revealed in a dynamic leaderboard open to all the contestants. At the end of the competition, only the best solution will be entitled to the reward. Similarly, in auctions, all participants submit bids and only the winning bid will be entitled to the good/service. Thus, solution submission behaviors in these competitions are somewhat similar to bidding behaviors in auctions. The online auction literature has demonstrated that carefully choosing bid timing could significantly influence the probability of winning the auction [43]. More specifically, the auction literature has identified benefits from the late-bidding or “sniping” strategy. Bidders tend to submit their bids at the end of the auction to avoid competition with other bidders [43]. Furthermore, in addition to avoiding bidding wars that lead to a high transaction price, it protects information spillovers [42]. We expect a similar behavior in the online tournament setting. Late submissions reduce information spillover, do not provide sufficient time for rivals to respond, avoid submission wars that lead to very high efforts, and provide an opportunity for contestants to learn about their rivals’ performances. Thus, we argue that solvers strategically exert more effort toward the end of the competition. In the context of our study, the number of solution submissions is a reflection of the level of effort exerted. Hence, we have the following hypothesis:

Innovation tournaments explored in this study have features of allpay auction. In all-pay auctions, all the bidders must pay their bid amount regardless of whether they win the auction or not. Only the highest bid will receive the good or service. In an innovation tournament, when contestants are allowed to submit solutions anytime during the contest, the submissions (effort exertions) are similar to bidding in auctions [19,41]. Bid amount in all-pay auctions is analogous to efforts of solvers in innovation competitions. In innovation competitions, all solvers expend efforts, but only the winner will be entitled to a reward [19]. Thus, spending the right amount of effort at the right time is critical in this setting to increase the chance of winning without wasting significant amounts of effort. In an online auction context, researchers have investigated dynamic bidding strategies and their benefits. In general, the auction literature has found mixed results regarding the effect of early and late bidding strategies. Some scholars have argued that bidders prefer to strategically wait until the last minute to avoid an early bidding war that increases the transaction price. Moreover, bidding late helps informed bidders protect their information, thus preventing their competitors from learning. This last-minute bidding practice is called “sniping” in an auction context [42]. Revealing their true value early in the auction will give their rivals a competitive edge. While bidding near the end of the auction will not give competitors sufficient time to respond, early movers may reveal strategies that bidders can learn. On the other hand, a bidder may decide to make an early high bid to make others less interested in the auction [15,30]. Roth and Ockenfels [42] showed that late bidding (sniping) is more prevalent in eBay auctions −where end times are fixed − than in Amazon auctions where close times could be automatically extended based on when the last bid is received. They also observed that bidding patterns varied by the experience level of bidders. In an innovation tournament setting, the end times of the contest are fixed. Thus, sniping is expected to be a dominant strategy. Solvers in an innovation tournament context are quite unlike some of these online auctions, where the reservation price can be simply fixed and the system (sniping agent [33]) can then automatically raise bids by minimum increments above the previous high bid. Solvers must decide how much effort to put forth. Also, while bid prices in auctions can be made quickly, solutions to problems in an innovation tournament setting cannot be evolved rapidly. Solution improvements not only take time but are also contingent on the skill levels of participants. Solvers with high skills, as opposed to low-skilled ones, could quickly improve their solution. Jian et al. [27] conducted an experiment to compare and contrast effort levels in simultaneous all-pay auctions and sequential all-pay auctions. They found that the expected maximum effort is higher in simultaneous auctions than in sequential auctions. Furthermore, they found that in a simultaneous auction setting, users with high ability tend to exert less effort when faced with multiple opponents. Our context, like the one in

Hypothesis 1 (Timing Strategy Hypothesis). Time elapsed is positively related to the level effort. That is, teams tend to make more submissions toward the end of the competition. According to the conceptual underpinnings of all-pay auction with complete information, the best strategy for the strongest player is to enter late [18]. Liu et al. [19] empirically demonstrated that experienced users submit their initial solutions later than inexperienced ones. Thus, based on the theory and prior literature, we argue that high-skill players will benefit through strategically delaying their submissions. High-skill teams possess the necessary expertise to adapt, learn, and 4

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

work fast; hence, they − unlike low-skill teams − can afford to wait. Moreover, information spillover is more critical to high-skill solvers, for the competitive edge they have − because of their task-related skills − may be lost if they revealed information about the quality of their solutions to their rivals. In the same setting, Wooten [31] showed that a substantially improved submission by a solver increases the subsequent efforts of other participants. High-skill contestants are reluctant to submit a good solution early because that would set a higher standard for everyone, inducing contestants to expend more effort, which, in turn, would require the early contributors of high-quality solutions to increase their efforts to win the competition. The goal of high-skill solvers, therefore, is not to submit their best solution, but to submit a solution that is good enough to outperform their rivals’ solutions. Hence, we argue that high-skill solvers strategically expend more effort (as evidenced by the number of submissions) toward the end of a competition when compared with low-skill solvers.

contest j at time (t-1), and the interaction of the skill of team i and the time elapsed. In addition, we added a term (square of the time elapsed) to account for non-linearity. Furthermore, we controlled for contestspecific effects using fixed effects. Also, we added lagged dependent variable to control for persistency. We also accounted for team-specific effects by controlling for size and skill of team i in contest j. We summarize our econometric model as follows:

Effortijt = α 0 + α1 Effortij (t − 1) + α2 TimeElapsedjt + α3 TimeElapsedjt 2 + α4 Rankij (t − 1) + α5 TimeElapsedjt *TeamSkillij + α6 TeamSizeij + α 7 TeamSkillij + δj + ϵij

(1)

where i, j, and t denote teams, contests, and time periods, respectively; αk (k = 0…7) represents the coefficients of the variables; and δj denotes the contest dummies, which are included to control for the contest heterogeneity. The dependent and explanatory variables included in this model are explained in the next section.

Hypothesis 2 (Skill Moderation Hypothesis). Solvers’ skill positively moderates the relationship between time elapsed and effort, such that high-skill solvers expend more effort toward the end of competition.

4. Data collection and variable definitions

Ranking disclosures in tournaments facilitate social comparisons. That is, it allows contestants to compare their abilities with those of their opponents. Social comparison helps participants make informed decisions about how much effort they need to put forth to enhance their chances of winning while reducing the cost of effort. Thus, solvers can effectively tailor their efforts based on interim feedback [39]. According to social comparison theory [20], contestants in competitive innovation tournaments, such as the one in our study, are likely to have a “drive upward” to perform well as they continually evaluate themselves vis-à-vis those who are placed higher on the leaderboard. The theory suggests that there are several predictors that account for participants’ strategic behaviors, namely mutually relevant dimension, meaningful standard, and commensurate rivals. In our context, the monetary reward would constitute the mutually relevant dimension. Only the winner, or the top ranker, would be eligible for the reward. Thus, the meaningful standard is the number one position (i.e., top ranking). Solvers with adjacent ranks are referred to as commensurate rivals. The game is dynamic because rankings on the leaderboard are updated regularly. Although the mutually relevant dimension stays the same throughout the competition, the proximity to a meaningful standard and to commensurate rivals could change every time a solver makes a new submission. Drawing on social comparison theory, Garcia et al. [21] showed that “rankings that coincide with a standard intensify the social comparison process to a greater extent than rankings that do not.” Interim ranking disclosure through open leaderboard would make the intensity of competition more unequally distributed among competing solvers. Since only the top ranker is entitled to the reward, competitive behavior intensifies among commensurate rivals who have high rankings (e.g., #2 vs. #3), whereas the competition among those who have low rankings (e.g., #50 vs. # 51) is not likely to be intense [21]. In addition, from the perspective of economic theory, participants are motivated to increase their efforts as the likelihood of winning increases [35]. Thus, we argue that when solvers get closer to the winning position, they tend to exert more effort.

4.1. Data collection As mentioned earlier, data for our study were obtained from Kaggle.com, a specialized innovation tournament platform that focuses on predictive analytics projects (Fig. 1). Kaggle has a pool of more than 100,000 data scientists coming from over 100 countries and 200 universities. These data scientists are experts in various quantitative fields, such as computer science, statistics, economics, mathematics, and physics. Over the last few years, Kaggle has served many companies, including GE, Allstate, Merck, Ford, and Facebook, and has helped them to improve sales forecasting, increase customer retention, reduce operating costs, accelerate product development, and gather information from social media (Kaggle.com). Companies, government organizations, and researchers provide problem descriptions and relevant datasets to Kaggle and often specify the monetary reward they are willing to pay the winners. Based on these inputs, Kaggle sets up innovation tournaments or contests. Kaggle typically provides training and test datasets to the contestants. Each participant or participating team can submit multiple solutions before the contest deadline. Kaggle evaluates all submissions in real time and provides instant feedback to the participants. This feedback is displayed on a leaderboard that is open to the public. Contestants not only learn how good their models are but also get to know how smart their rivals are. Thus, these tournaments show some characteristics of all-pay auctions with complete information. Every time a solver submits a solution, the leaderboard is dynamically updated to show the current rank of solvers and the prediction accuracy score of their solution. The accuracy score and ranking are unique and serve as objective measures of solution quality, which is unavailable in most other innovation tournament initiatives. Moreover, Kaggle’s website provides each solver with an online profile, which shows a solver’s personal information and overall performance score based on their final rankings in previous contests (see Fig. 1). For this study, we collected weekly leaderboard data for 25 tournaments from September 2013 to November 2014. The dataset consists of more than 10,000 teams and 73,670 observations. Tables 2 and 3 summarize the descriptive statistics and correlation matrix, respectively.

Hypothesis 3 (Ranking Hypothesis). Solvers’ rankings of the previous week positively influence their efforts in the current week (i.e., topranked players in the previous week’s leaderboard make more efforts, while bottom-ranked players make less effort). As indicated above, prior literature showed that the effort put forth by participants is influenced by factors such as the gap between the current and the winning position [21,38], timing of the contest [15,29], contest-specific characteristics, and participant-specific characteristics [31]. In our model, the dependent variable is the effort of team i in contest j at time t. Our main explanatory variables were as follows: contest time elapsed in contest j as of time t, interim rank of team i in

4.2. Dependent and independent variables 4.2.1. Effort We used the number of solution submissions as a measure of team effort, the main dependent variable in our study. The number of submissions made within a week was used as a proxy for team effort in that 5

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

Fig. 1. Screenshots of the Data Source: Kaggle.com.

uses a formula to calculate each individual’s profile score based on their performance in prior competitions. The maximum achievable score in a competition is derived from the total number of participants and the level of difficulty of the contest. According to Kaggle, “the current formula for each competition splits the points among the team members, decays the points for lower finishes, adjusts for the number of teams that entered the competition, and linearly decays the points to 0 over a two-year period from the end of the competition.” Kaggle updates each individual’s skill scores after each competition. The team skill score in our dataset ranged from 0 to 877 K, with a mean of 33 K. We used a log transformation to address scaling issues.

Table 2 Descriptive Statistics. Variable

Mean

Std. Dev

Min

Max

Team Size Team Skill Submissions Team Rank Time Elapsed Std. Score

1.16 33475.64 2.14 375.73 0.73 78.86

0.68 66731.47 6.49 356.29 0.23 27.03

1.00 0.00 0.00 1.00 0.07 0.00

24.00 876551.40 334.00 1792.00 1.00 100.00

week. Consistent with prior literature [14], effort was measured by taking the difference between the number of submissions for two consecutive weeks on the leaderboard shown in Fig. 1. In addition, we used predicted accuracy of the solution based on efforts as an alternative measure of dependent variable in the section on robustness tests. Our primary independent variables are team rank and time elapsed. Team skill was used as a moderator variable.

4.3. Control variables Team-specific variables, such as team size and team skill, were used as controls. We used a fixed-effects model to control for tournamentspecific effects and lagged the dependent variable to control for persistency. Moreover, we re-ran the model with additional controls to control for the distance from the best score. These results are presented in the robustness tests section.

4.2.2. Time Elapsedt This is the percentage of contest time elapsed as of the current week (t).

4.3.1. Team size This is the number of members in a team. Previous studies [44,45] indicated that team size has an impact on team performance. The team size in our dataset ranged from 1 to 24, with a mean of 1.2.

4.2.3. Team Rank(t-1) This is the relative position of a team in a contest at the end of the previous week (t-1). This figure ranged from 1 to 1792, with a mean of 375.73.

4.3.2. Effort(t-1) We also include the previous week’s effort (lagged dependent variable) in the model to address the persistency of effort variable. It is likely that, for most teams, efforts are correlated over time due to some

4.2.4. Team skill This is the average profile score of the members in a team. Kaggle Table 3 Correlation Matrix.

1 2 3 4 5 6

Team Size Team Skill Submissions Team Rank Time Elapsed Std. Score

1

2

3

5

7

8

1.0000 −0.0160 0.1566 −0.0758 0.0021 0.0113

1.0000 0.2284 −0.2669 −0.0330 0.1155

1.0000 −0.2663 0.1038 0.1663

1.0000 0.2720 −0.3432

1.0000 0.0662

1.0000

6

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

historical factors, such as work ethic, abilities, and other unobservable behaviors. Thus, the lagged dependent variable allows control for some team-specific omitted factors.

Table 5 Team Skill Moderation Results.

4.3.3. ΔBestscore This is the distance between the best score and the team’s score in the previous week (t-1) in a given contest. The auction literature has shown that jump bid (i.e., a bid with significant increment) impacts the subsequent bidding behavior of the bidders by seriously deterring competitors through high entry cost [46,47]. Moreover, in the same setting, Wooten [31] has shown that a significantly improved solution results in an increase in subsequent efforts of other contestants. To control for this, the deviation of the solver’s solution score from the best solution score was added to the model. We divided all values by the range to standardize scales. The distance from best score for team i, in contest j, and time (t-1) is given by

ΔBestScorei,(t − 1), j =

(2)

5.1. Negative binomial model We used the negative binomial model for our analysis because it fits well with our data characteristics. Our main dependent variable is count data, and OLS regression is not appropriate because of the skewness of the data. Poisson and negative binomial models are commonly used for count data. However, our data present over-dispersion relative to the Poisson distribution. The log likelihood ratio test of alpha suggested that negative binomial distribution is superior to Poisson in this case [48,49]. Results are summarized in Table 4. Models 1 and 2 show the results without considering the lagged effect of the dependent variable, while Model 3 shows the results after controlling for the lagged effect. In both Models 1 and 2, the coefficients of time elapsed (α2 = 0.895, p < 0.01; α2 = 0.863, p < 0.01) and square of time elapsed (α3 = 2.427, p < 0.01; α3 = 2.443, p < 0.01) are positive and significant. Thus, “Timing Strategy Hypothesis (H1)” is supported. The results suggest that solvers strategically delay their efforts to enhance the chances of winning while reducing the cost of efforts. The coefficient of the interaction effect of team skill and the time elapsed is positive and significant in both Models 2 and 3 (α5 = 0.11, p < 0.01; α5 = 0.12, p < 0.01). Thus, “Skill Moderation Hypothesis (H2)” is supported. Results show that strong teams strategically delay their efforts

Time Elapsed Sq

Team Skill Team Size Log likelihood Observations

−0.3740 (0.2689) 0.3654 *** (0.0581) 0.4670 *** (0.0456) 0.5917 *** (0.0402) 0.7441 *** (0.0681) 0.9057 *** (0.1112) 73670

−7.0695 *** (0.4297) −0.2470 *** (0.0172) 0.8413 *** (0.488) 0.9866 *** (0.0448) 1.1641 *** (0.0821) 1.3525 *** (0.1120) 73670

Model 2

Model 3

−0.0013 *** (0.0000) 0.8951 *** (0.0566) 2.4267 *** (0.1919)

−0.0013 *** (0.0000) 0.8630 *** (0.0572) 2.4430 *** (0.1919) 0.1090 *** (0.0298)

0.3547 *** (0.0078) 0.5071 *** (0.0200) −107343.6 73670

0.3565 *** (0.0078) 0.5119 *** (0.0201) −107336.93 73670

−0.0008 *** (0.0000) 0.7796 *** (0.0626) 1.0177 *** (0.2446) 0.1245 *** (0.0355) 0.1621 *** (0.0027) 0.3202 *** (0.0085) 0.4808 *** (0.0214) −75532.94 61084

(3)

where αˆ2 signifies the direct impact of time elapsed on effort. We also expect the effort impact of time elapsed to be more pronounced for higher skill levels, which requires αˆ 5 to be positive and significant. To evaluate this conditional effect further, we estimated the time elapsed impact on effort at various percentiles of TeamSkill variable. For instance, at the 25th percentile of the TeamSkill variable (without the Time Elapsed Square), we obtain the point estimate of 0.3654 for the effort impact of time elapsed. However, in order to know whether the estimates obtained at different percentiles of TeamSkill variable are statistically significant, we also need to compute the corresponding standard errors. One can predict these standard errors by using the delta method [50], jackknife [51], or bootstrapping [52]. In this paper, we used the nonparametric bootstrap method introduced by Efron [52] that randomly resamples from our original sample of size n with replacement, obtains the bootstrapped sample of (R1, R2 , ......, Rn ),and computes the corresponding estimate δˆb from each of the B-bootstrapped samples. We then obtain the corresponding standard error as B

Model 1

Time Elapsed*Team Skill Effort(t-1)

Minimum 25th Percentile 50th Percentile 75th Percentile 95th Percentile Maximum Observations

∂Effort = αˆ2 + αˆ 5 TeamSkill ∂TimeElapsed

Table 4 Negative Binomial Regression Results.

Time Elapsed

Time Elapsed Impact (With Time Elapsed Sq)

compared to weak teams. The coefficient of rank is negative and significant in both models (α4 = −0.001, p < 0.01; α4 = −0.001, p < 0.01). Thus, “Ranking Hypothesis (H3)” is supported. This implies that when teams get closer to the winning position, they strategically exert more effort. This could be due to an increased confidence in winning. Our results show that the lagged dependent variable has a positive and significant impact on the dependent variable. Moreover, results suggest that all three hypotheses are supported even after controlling for persistency. In addition, we used a bootstrap method to further evaluate the impact of the time elapsed at different levels of team skill (Skill Moderation Hypothesis (H2)). Table 5 summarizes the results of bootstrap analysis. Given Eq. (1), the effort impact of the time elapsed can be written as

5. Results

Team Rank(t-1)

Time Elapsed Impact (Without Time Elapsed Sq)

***p < 0.01, **p < 0.05, *p < 0.1.

BestScore (t − 1), j − Scorei,(t − 1), j Scoremax , j − Scoremin, j

Team Skill

1/2

2 ⎫ ⎧ seˆB = ∑ [δˆb − δˆ ] /(B − 1) ⎬ ⎨ b=1 ⎭ ⎩

, where δˆb is the estimate from the bth-

resample (b=1,……., B) and δˆ =

B

∑ b=1

λˆb / B,is the mean of the resampled

values (see [53] for details). As the results indicate, as the level of skills increases, teams tend to make more efforts toward the end of the competition. These results are consistent with the behavior of strong players in complete information all-pay auctions. That is, strong players have an incentive to exert more efforts toward the end of the competition. Results also indicate that teams with low skills make more efforts at the beginning of the contest. Low-skill teams need more time to come up with a quality solution compared with high-skill teams; thus, they cannot afford to wait. Furthermore, we re-ran the model including ΔBestscore to control for the effect of quality of the best solution in a given time on efforts. As shown in Table 6, we observed that high-quality solutions (jump bids) negatively influence solver efforts. Consistent with the findings in the

***p < 0.01, **p < 0.05, *p < 0.1.

7

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

Table 6 Negative Binomial Regression Results (with ΔBestscore).

Team Rank(t-1) Time Elapsed Time Elapsed Sq Time Elapsed*Team Skill ΔBestscore (t-1) Team Skill Team Size Log likelihood Observations

Table 7 Seemingly Unrelated Regression Results.

Model 1

Model 2

−0.0003 *** (0.0000) 0.3738 *** (0.0553) 2.1973 *** (0.1854)

−0.0002 *** (0.0000) 2.2630 *** (0.0561) 2.2086 *** (0.1843) 0.2970 *** (0.0284) −4.0888 *** (0.0742) 0.2979 *** (0.0076) 0.4416 *** (0.0193) −105761.2 73670

−3.9871 *** (0.0732) 0.2932 *** (0.0076) 0.4299 *** (0.0192) −105815.7 73670

Effortt Effortt Team Rank(t-1) Time Elapsed Time Elapsed Sq Time Elapsed*Team Skill Team Skill Team Size Observations

−0.0027 *** (0.0000) 2.7681 *** (0.1258) 5.1923 *** (0.4202) 0.5545 *** (0.0629) 0.9022 *** (0.0165) 1.5373 *** (0.0335) 73670

Rankt −3.3380 *** (0.0464) 0.9857 *** (0.0011)

−6.2682 *** (0.2113) −1.5501 *** (0.4289) 73670

***p < 0.01, **p < 0.05, *p < 0.1. ***p < 0.01, **p < 0.05, *p < 0.1. Table 8 Zero-Inflated Negative Binomial Regression Results.

auction literature, this observation indicates that a solution with a very high quality could be a serious deterrent to rivals, as a consequence of which they face very high entry cost (i.e., they have to put forth high efforts).

Team Rank(t-1) Time Elapsed Time Elapsed Sq Time Elapsed*Team Skill Effort(t-1) Team Skill Team Size Log likelihood Observations

5.2. Robustness tests 5.2.1. Seemingly unrelated regression model Studies show that the performance of solvers is a function of their skill set as well as the effort they put forth [9]. Thus, enhancing either the skills or the efforts of participants increases the chances of winning through improved quality of the solution. However, increasing the effort level is costly because only the winner will be entitled to a reward. Hence, solvers need to develop some strategies to increase the likelihood of winning without incurring significant cost of expending effort. They may achieve this by strategically allocating their efforts. To clarify the relationship, we identify the following statistical models, given data availability and the existing theories that explain the structural determinants of team effort and rank:

−0.0010 *** (0.0000) 0.8673 *** (0.0591) 0.9170 *** (0.2442)

−0.0010 *** (0.0000) 0.7727 *** (0.0617) 0.9901 *** (0.2445) 0.2032 *** (0.0366) 0.1561 *** (0.0036) 0.0000 *** (0.0000) 0.2842 *** (0.0223) −76254 61084

0.1544 *** (0.0036) 0.0000 *** (0.0000) 0.2718 *** (0.0219) −76269 61084

5.2.2. Zero-inflated negative binomial model As with many empirical count data, our dataset suffers from the excess zero problem. The over-dispersion in the data could be a result of that. Therefore, we also ran our model using zero-inflated negative binomial model. Some teams may not submit their solutions frequently. This model is capable of separating out “always zero” group from “not always zero” group [49]. Table 8 summarizes the results. Model 1 shows the results without considering the interaction effect of team skill and time elapsed, while Model 2 shows the results including the moderation effect. As shown in Table 8, all three hypotheses are supported, thus demonstrating the robustness of our results.

+ α4 Rankij (t − 1) + α5 TimeElapsedjt *TeamSkillij + α 6 TeamSizeij (4)

Rankijt = β0 + β1 Effortijt + β2 Rankij (t − 1) + β3 TeamSizeij + β4 TeamSkillij + δ2j + ε2ij

Model 2

***p < 0.01, **p < 0.05, *p < 0.1.

Effortijt = α 0 + α1 Effortij (t − 1) + α2 TimeElapsedjt + α3 TimeElapsedjt 2

+ α 7 TeamSkillij + δ1j + ϵ1ij

Model 1

5.2.3. Effort versus team percentile Kaggle evaluates team percentiles based on team rankings on the final leaderboard (i.e., the ratio of team ranking to the maximum ranking on the final leaderboard). To avoid confusion and to be consistent with definitions, we redefined team percentile as one minus the current value. Thus, high percentile indicates best teams and vice versa. We categorized teams into two groups based on their skill compared to median-skill, high-skill, and low-skill teams. Then, for each group, we estimate the impact of time elapsed on effort at different levels of team percentiles using the bootstrap method. Results show that for each percentile, the effect of time elapsed on effort is high for high-skill group compared with low-skill group. Moreover, within each skill group, teams that performed well made more efforts toward the end of the tournament (Table 9). We categorized teams into four groups based on their performance. Then, we plotted a graph of cumulative number of submissions over

(5)

Since team effort and rank are expected to be driven by some common observable and unobservable variables, we model them as a set of relations. The unobservable factors or the omitted variables that may drive both effort and rank are part of the error terms ε1 and ε2. Specifically, it is likely that the information included in these omitted variables will be included in both error terms. Thus, the two equations in the system are linked, since the error term in the effort equation is likely to be significantly correlated with the error term in the rank equation. To identify this setup, we follow Zellner [54] and estimate a “seemingly unrelated regression” (SUR) model that allows correlated error terms. It involves a two-stage estimation procedure that is both consistent and efficient. Our results from a generalized least-squares estimation indicate that residuals of the equations in the system are in fact significantly correlated, validating the use of SUR model to account for this correlation. Specifically, given our Brusch–Pagan test statistic, we reject the null hypothesis of no correlation between error terms ε1 and ε2 at the significance level of 0.1. As the results in Table 7 show, at higher levels of skill and effort, the ranking (relative performance) improves as well. Most importantly, the results from the equation system that considers the cross-equation correlations harmoniously support all three of our hypotheses. Thus, we conclude that solvers strategically allocate effort to enhance their chances of winning with a minimum cost.

Table 9 Team Skill Moderation Results (Percentiles).

25th Percentile 50th Percentile 75th Percentile Observations

Low-Skill Teams

High-Skill Teams

−0.0867 (0.1285) 0.1860 (0.1312) 0.4588 * (0.2531) 36835

0.7891 *** (0.0633) 0.9060 *** (0.0900) 1.0230 *** (0.1228) 36835

***p < 0.01, **p < 0.05, *p < 0.1.

8

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

Fig. 2. Cumulative Effort over Time.

Second, our results show that interim rank disclosure influences the effort allocation decisions of contestants. Solvers who are closer to the winners tend to exert more efforts. One plausible explanation is that when they get closer to winning, the risk of losing is reduced and they are therefore motivated to exert more effort. Moreover, teams with higher rankings get more competitive. In order to ensure that our findings were robust, we ran our data using a zero-inflated negative binomial model to account for the excess zero issue in our dataset. We also used estimated solution accuracy as a measure of dependent variables. In addition, we ran our data using a seemingly unrelated regression model that included the relationship between effort and performance. The results were consistent with the main model, thus affirming the robustness of our results. Furthermore, we explored whether timing strategies are different for different teams. First, we calculated the impact of time elapsed for team with different skill percentiles. Our study demonstrates that as the level of skill increases, the positive impact of time elapsed on effort is enhanced. Second, we categorized teams into four groups based on their performance and plotted a graph of the cumulative number of submissions over time. We saw different submission patterns for these groups. Though all groups, on average, tend to make more submissions toward the end of the tournament, graphs clearly showed that winning teams exert a considerably higher amount of effort toward the end of a tournament deadline. Third, we categorized teams by skill and their performance percentile. Results clearly show that within each skill level, winning teams exert more effort toward the end of the contest. The finding that the best teams strategically delay their efforts to enhance their chances of winning strengthens the argument for sniping. Furthermore, our results showed evidence for persistency of efforts. Finally, the impact of explanatory variables was significant even after controlling for persistency in team efforts. Our findings have several implications for theory and practice.

time (Fig. 2). We saw different submission patterns for these groups. The graphs show that winning teams (best teams) exert a considerably higher amount of effort toward the end of a tournament deadline by submitting more solutions. A sample graph for a contest is shown below. 5.2.4. Alternative measure of dependent variable The model was re-estimated with the predicted accuracy of the solution as a function of effort serving as our dependent variable. Results were consistent with previous findings. Table 10 summarizes the results with prediction accuracy as the dependent variable. 6. Discussion 6.1. Key findings This study provides valuable insight into solvers’ strategic behaviors in dynamic innovation tournaments. It demonstrates that participants strategically vary their efforts to enhance their chances of winning, while minimizing the cost of their efforts. Our study makes several noteworthy contributions to the existing body of knowledge. First, our research found that in open innovation tournaments, solvers tend to put more effort toward the end of the contest. This could be to reduce the information spillovers, avoid submission wars, and/or to ensure that rivals do not have sufficient time to respond. Furthermore, our results clearly show that strong teams exert relatively more efforts toward the end of the competition than weak teams. Strategically delaying efforts helps strong teams to win the competition with minimum effort. Table 10 OLS Regression Results with Estimated DV.

Team Rank(t-1) Time Elapsed Time Elapsed Sq Time Elapsed*Team Skill Team Skill Team Size R2 Observations

Model 1

Model 2

−0.0012 *** (0.0000) 1.2807 *** (0.0563) 2.1910 *** (0.1884)

−0.0012 *** (0.0000) 1.2361 *** (0.0565) 2.3174 *** (0.1888) 0.2484 *** (0.0283) 0.4054 *** (0.0074) 0.6907 *** (0.0150) 12.8% 73670

0.3992 *** (0.0074) 0.6847 *** (0.0150) 12.7% 73670

6.2. Theoretical implications Our study is not only anchored in established conceptual foundations but also methodologically robust. Specifically, our research brings together two distinct streams of research to investigate solvers’ effort allocation strategies in open innovation tournaments. From a theoretical perspective, it is important to investigate the combined effect as open innovation tournaments exhibit characteristics of both all-pay

***p < 0.01, **p < 0.05, *p < 0.1.

9

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

the possibility of increasing overall efforts by conducting contests in stages or multiple phases. The consequence of inducing contestants to expend more effort would be a winning solution of superior quality. However, additional research and in-depth empirical validations are needed to fully understand how these design choices impact the ultimate quality of the winning solution. As far as solvers are concerned, the findings of our study would be useful in understanding how a strategic adjustment in the level of effort can maximize their utility. Finally, the results of this study could help managers in organizational settings to provide a work environment that encourages employees to expend the right amount of effort to deliver their best performance.

auctions (e.g., timing strategies in bidding) and tournaments (e.g., interim rank feedbacks in sport competitions). Our study makes several contributions to the emerging literature on innovation tournaments. First, our research extends the application of bidding strategies beyond an auction setting to an innovation tournament context. It shows evidence for the existence of sniping in dynamic innovation tournaments. Specifically, consistent with findings of all-pay auction in complete information settings, our study shows that strong teams exert more efforts toward the end of innovation tournaments. Second, it extends the application of social comparison theory to an innovation tournament context and elucidates how interim rank disclosure affects the solvers’ decision with regard to the amount of effort to allocate. Solvers compare their performance with others and strategically alter their efforts to increase their utility. Thus, it enriches the evolving theoretical foundation of innovation tournaments by confirming the transferability of timing and interim rank feedback effects in this context. Furthermore, it adds to the existing body of knowledge on social comparison theory by demonstrating how its tenets can be used in an innovation tournament setting. Third, while prior research primarily focused on blind contests where submission behaviors and interim solution quality were unobservable, our study leveraged unique data from a leading predictive analytics platform (Kaggle) that continually provided objective measures of efforts and ranking of every team throughout the contest. This rich dataset allowed us to control for teams’ persistency, contest heterogeneity, and to use within-contest effort variations to assess teams’ strategies for exerting effort. Finally, unlike prior studies that mainly focused on static one-shot interactions in innovation tournaments, our study utilizes time variant data to investigate how solvers change their effort allocation patterns throughout the competition based on the dynamics of the competition. Thus, our study extends the boundaries of knowledge by providing deep insight into the dynamics of innovation tournaments.

6.4. Conclusion and future research In a turbulent business environment characterized by hyper-competition and uncertainty, it is imperative that organizations continually and expeditiously derive actionable insights from data. Predictive modeling is at the heart of this endeavor. However, the limited resources that organizations have, coupled with a dearth of analytics talent, severely hamper an organization’s efforts to derive value from data analytics. In order to augment their innovative capabilities, organizations crowdsource analytics solutions (e.g., machine learning and predictive modeling solutions) using popular contest platforms such as Kaggle. Given the increasing importance of such platforms, it is imperative that we study their dynamics to gain insight into what drives the participants and what factors influence the quality of the winning solution. As with many other empirical studies, our study has some limitations. However, we believe that these shortcomings are minor and do not detract from our findings or contributions in any way. Furthermore, being aware of these limitations gives us an opportunity to pursue further research to gain a deeper understanding of the innovation tournament context. First, our data only include information that is publicly available on the website. For instance, we do not observe participants’ real efforts, and the number of submissions was used as a proxy for effort. Future research should definitely consider a more expanded measure of effort. Additional methods, such as follow-up surveys, may also afford richer data, which, in turn, can lead to keener insights. Second, our results are based on data from a specific type of tournament platform. Given the rapid development in innovation tournaments applications, richer data and cases will become available to enable further research on participants’ strategic behaviors and their implications for different tournament settings. Finally, future studies may investigate the effects of gamification (e.g., [22]) in the context of innovation tournaments. For example, would mechanisms commonly employed in games, such as leaderboards, points, badges, and levels, to name but a few, have the desired effect of persuading participants to expend more effort and produce superior solutions? Our study is a small but important step toward providing insights that will enable providers such as Kaggle to design platforms that will engage and motivate participants to perform at their best level to deliver superior solutions. In addition, it is a useful benchmark for future studies that attempt to shed more light on the dynamics of innovation tournament platforms that organizations are increasingly turning to for creative solutions to their challenging business analytics problems.

6.3. Managerial implications The findings of this study have several implications for tournament platform providers and solvers. Platform providers can use the results of our study to design their platforms in such a manner that they increase the likelihood of getting higher quality solutions to the problems posted by companies, researchers, and others. For instance, the insights related to the timing and level of information disclosure and feedback and how they impact effort allocation decisions can be particularly useful in designing a platform that engages and motivates participants throughout the contest. Specifically, they could draw inspiration from games (e.g., video games) that commonly feature leaderboards, points, levels, and badges to involve participants and to encourage them to achieve higher levels [22]. Furthermore, our study demonstrates that competition intensifies among top rankers as only the best solution is rewarded. In order to motivate the lower performers, Kaggle and other platform designers can have a minimum standard to award badges or points that give competitors some sense of accomplishment. For example, if the competition said that the top 500 would receive some points that would establish or enhance their status on such platforms, it might motivate not just the leaders but those closer to the lower standard as well. In addition, platform providers can explore how a combination of blind (private) and un-blind (public) disclosures may improve the solution quality. For example, if they made early feedback private and late feedback public, would that encourage strong players to submit their solutions early? Clearly, encouraging top-ranked players to submit early and often would result in superior solutions. In addition to providing feedback, platform providers may also induce participants, particularly the high-ranked ones, to submit early by providing incentives. Our finding that solvers tend to put more efforts toward the end of a tournament suggests that platform providers should perhaps explore

References [1] J.O. Wooten, K.T. Ulrich, Idea generation and the role of feedback: evidence from field experiments with innovation tournaments, Prod. Oper. Manag. 26 (2017) 80–99. [2] M.K. Poetz, M. Schreier, The value of crowdsourcing: can users really compete with professionals in generating new product ideas? J. Prod. Innov. Manag. 29 (2012) 245–256. [3] K. Boudreau, C.E. Helfat, K.R. Lakhani, M.E. Menietti, Field evidence on individual

10

Information & Management xxx (xxxx) xxx–xxx

I. Dissanayake et al.

[4] [5]

[6] [7] [8] [9] [10]

[11]

[12] [13]

[14] [15]

[16] [17] [18] [19] [20] [21] [22] [23] [24] [25]

[26]

[27]

[28]

[29]

[30]

[31] [32] [33]

[34]

[35]

[36] [37]

[38] P. Casas-Arce, F.A. Martínez-Jerez, Relative performance compensation contests, and dynamic incentives, Manag. Sci. 55 (2009) 1306–1320. [39] F. Ederer, Feedback and motivation in dynamic tournaments, J. Econ. Manag. Strategy 19 (2010) 733–769. [40] T. Eriksson, A. Poulsen, M.C. Villeval, Feedback and incentives. Experimental evidence, Labour Econ. 16 (2009) 679–688. [41] D. DiPalantino, M. Vojnovic, Crowdsourcing and all-pay auctions, Proc. 10th ACM Conf. Electron. Commer. ACM, New York, 2009, pp. 119–128. [42] A.E. Roth, A. Ockenfels, Last minute bidding and the rules for ending second price auctions: evidence from eBay and amazon auctions on the internet, Am. Econ. Rev. 92 (2002) 1093–1103. [43] W. Guo, Exploring and Modeling of Bidding Behavior and Strategies of Online Auctions, ProQuest Dissertations Publishing, University of Maryland, 2013http:// drum.lib.umd.edu/bitstream/handle/1903/14125/Guo_umd_0117E_14236.pdf? sequence=1&isAllowed=y. [44] R. Guimera, B. Uzzi, J. Spiro, L.A.N. Amaral, Team assembly mechanisms determine collaboration network structure and team performance, Science 308 (2005) 697–702. [45] S.G. Cohen, D.E. Bailey, What makes teams work: group effectiveness research from the shop floor to the executive suite, J. Manag. 23 (1997) 239–290. [46] E.T. Bradlow, Y.H. Park, Bayesian estimation of bid sequences in internet auctions using a generalized record-breaking model, Mark. Sci. 26 (2007) 218–229. [47] X. Cui, V.S. Lai, Bidding strategies in online single-unit auctions: their impact and satisfaction, Inf. Manag. 50 (2013) 314–321. [48] A.C. Cameron, P.K. Trivedi, Regression Analysis of Count Data, Econometric Society Monograph No.30, Cambridge University Press, 1998. [49] R. Martinez-Espineira, Adopt a hypothetical pup: a count data approach to the valuation of wildlife, Environ. Resour. Econ. 37 (2007) 335–360. [50] A.R. Gallant, A. Holly, Statistical inference in an implicit nonlinear, simultaneous equation model in the context of maximum likelihood estimation, Econometrica 48 (1980) 697–720. [51] M.H. Quenouille, Notes on bias in estimation, Biometrika 4 (1956) 353–360. [52] B. Efron, Bootstrap another look at the jackknife, Annu. Stat. 7 (1979) 1–26. [53] M. Yaşar, C.J.M. Paul, Capital-skill complementarity, productivity and wages: evidence from plant-level data for a developing country, Labour Econ. 15 (2008) 1–17. [54] A. Zellner, D.S. Huang, Further properties of efficient estimators for seemingly unrelated regression equations, Int. Econ. Rev. 3 (1962) 300–313.

behavior & performance in rank-order tournaments, Harv. Bus. Sch. Work. Pap. (2012), https://dash.harvard.edu/handle/1/9502862. A. Vance, Fight club for geeks, Bus. Week (2012) 37–38. I. Dissanayake, J. Zhang, B. Gu, Task division for team success in crowdsourcing contests: resource allocation and alignment effects, J. Manag. Inf. Syst. 32 (2015) 8–39. N. Savage, Gaining wisdom from crowds, Commun. ACM 55 (2012) 13–15. K.J. Boudreau, N. Lacetera, K.R. Lakhani, Incentives and problem uncertainty in innovation contests: an empirical analysis, Manag. Sci. 57 (2011) 843–863. D. Liu, X. Geng, A.B. Whinston, Optimal design of consumer contests, J. Mark. 71 (2007) 140–155. C. Terwiesch, Y. Xu, Innovation contests open innovation, and multiagent problem solving, Manag. Sci. 54 (2008) 1529–1543. Y. Yang, P.Y. Chen, P. Pavlou, Open innovation: strategic design of online contests, Proc. 20th Workshop Inf. Syst. Econ, Association for Information Systems, Atlanta, GA, 2009, pp. 14–15. N. Archak, Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com, Proc. 19th Int. Conf. World Wide Web, Association for Computer Machinery, New York, 2010, pp. 21–30. B.L. Bayus, Crowdsourcing new product ideas over time: an analysis of the Dell IdeaStorm community, Manag. Sci. 59 (2013) 226–244. Y. Huang, P. Singh, K. Srinivasan, Crowdsourcing Blockbuster ideas: a dynamic structural model of ideation, Proc. 32nd Int. Conf. Inf. Syst. Atlanta, GA, 2011, pp. 19–22. J. Mo, Z. Zheng, X. Geng, Winning Crowdsourcing Contests: A Micro-Structural Analysis of Multi-Relational Networks., In: Harbin, China, (2011). Y. Yang, P.Y. Chen, R. Banker, Impact of past performance and strategic bidding on winner determination of open innovation contest, Proc. 21 St Workshop Inf. Syst. Econ. Association for Information Systems, Atlanta, GA, 2010, pp. 11–12. K. Girotra, C. Terwiesch, K.T. Ulrich, Idea generation and the quality of the best idea, Manag. Sci. 56 (2010) 591–605. D. Liu, X. Li, R. Santhanam, Digital games and beyond: what happens when players compete, MIS Q. 37 (2013) 111–124. K.A. Konrad, W. Leininger, The generalized stackelberg equilibrium of the all-pay auction with complete information, Rev. Econ. Des. 11 (2007) 165–174. T.X. Liu, J. Yang, L.A. Adamic, Y. Chen, Crowdsourcing with all-pay auctions: a field experiment on Taskcn, Manag. Sci. 60 (2014) 2020–2037. L. Festinger, A theory of social comparison, Hum. Relat. 7 (1954) 117–140. S.M. Garcia, A. Tor, R. Gonzalez, Ranks and rivals. A theory of competition, Pers. Soc. Psychol. Bull. 32 (2006) 970–982. J. Simões, R.D. Redondo, A.F. Vilas, A social gamification framework for a K-6 learning platform, Comput. Hum. Behav. 29 (2013) 345–353. C. Genakos, M. Pagliero, Interim rank risk taking, and performance in dynamic tournaments, J. Polit. Econ. 120 (2012) 782–813. H. Yin, H. Zhang, Tournaments of financial analysts, Rev. Account. Stud. 19 (2014) 573–605. J. Bockstedt, A. Mishra, C. Druehl, Do Participation Strategy and Experience Impact the Likelihood of Winning in Unblind Innovation Contests, (2011) http://papers. ssrn.com/abstract=1961244. K.J. Boudreau, N. Lacetera, M. Menietti, Performance responses to competition across skill levels in rank-order tournaments: field evidence and implications for tournament design, Rand J. Econ. 47 (2016) 140–165. L. Jian, Z. Li, T.X. Liu, Simultaneous versus sequential all-pay auctions: an experimental study, Exp. Econ. (2016) 1–22, http://dx.doi.org/10.1007/s10683-0169504-1. J. Bockstedt, C. Druehl, A. Mishra, Heterogeneous submission behavior and its Implications for success in innovation contests with public submissions, Prod. Oper. Manag. 0 (2016) 1–20. L. Chen, D. Liu, Comparing Strategies for Winning Expert-Rated and Crowd-Rated Crowdsourcing Contest, (2012) http://aisel.aisnet.org/amcis2012/proceedings/ VirtualCommunities/16. A. Al-Hasan, I.H. Hann, S. Viswanathan, Information Spillovers and Strategic Behaviors in Open Innovation Crowdsourcing Contests: An Empirical Investigation (n.d.), (2017) Retrieved from https://pdfs.semanticscholar.org/fcea/ 623f6c9ec21f975c710cf6e75d9d74af3eb3.pdf. J.O. Wooten, Leaps in Innovation: The Effect of Discontinuous Progress in Algorithmic Tournaments, (2013) http://ssrn.com/abstract=2376350. S.M. Garcia, A. Tor, T.M. Schiff, The psychology of competition: a social comparison perspective, Perspect. Psychol. Sci. 8 (2013) 634–650. T. Teubner, M. Adam, R. Riordan, The impact of computerized agents on immediate emotions, overall arousal and bidding behavior in electronic auctions, J. Assoc. Inf. Syst. 16 (2015) 838–879. G. Azmat, N. Iriberri, The importance of relative performance feedback information: evidence from a natural experiment using high school students, J. Public Econ. 94 (2010) 435–452. R.L. Hannan, R. Krishnan, A.H. Newman, The effects of disseminating relative performance feedback in tournament and individual performance compensation plans, Account. Rev. 83 (2008) 893–913. S.E. Taylor, M. Lobel, Social comparison activity under threat: downward evaluation and upward contacts, Psychol. Rev. 96 (1989) 569–575. I. Barankay, Rankings and Social Tournaments: Evidence from a Field Experiment, (2010) http://www8.gsb.columbia.edu/rtfiles/CDA%20Strategy/Barankay%20%20Rankings%20and%20Social%20Tournaments%20MS.pdf.

Indika Dissanayake is an Assistant Professor of Information Systems and Supply Chain Management at the Bryan School of Business and Economics, the University of North Carolina Greensboro. She received her Ph.D. in Information Systems from the College of Business Administration, the University of Texas at Arlington. Her research interests include crowdsourcing, social media, and virtual communities. Her research has appeared in journals and conference proceedings such as Journal of Management Information Systems, International Conference on Information Systems, Americas Conference on Information Systems, Decision Science Institute, and Hawaii International Conference on System Sciences. Jie (Jennifer) Zhang is an Associate Professor of Information Systems at the College of Business Administration, the University of Texas at Arlington. She received her Ph.D. in Computer Information Systems from the William E. Simon Graduate School of Business at the University of Rochester. She employs analytical and empirical techniques to examine a number of issues in ecommerce, software licensing, online reputation systems, web analytics, and social media. Her research appears in MIS Quarterly, Information Systems Research, Journal of Economics and Management Strategies, Journal of Management Information Systems, Communications of the ACM, and others. Mahmut Yasar is an Associate Professor of Economics at the College of Business Administration, the University of Texas at Arlington (UTA). He is also an Adjunct Professor of Economics at the Emory University, and the Zhongnan University of Economics and Law in China. Before coming to UTA, he taught at the Goizueta Business School Department of Finance and the Department of Economics at the Emory University from 2003 to 2007. Dr. Yasar's primary research interests center around the microeconomics of trade and investment, productivity, knowledge and technology transfer, innovation, and applied micro-econometrics. He has also worked on issues in environmental economics and corporate finance. His research has appeared in journals such as Journal of Business and Economic Statistics, Journal of International Economics, and Weltwirtschaftliches Archiv. He has served as an Associate Editor of the International Economic Journal. Sridhar Nerur is a Professor of Information Systems at the University of Texas at Arlington. He holds an engineering degree in electronics from the Bangalore University, a PGDM (MBA) from the Indian Institute of Management, Bangalore, India, and a Ph.D. in business administration from the University of Texas at Arlington. His research has been published in MIS Quarterly, Strategic Management Journal, Communications of the ACM, Communications of the AIS, The DATA BASE for Advances in Information Systems, European Journal of Information Systems, Information Systems Management, and Journal of International Business Studies. He has served as an Associate Editor of the European Journal of Information Systems and was on the editorial board of the Journal of AIS until December 2016. His research and teaching interests include social networks, machine learning, text analytics, cognitive aspects of design, dynamic IT capabilities, and agile software development.

11