Measuring the effectiveness and impact of an open innovation platform

Accepted Manuscript Title: Measuring the Effectiveness and Impact of an Open Innovation Platform Author: Glenn P. Carroll Sanjay Srivastava PhD Adam S. Volini Marta M. Pineiro-Nunez PhD Tatiana Vetman PII: DOI: Reference:

S1359-6446(17)30034-X http://dx.doi.org/doi:10.1016/j.drudis.2017.01.009 DRUDIS 1970

To appear in: Received date: Revised date: Accepted date:

4-10-2016 5-12-2016 16-1-2017

Please cite this article as: Carroll, G.P., Srivastava, S., Volini, A.S., Pi˜neiro-N´un˜ ez, M.M., Vetman, T.,Measuring the Effectiveness and Impact of an Open Innovation Platform, Drug Discovery Today (2017), http://dx.doi.org/10.1016/j.drudis.2017.01.009 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

#

Deloitte National Office, 30 Rockefeller Plaza, New York, NY, 10112-0015 Lilly Corporate Center, Indianapolis, IN, 46285

Corresponding authors: Glenn Carroll ([email protected]), 646 334 8669 Marta M Piñeiro-Núñez ([email protected]), 317 276 3046

us

∗

cr

Glenn P. Carroll,# Sanjay Srivastava PhD#, Adam S. Volini,# Marta M. Piñeiro-Núñez PhD,∗ Tatiana Vetman∗

ip t

Measuring the Effectiveness and Impact of an Open Innovation Platform

an

Teaser: This article provides a framework for evaluating the effectiveness of the Lilly Open Innovation Drug Discovery program and proposes a global leading indicator dashboard incorporating qualitative and quantitative metrics.

M

Keywords: open innovation, drug discovery, R&D, measures, dashboard, efficiency, OIDD

Abstract

Ac ce

pt

ed

Today, most pharmaceutical companies complement their traditional R&D models with some variation on the Open Innovation (OI) approach in an effort to better access global scientific talent, ideas and hypotheses. Traditional performance indicators that measure economic returns from R&D through commercialization are often not applicable to the practical assessment of these OI approaches, particularly within the context of early drug discovery. This leaves OI programs focused on early R&D without a standard assessment framework from which to evaluate overall performance. This paper proposes a practical dashboard for such assessment, encompassing quantitative and qualitative elements, to enable decision-making and improvement of future performance. The use of this dashboard is illustrated using real-time data from the Lilly Open Innovation Drug Discovery (OIDD) program.

Introduction

The concept of Open Innovation (OI) was first coined by Henry Chesbrough [1] to describe the paradigm by which enterprises allow free flow of ideas, products and services from the outside to the inside and vice versa in order to remain competitive, particularly in rapidly evolving fields where there is abundant, relevant knowledge outside the traditional walls of the enterprise. Initially, this idea was applied to the emerging realms of computer hardware and software, but since then it has spread to a number of areas in industry, academia and government. According to Chesbrough, there is a continuum from fully open to fully closed enterprises, with the degree of openness depending upon a number of internal and external factors affecting that particular enterprise, including intellectual property (IP) and level of regulation [1]. Today, not only has research and publication into the topic of OI grown exponentially, but typical OI methodologies – including idea competitions, customer immersion and crowdsourcing – 1 Joint Deloitte/Lilly Manuscript

Page 1 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

are routinely understood and leveraged to seek, refine and spur innovation in a variety of industries [2]. However, given its complexity, highly regulated environment, and reliance on IP concepts [3-7], the Pharmaceutical industry has been relatively slow to adopt OI approaches, particularly in the R&D arena.

us

cr

ip t

Notwithstanding the undeniable barriers to implementation of OI methodologies in pharmaceutical R&D, the literature seems to indicate a shift from the debate around the concept of OI to focus instead in the discussion of actual examples of OI approaches. These approaches were developed by different research institutions worldwide with an emphasis on the description of collaborative models that have shown a certain level of success, and the analysis of the best practices underlying such emerging models [8-12]. At its core, this shift accentuates the significant pull for change driven by the well-known shortcomings of the traditional, closed model for pharmaceutical R&D, paired with the actual usefulness of OI models when correctly deployed and implemented. It could be argued that the recent proliferation of OI approaches in pharmaceutical R&D is actually a manifestation of the inherent openness of the scientific discovery process, and that the implementation of OI methods in pharmaceutical R&D is no longer a philosophical problem of “whether to do it” but instead becomes a logistical problem of “how to do it” [11-12].

Ac ce

pt

ed

M

an

One way to understand the apparent paradox between the difference and complementarity of traditional closed vs OI models in pharmaceutical R&D is to consider that the long-term objectives of both approaches are actually the same – that is, the successful production of new medicines. The main contrast between the two approaches can then be described in terms of different short-term goals and tactics to achieve this end. Thus, the traditional closed innovation model generates, refines and develops all inputs, ideas and hypotheses inside the organizational walls. In contrast, the OI models seek to enrich the diversity of available options by incorporating external inputs, ideas and hypotheses and by possibly refining and developing them internally and externally. Likewise, organizations large and small may not have all the expertise they needed within their walls to resolve a particular challenge. In this manner, OI programs attempt to leverage external parties to develop a large, broad pool of opportunities and talent complementary to those available internally. While traditional R&D concentrates on the systematic prioritization of internal opportunities to focus on those options that survive the internal decision-making funnel, open innovation casts a wider net to capture the largest possible library of options before the utilization of prioritization and selection mechanisms that help to focus on the most interesting or promising opportunities. In addition, traditional innovation flow tends to be depicted as a linear process, whereby promising opportunities move forward and others get put back on a shelf; open innovation requires an iterative process in which inputs may be continually screened and recycled in future searches [13]. Moreover, the efficiency of the traditional R&D paradigm tends to be measured using long term economic indicators (such as number of IND/launches, or cost per IND/launched medicine) that rely on retrospective historical analyses that incorporate the entire R&D process from idea inception to commercialization [14-16]. In contrast, the evaluation of OI models used in early drug discovery struggles to benefit from such historical analyses given a) the relative novelty of the models and b) the difficulty of segregating the contributions of internal vs external inputs with respect to the outcomes of the enterprise at large. More importantly, we argue that macroscopic, retrospective and economic leading indicators traditionally used to measure R&D productivity provide an inadequate way of measuring the efficiency of the early phases of drug discovery in general and of OI approaches in particular. The Pharmaceutical R&D Process 2 Joint Deloitte/Lilly Manuscript

Page 2 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

The pharmaceutical R&D process is generally depicted as a series of sequential stages progressing in linear fashion from hypothesis generation and validation through identification of a clinical candidate, followed by clinical development and finally commercial distribution [15]. While the overall process can be understood as linear, there is a clear inflexion point at Candidate Identification (CI) with a marked differentiation between phases before and after CI, with respect to the types of activities implemented and the manner in which progress is achieved [CHART 1].

an

us

cr

ip t

The R&D process after CI is focused on collecting, understanding and leveraging vast amounts of information about a single molecular entity that is considered competent to test the clinical hypothesis in humans, followed by the careful design and implementation of the clinical experiments required to demonstrate its safety and efficacy in the clinical setting. Thus, development activities after CI are indeed linear and sequential, defined by pre-established protocols and subject to high levels of external regulation and therefore data collection. Performance metrics usually chosen to evaluate the pharmaceutical R&D process in its entirety often emphasize the post-CI phase and are focused on clinical success/attrition, overall published costs and dates, and other public domain or retrospectively collected information which more or less aims to describe the progress of a single Candidate Molecule from its identification as such to its approval for distribution as a medicine [14-16]. CHART 1 Insert Chart 1

Ac ce

pt

ed

M

In contrast, the R&D process before CI takes place iteratively through the repetition of a series of “design-test-analyze” learning cycles fueled by the constant evaluation and prioritization of thousands of distinct molecular entities undergoing biological tests which are carefully designed and selected to provide some sort of indirect indication of future clinical relevance. At this stage, there are massive amounts of information collected on many new molecular entities, but relatively little is known about each one – and rarely does one single chemotype follow a linear path of progression to the end game [17-18]. The process through which diverse chemotypes are evaluated, prioritized and refined – while at the same time probing the biological system of interest – is labyrinthine, fraught with detours and dead ends. The overall goal becomes the generation and incorporation of all information into progressively better biological understanding and better structural designs that are capable of regulating the system in a manner that is most consistent with the underlying clinical hypothesis. Thus, the exercise of calculating leading indicators of success or process performance at this time counts with the inherent difficulties that a) the process has an inherently cyclical nature, and b) the actual chemical entity that may eventually reach clinical development is not yet known, nor are its properties. Meaningful metrics must then be capable of estimating the efficient transmission of information to enable option prioritization with minimal possible investment, regardless of the final outcome. How Should OI Models Be Measured? Open innovation approaches in pharmaceutical R&D are quite varied, based on the details of execution and/or implementation chosen by each company. For example, some of the factors contributing to this variety may include the nature of the arrangements between partners; the actual exchange taking place within the partnership, the type of incentives used and the short-term goals the partners intend to achieve [4, 7, 9, 11, 12]. On one hand, this variety underscores the vibrancy of an emerging field and contributes to the concept of a balanced “ecosystem” with plenty of offerings for all interested participants. On the other, it also increases the complexity of isolating common measurable elements within such a heterogeneous mixture.

3 Joint Deloitte/Lilly Manuscript

Page 3 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

cr

ip t

Approaches to measure the degree of “openness” in the biopharmaceutical industry that are based on a retrospective analysis of publicly available information have been published [19]. Likewise, a general framework for the assessment of OI approaches incorporating IPOO metrics (inputs, process, output, outcomes) but also considering instrumental, conceptual and symbolic metrics has been described [20]. We have previously proposed an OI framework for biopharmaceutical companies composed of a set of qualitative factors, which incorporate aspects of operational complexity and degree of ownership and control [13]. Within the context of the OI models specifically applied to early drug discovery, there remains a need for the description and publication of generally applicable, easily implementable metrics to facilitate the determination of whether a novel OI business approach is adding value to the “traditional” innovation process within a given organization. This understanding is of paramount importance to the survival of any OI approach, not only for the enablement of decision-making within the program per se, but also for the justification of its very existence within the organization.

an

us

With an emphasis on the avoidance of high-level metrics focused on the overall R&D process, which use retrospective analyses of economic returns as described elsewhere [14-16], we now propose a framework incorporating four broad categories that can be applied to the assessment of any OI program focused on the early stages of drug discovery: Investment, Pipeline Health, Return, and Culture & Capabilities [CHART 2]. Although there are a variety of different OI strategies currently being used in the market, this framework can be adapted to assess the key performance indicators of specific programs of interest.

M

CHART 2 Insert Chart 2

Ac ce

pt

ed

INVESTMENT The first category for assessing OI programs in pharmaceutical R&D is Investment, or resources that are being used to support the program. All OI programs require some sort of investment, whether it is being set up as a unique organizational unit or leveraging the resources of other units within the organization. Within the early drug discovery space, broad groupings that may be helpful to consider include compound inventorying costs, data collection and lab costs, dedicated human resources, collaboration incentives, and supporting technology (platform, software, website, etc.) costs. In addition, startup costs may be considered (though these will vary significantly between different programs), and other costs will be shared with other areas within the wider organization. (One example of shared resources might be using empty instrumental capacity for an OI-driven experiment in a traditional R&D unit.) Thus, the specifics of how shared investments may be allocated, whether or not to include startup costs, and other decisions will depend on the particular program being assessed. In the broadest sense, the Investment category will provide insight into the overall resources being used for the purpose of advancing OI-based ideas. This assessment category may also guide decisions by Leadership on whether to scale back or increase investments based on budgetary or operational needs in synergy with internalbased experimentation. PIPELINE HEALTH The second category to consider in evaluating an OI program in pharmaceutical R&D is the Strength of its Opportunity Pipeline. One of the primary goals of any OI program is to source a wide range of ideas and opportunities from a diverse audience, in order to expand the accessible research substrate beyond the internal teams within the organization. Broad groupings to consider include the number and/or variety of proposals received (e.g. new targets, new chemical structures, and new enabling technologies), collected data, as well as the ability for data to be shared across the organization, 4 Joint Deloitte/Lilly Manuscript

Page 4 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

changes in data generation over time, and published scientific results. The opportunity pipeline category will shed light on the diverse nature of opportunities being received and how these opportunities are being moved forward through the program alongside already available internal opportunities. These features allow Leadership to understand how OI programs are increasing possibilities beyond the work being done through more “traditional” R&D means and approaches.

M

an

us

cr

ip t

RETURNS Generally, the returns from early drug discovery are not realized in dollars or clinical validation since clinical trials, let alone product commercialization, are yet to occur. As the ultimate proof of medical and commercial success is poorly linked to its leading indicators at this stage of the process, Returns from OI programs largely take the form of hypotheses tested (proven or rejected) and new research outcomes available to the organization (conclusions or future directions). Potential groupings of measurable outcomes to consider within this space could be the type and variety of therapeutic areas benefitting from the external inputs, the percent of opportunities being moved forward towards advanced evaluation, and the number and type of follow-up research collaborations that result. These returns demonstrate the added value being produced through the OI program vis-à-vis the traditional approaches, and support continued investment in the program. Undoubtedly, a strong program will produce valuable returns both to the pharmaceutical company as well as the partnering institutions in order to remain sustainable. Follow-up collaborations generated from OI programs deliver value to both partners and combine internal and external resources to help advance promising opportunities along the arduous R&D discovery process while also informing other research throughout both organizations.

Ac ce

pt

ed

CULTURE & CAPABILITIES Our final proposed assessment category is focused on the formation of the external network of participants, their adherence to the OI program, and the type and variety of internal capabilities being offered to fuel the pipeline of opportunities. All OI models thrive when they are able to access diverse networks which provide access to “non-liquid” innovation, that is, innovation not readily accessible by more traditional means. In its broadest sense, the Culture of the OI program is reflected in its ability to form and sustain a network, and the Capabilities offered by both the organization and the participants become the reciprocal incentives that make the network possible. In the case of OI programs focused on early drug discovery, a diverse and healthy network is particularly relevant as it produces wider divergence of possible opportunities for consideration at the outset where the investment is minimal; in turn, the capabilities of the organization are leveraged to help differentiate and advance the most promising opportunities produced by the capabilities of the participants. Specific examples of cultural metrics would be the geographic diversity and types of participants and program satisfaction scores. In terms of capabilities, the number and diversity of offerings included in the program are important factors to consider. Overall, these metrics show how successful OI practices result in the creation of a satisfied and engaged network, and to which extent are the internal capabilities successfully leveraged to incentivize the network.

Case Study: Study: the Lilly Open Innovation Drug Discovery Program In order to test and substantiate the metrics framework described above, we obtained detailed data from the Lilly Open Innovation Drug Discovery (OIDD) [21]. Lilly is regarded by many to be a pioneer in the implementation of open innovation models in the pharmaceutical industry, with a long, documented history of collaborative approaches along the entire R&D value chain [22-24]. The OIDD program, first launched in 2009 [25], is a web-based platform that allows external investigators access to Lilly’s early drug discovery scientific capabilities to increase the pool of potential opportunities for further 5 Joint Deloitte/Lilly Manuscript

Page 5 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

us

cr

ip t

development available to Lilly. The program targeted participation by academic laboratories and smaller, independent companies wishing to partner with Lilly on early drug discovery projects. In exchange for enabling the external investigators’ research, Lilly obtains first right of negotiation to access promising opportunities. Within the OIDD program, the fundamental concerns that are central to the invention process are addressed directly at the outset, by explicitly establishing within the legal framework of the program that all IP rights remain with the submitter at all times [26]. To date, several hundred scientists at more than 350 affiliated institutions in 28 countries worldwide have joined the program and are actively engaged in program participation, and details regarding first-generation collaborative outcomes based on the screening offering have been recently described in the literature [27]. Given the high level of external acceptance of the program and the internal impact already realized, the company recently expanded the available offerings by introducing additional engagement models based on capabilities other than compound screening. Thus, participants are able to access computational tools for designing molecules with drug-like properties, submit libraries to be synthesized remotely using Lilly’s automated equipment and request a library of proprietary compounds to test emerging biological assays [28].

Ac ce

pt

ed

M

an

For the purposes of the scope of this publication, we focused our analyses on the longest established offering, OIDD Screening [29], which enables participating scientists to submit chemical compounds for experimental evaluation in Lilly's proprietary biological assays. Submitters electronically enter chemical structures, also known as virtual compounds, into the OIDD user interface, applying an algorithm that allows an automatic selection process to be run in structure-blinded fashion in order to protect the IP rights of the submitters [30] . Once a compound is determined to be eligible for screening, the submitter ships a physical sample to Lilly’s laboratories for testing across a variety of projects or modules associated with specific therapeutic areas of interest to Lilly. Each module incorporates a number of tiered in vitro biological assays (primary, profiling and confirmatory), each of which generates scientific results or datapoints that are then electronically shared with the submitter in real time. As compounds make progress down each module’s testing scheme, associated results are constantly evaluated and used in selecting the compound subset with the desired biological profile: a final set of promising compounds called confirmed actives is then identified [31]. The full package of accumulated datapoints for a given confirmed active compound forms the basis for the establishment of a follow-up collaborative agreement, or a possible publication by the submitter. At this stage, follow up collaborations take many forms depending upon the specific module and data package for the active, as well as the interests and capabilities of the submitter and/or institution. Broad categories of collaborations are those focused on biology (chemical scaffold evaluation or material scale-up) or on chemical diversity (sample acquisition and compound harvesting). As the OIDD Screening offering has been continuously running since the program inception in 2009, there is a robust set of continuous data available for evaluation and analysis. Within this context, we have applied the concepts previously illustrated on Chart 2. For the remainder of this paper, the term OIDD program will be synonymous with the Screening offering as described above. INVESTMENT Note: In our analysis of the OIDD program, we have excluded one-time startup costs such as website and platform software development, initial marketing, publications and infrastructure and have instead focused on understanding and optimizing run rates. The primary investment categories for the OIDD program include 1) the cost of shipping compound samples from affiliated organizations, 2) generating datapoints, 3) collaboration incentives, and 4) 6 Joint Deloitte/Lilly Manuscript

Page 6 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

dedicated full-time employees, or FTEs. These categories, detailed below, provide an understanding of the recurrent investment needed to operate OIDD since its inception, and allow effective business planning and identification of opportunities for increased efficiency.

us

cr

ip t

First, shipping costs include sending empty vials from Lilly to the submitter, and returning filled vials to the Lilly site for evaluation. Second, the program utilized datapoint generation as the actual program currency used to attract participation from submitters. Regarding OIDD datapoint generation, we must note an important nuance that differentiates this investment from billable expenses such as shipping costs: the total amount invested in datapoint generation is not an additional operational expense required to support OIDD activities – rather, this amount results from the estimation of available capacity captured by OIDD research activities, fitting within the natural ebb and flow of ongoing internal research activity, already paid for by normal business operations – that is, sunk costs for instrumentation, plus reagents/materials and FTEs.

M

an

Third, we calculated full spend on collaboration incentives, derived from all follow-up agreements regardless of type. As noted in Figure 1, about half of the follow-up collaborations signed have relied exclusively on some sort of in-kind contribution, requiring no further financial support from Lilly. These collaborations leveraged existing Lilly internal infrastructure and/or scientific capabilities, applied to the completion of experimental work relevant to the collaboration. The remaining balance, or incentive collaborations, comprises those cases where the relevant experimental work is instead completed by the partner, after the transfer of financial support from Lilly. The above categories taken altogether, the OIDD program has invested approximately $7.1M to date since its inception ($150 per compound), as a mix of operational and capabilities investment.

ed

Figure 1 Insert Figure 1

Ac ce

pt

The fourth and final investment to consider is FTE count, which serves as an indicator of program growth and evolution over time. As shown in Figure 2, the OIDD program infrastructure was originally established with IT and scientific business functions alone – before scientists were directly involved. As the program grew beyond the original proof of concept and the number and type of scientific modules included increased over time, the composition of the dedicated team reached its current distribution – an almost even split between employees from IT, Business, and Science functions. Once a critical mass is reached, the number of dedicated FTEs may be scaled up or down based on budgetary and other internal considerations. The OIDD program investment thus described should be viewed in the context of the other evaluation categories, discussed below. Figure 2 Insert Figure 2

PIPELINE HEALTH The second dashboard category is Pipeline Health, which focuses on the progression of opportunities from inception to conclusion. Within the context of the OIDD program, this pertains to the compound structures moving through submission, filtering and acceptance; followed by the associated compound samples progressing through registration, testing, and selection of confirmed actives; and culminating in the negotiation and execution of follow-up collaborations for promising opportunities. Important factors to consider in this category include the pass/fail rate at each phase of the process; the types of 7 Joint Deloitte/Lilly Manuscript

Page 7 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

therapeutic areas benefitting from external opportunities identified by the program; and the number and types of follow-up collaborations being established.

us

cr

ip t

Over the life of the program, OIDD program participants have uploaded more than 400,000 chemical structures to the platform and submitted almost 300,000 for consideration by Lilly, of which about 140,000 structures were accepted by the filtering algorithm. These virtual compounds resulted in the registration of almost 50,000 physical compound samples received at Lilly labs for biological testing. Considering all sources, including external vendors or internal synthesis, this accounts for approximately 10% of the total set of registered chemical entities. Statistical analysis of OIDD inputs over the life of the program demonstrates that on average 500 new compounds are registered each month, resulting in a steady increase in the OIDD compound collection over time [Figure 3]. These figures help Leadership understand the relative value proposition added by OIDD to internal teams, and also demonstrates the efficiency of the platform at narrowing down a large number of varied opportunities to focus on those with the most potential interest to internal teams. A detailed measure of the level of increased chemical diversity accessed by the OIDD program has been discussed elsewhere [29].

Ac ce

pt

ed

M

an

The scientific results generated through OIDD have a double value proposition in that they can be used both by Lilly and by program participants – that is, individual investigators and affiliated institutions who want to publish their research. Each datapoint represents “knowledge gained” or “hypothesis tested” as discussed above, which can be leveraged by both partners at the same time for different purposes (decision-making vs publication). The submitter obtains all datapoints in real time through the OIDD platform, while Lilly scientists are able to identify opportunities to advance internal projects. Additionally, as each compound sample can be tested multiple times in many different modules while inventory remains available, Lilly has the possibility of generating multiple opportunities from each compound accepted and tested with the ultimate goal of identifying previously unknown compounds with a promising biological profile warranting advanced in vivo testing. To date, the OIDD program has generated more than 1.8M datapoints as a result of the testing of more than 45,000 crowdsourced compounds, with clear predomination of Oncology and Endocrine / Cardiovascular therapeutic areas as a reflection of the number and types of Lilly projects launching active screening campaigns over the life of the program. The accumulated datapoints can also be viewed in terms of the phase during which they were produced, with nearly 69% of all data generated during the primary screening phase and 26% as secondary screening. As the complexity and cost of assays increases in that order, this is also a demonstration of applied efficiency of the OIDD program to operationalize a “phased investment” approach. As fewer, most promising opportunities are advanced to later phases, the data generated in the confirmatory phase drops to 5% of total datapoints. Figure 3 Insert Figure 3

RETURNS The almost 50,000 registered compounds are tested in a variety of project modules spanning therapeutic areas of internal interest. For each module, a subset of confirmed actives identified by the testing scheme as the most promising opportunities with the desired biological profile is chosen for advanced evaluation [31]. To date, approximately 1,800 confirmed actives have been moved forward, with approximately two thirds of those associated with Oncology projects [Figure 4]. As indicated above, this simply reflects the preponderance of Oncology projects in the Lilly active project pipeline over the life of the program but it does speak to the alignment of OIDD with internal priorities. Given 8 Joint Deloitte/Lilly Manuscript

Page 8 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

the specific design features of the OIDD platform, about 90% of confirmed actives are classified as small molecules, with the remainder falling into Natural Product, Peptide, and Chemical Probe categories. As an important reference point, this total number of confirmed actives yields an approximate 4% confirmed hit rate over all projects, which provides useful perspective with respect to typical success indicators for early drug discovery.

us

cr

ip t

As some of the confirmed actives identified by OIDD projects move forward to the collaboration negotiation phase, selected submitters may be invited to have these actives added to the Lilly collection for further screening, synthesis and other evaluation. In these cases, larger samples are purchased by Lilly and become available to a wider audience for future internal research uses, therefore providing additional opportunities for collaboration. An additional 1.2% of registered compounds have been added to the Lilly collection via this route, increasing its chemical diversity and providing a further endpoint for the crowdsourcing model [32]. Figure 4 Insert Figure 4

pt

ed

M

an

CULTURE & CAPABILITIES Creating a sustainable network of potential partners and a culture of participation is of paramount importance for an OI approach regardless of the enterprise context. Within Pharma, a vibrant OI community is a leading indicator of the potential richness of inputs – a diverse cohort of external inputs, ideas and hypotheses. Assessing the number of affiliated institutions and submitters is perhaps the best indication on of the size of the OI network and a measure of the diversity of potential opportunities. To date, OIDD has 576 registered affiliates as of 2016, with 388 of those currently considered active. As seen in Figure 5, there was 38% attrition from 2014-2015 following the revision of the Program Agreement in March 2015, which expanded the scope of the agreement between Lilly and OIDD participants [26]. One trend that can be seen in the data is the increased number of active submitters overall, which also shows expansion in the use of OIDD offerings from within affiliated organizations. The average annual affiliation growth rate is 28.4%, and the average annual submitter growth rate is 57.6%, demonstrating strong user intake in the years since program inception.

Ac ce

Other factors that might be of interest to further understand the network composition include types of participant organizations and their geographic distribution. Currently, half of the participating institutions in OIDD are based in the U.S. and about a third is in Europe, with the remaining 20% distributed across all continents. This distribution is also reflected within the context of follow-up collaborations, underscoring the unbiased nature of the crowdsourcing approach and the value of continued investment in leveraging platform technology to enable access to traditionally untapped geographies. Another way to look at the OIDD culture is to analyze participant survey results. OIDD institutions were administered a questionnaire that contained a variety of questions on topics such as their experiences with the OIDD program and the technology platform, and their overall satisfaction. The results indicate that the vast majority (86%) of survey respondents felt highly satisfied with the program, as measured by their likelihood to recommend OIDD participation to others. Figure 5 Insert Figure 5

9 Joint Deloitte/Lilly Manuscript

Page 9 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

cr

ip t

In addition, by segmenting out the survey results by type of responder (e.g. Academic vs. Small Company) or by type of respondent role (Administrator vs Investigator), some interesting differences become apparent. For example, a larger percentage of Small Company participants rated the program highly when compared to Academic participants. The Small Company participants also felt more familiar with, and had more frequent use of, the program offerings and platform. Contributing to this difference is a lesser extent of separation of roles (Administrator vs Investigator) in the industrial versus the academic setting, pointing to interesting cultural differences between customer segments. These results have been helpful to program Leadership as they develop strategies for expanding and managing the growth of OIDD offerings going forward, and may influence approaches used in future collaborative negotiations with potential partners.

M

an

us

Finally, consideration of capabilities within the OIDD context is understood in terms of offerings and modules, as described above [33]. As program capabilities are the central component of participation incentive, there is clear understanding by Lilly Leadership of the need to maintain a sustainable flow of scientifically relevant modules and assays. We looked at the evolution of both program modules and assays over time and observed a decline in the number of assays but only a slight fluctuation in the number of modules (projects). This reflects team-level decisions on which individual screens are going to be included in the primary, secondary and confirmatory stages: thus, we inferred that the decline reflected a continuous push for increased program efficiency and a more deliberate use of datapoints produced for relevant decision-making. On the other hand, the relatively steady number of modules over time reflects on Lilly Leadership’s decisions to expand the scope of the program well beyond the original small selection of projects which were included at the outset [25], to better leverage the value of the crowdsourcing approach across a greater variety of drug discovery areas.

ed

Conclusions

Ac ce

pt

The pharmaceutical R&D environment – particularly in its earliest stages – has been traditionally known as closed and secretive, specifically regarding its protection of intellectual property. At the same time, the need for efficiency and a streamlined R&D process has driven the industry towards change and increased openness. OI approaches in pharmaceutical R&D have the potential to increase the pool of potential opportunities by incorporating starting points beyond the bias of internal knowledge, while also helping introduce efficiencies. Furthermore, OI programs inherently support an iterative process of innovation, allowing for greater number of starting points and therefore potential opportunities, which is consistent with the iterative nature of the early drug discovery process. However, it is challenging to measure the effectiveness of an OI approach in early drug discovery due to the long cycle times, high attrition rates and complexity of the overall process. The proposed dashboard described in this article offers an approach that accommodates the need to quantify the value and impact of early OI programs by focusing on leading indicators that can be measured and controlled: the resource dedication to the program, the robustness and advancement of identified opportunities, and the interaction/sentiment between the organization and its external partners. As an example of this proposal, the dashboard was applied to data facilitated by the Lilly Open Innovation Drug Discovery program (OIDD). The OIDD program includes confidentiality and intellectual property provisions for all our participants, which addresses a fundamental challenge in OI approaches. The total number of OIDD affiliated institutions and submitters is a good indication of network size and measures the diversity of potential opportunities that the organization can secure – a diverse cohort of 10 Joint Deloitte/Lilly Manuscript

Page 10 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

partners generates more inputs, ideas and hypotheses. To date, the OIDD program has generated more than 1.8M datapoints as a result of the testing of more than 45,000 crowdsourced compounds, yielding approximately 4% confirmed actives across all projects. This value is comparable to typical early drug discovery success rates, higher for some projects.

us

cr

ip t

In the case of Lilly, our application of this strategy has also generated insights around the importance of the technological infrastructure that is needed to support OIDD offerings and the ability of the program to: 1) facilitate the exchange of information with external participants while ensuring the protection of their IP, 2) have mechanisms in place to track generated opportunities, 3) integrate with internal systems in order to understand its complementarity with traditional R&D efforts, and 4) have flexibility to quickly incorporate new modules as the program offerings develop over time. Looking forward, the pharmaceutical industry should continue to test, leverage, and benefit from models that improve drug discovery and R&D processes, while effectively quantifying metrics to pragmatically and continuously evaluate these programs – things we believe the proposed dashboard offers.

Acknowledgements Acknowledgements

M

an

The Lilly authors wish to thank Dr Alan Palkowitz and Dr Andy Dahlem for their sponsorship and support of this article; Dr Tim Grese and Dr Jim Myers for technical review and editorial comments; and the entire OIDD team for their hard work, commitment and ingenuity. The Deloitte authors wish to thank Pierce Goldsmith, Rachel Skelly, and Yao Zhang PhD for their contributions to the data analysis and research found in this article. All authors have no conflicts of interest to declare.

ed

References

Ac ce

pt

[1] Chesbrough, H. W. (2003) Open Innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, Boston [2] Chesbrough, H. W., Vanhaverbeke, W., and West, J. (2014) New Frontiers in Open Innovation, Oxford University Press [3] Hunter, J. and Stephens, S. (2010) Is Open Innovation the Way Forward for Big Pharma? Nat Rev Drug Discov. 9, 87-88 [4] Bianchi, M., et al. (2011) Organisational modes for Open Innovation in the bio-pharmaceutical industry: An exploratory analysis. Technovation 31, 22-33 [5] Bennani, Y.L. (2011) Drug Discovery in the Next Decade: Innovation Needed ASAP. Drug Discovery Today 16, 779-792. [6] Khanna, I. (2012) Drug Discovery in Pharmaceutical Industry: Productivity Challenges and Trends, Drug Discovery Today 17, 1088-1102 [7] Schuhmacher, A., et al. (2013) Models for Open Innovation in the Pharmaceutical Industry. Drug Discovery Today 18, 1133-1137 [8] Hunter, J. (2014) Collaboration for innovation is the new mantra for the pharmaceutical industry. Spring 14. Drug Discovery World (http://www.ddw-online.com) [9] Ehrismann, D. and Patel, D.D. (2015) University-Industry Collaborations: Models, Drivers and Cultures, Swiss Med Wkly. DOI: 10.4414/smw.2015.14086 (http://www.smw.ch/content/smw-201514086/) [10] Bentzien, J., et al. (2015) Crowdsourcing in Pharma: a Strategic Framework. Drug Discovery Today DOI: 10.1016/j.drudis.2015.01.011 (http://dx.doi.org)

11 Joint Deloitte/Lilly Manuscript

Page 11 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Ac ce

pt

ed

M

an

us

cr

ip t

[11] Reichman, M. and Simpson, P.B. (2015) Open Innovation in Early Drug Discovery: Roadmaps and Roadblocks, Drug Discovery Toda DOI: 10.1016/j.drudis.2015.12.008 (http://dx.doi.org) [12] Holmes, D. (2016) A New Chapter in Innovation. Nature 533, S54-S55 [13] Executing an Open Innovation Model: Cooperation is Key to Competition for Biopharmaceutical Companies; Deloitte website, accessed 6 September 2016, http://www2.deloitte.com/content/dam/Deloitte/us/Documents/life-sciences-health-care/us-lshcopen-innovation.pdf [14] Kola, I. and Landis, J. (2004) Can the pharmaceutical industry reduce attrition rates? Nat Rev Drug Discov. 3, 711-716 [15] Scannell, J. W., et al. (2012) Diagnosing the Decline in Pharmaceutical R&D Efficiency. Nat Rev Drug Discov. 11, 191-200 [16] DiMasi, J.A., et al. (2016) Innovation in the Pharmaceutical Industry: New Estimates of R&D Costs. Journal of Health Economics 47, 20-33 [17] Hughes, J.P., et al. (2010) Principles of Early Drug Discovery. British Journal of Pharmacology DOI: 10.1111/j.1476-5381.2010.01127.x (http://onlinelibrary.wiley.com) [18] Research & Development in the Pharmaceutical Industry, OIDD website, Accessed September, 2016. https://openinnovation.lilly.com/dd/includes/pdf/R_and_D_White_Paper_Final.pdf [19] Michelino, F., et al. (2015), Measuring Open Innovation in the Bio-Pharmaceutical Industry. Creativity and Innovation Management DOI: 10.1111/caim.12072 (http://onlinelibrary.wiley.com) [20] Erkens, M., et al. (2014) Measuring Open Innovation: a Toolkit for Successful Innovation Teams, Performance 6, 12-23 [21] OIDD website, What is OIDD? https://openinnovation.lilly.com/dd/what-is-oidd/ Accessed September, 2016 [22] Schwartz, K. and Huff, B. (2010) The story of Eli Lilly's open innovation journey: how one company developed a mature model. PDMA Visions 34, 19-22 [23] Lyles, M., Lai, Y., and Dhanaraj, C. (2007), Innovation Without Walls: Alliance Management at Eli Lilly and Company. Harvard Business Review [24] Owens, P.K., et al. (2015) A Decade of Innovation in Pharmaceutical R&D: The Chorus Model, Nat Rev Drug Discov., 14, 17-28. [25] Lee, J.A. et al. (2011) Open Innovation for Phenotypic Drug Discovery: The PD2 Assay Panel, J Biomol Screen., 16(6), 588-602. [26] OIDD Website, Program Agreements, https://openinnovation.lilly.com/dd/getting-started/programagreements.html [27] McGilvray, A. (2016) Compound Screening: Fresh Hunting Ground, Nature 533, S65-S67 [28] OIDD website, What We Offer, https://openinnovation.lilly.com/dd/what-we-offer/ Accessed September 2016 [29] Alvim-Gaston, M., et al. (2014) Open Innovation Drug Discovery (OIDD): A Potential Path to Novel Therapeutic Chemical Space. Current Topics in Medicinal Chemistry 14, 294-303 [30] OIDD Website, IP Protection, https://openinnovation.lilly.com/dd/what-is-oidd/protecting-yourintellectual-property.html [31] Within Lilly, the term “confirmed active” excludes so-called PAIN compounds that interfere with assays or have promiscuous activity, thus over-inflating success rate. For an excellent description of PAINs, see Baell J., Walters M.A. (2014) Chemistry: Chemical Con Artists Foil Drug Discovery. Nature 513(7519):481-3. doi: 10.1038/513481a.[32] OIDD Website, Compound Acquisition, https://openinnovation.lilly.com/dd/what-we-offer/compound-acquisition.html [33] OIDD Website, OIDD Screening, https://openinnovation.lilly.com/dd/what-we-offer/screening.html

12 Joint Deloitte/Lilly Manuscript

Page 12 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Legends Chart 1: Within the R&D process there is a clear inflexion point at Candidate Identification (CI) with a marked differentiation between the iterative “design-test-analyze” learning cycles that define the pre-CI testing phases and the linear and sequential “protocol-driven” development activities that occur after CI.

•

Chart 2: A set of common metrics and a suitable executive dashboard for open innovation platforms, particularly for the OIDD program, focused around four crucial aspects of an open innovation approach: a) investment, b) pipeline health, c) return and d) culture and capabilities.

•

Figure 1: More than half of the follow-up collaborations signed have relied exclusively on some sort of in-kind contribution (leveraging only existing Lilly internal infrastructure and/or scientific capabilities). The remaining balance, or incentive collaborations, comprises those cases where the relevant experimental work (shown in the bar chart) is instead completed by the partner, after the transfer of financial support from Lilly.

•

Figure 2: The original OIDD program infrastructure was established with IT and Business functions alone. As the program grew, and the volume and complexity of submissions increased, the composition of the dedicated team evolved to an even distribution of IT, Business, and Scientific resources.

•

Figure 3: Statistical analysis of OIDD inputs over the life of the program demonstrates that on average 500 new compounds are registered each month, resulting in a steady increase in the OIDD compound collection over time. Since 2009 almost 2 million additional datapoints have been generated and leveraged to support the full complement of Lilly’s scientific research efforts.

•

Figure 4: The distribution of the 1,800 confirmed actives identified from the nearly 50,000 registered compounds represents an approximate 4% confirmed hit rate over all projects and reflects Lilly’s internal research priorities.

•

Figure 5: Growth of the OI network is a positive indicator of the increasing diversity of potential opportunities. The proliferation of submitters within active affiliations can be directly attributed to the realized value of leveraging the OIDD offerings by the affiliated organizations.

Ac ce

pt

ed

M

an

us

cr

ip t

•

Author biographies 13 Joint Deloitte/Lilly Manuscript

Page 13 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Glenn Carroll

Ac ce

pt

ed

M

an

us

cr

ip t

Glenn Carroll, Principal, Deloitte Life Sciences Consulting, has over 15 years of biopharmaceutical experience. Glenn has significant consulting experience in compliance, R&D, medical affairs and pharmacovigilance. He leads Deloitte’s Medical Affairs and Drug Safety Practices and deploys global teams to implement new operating models and technology transformations, including large compliance remediation efforts. Glenn has published eminence papers in R&D, pharmacovigilance, medical affairs and secondary uses of EHR in clinical development and post launch. He holds a BE in biochemical engineering from University College Dublin and an MBA from Imperial College London.

Adam Volini Adam Volini, Manager, Deloitte Life Sciences Consulting, has over 9 years of experience, including basic research in the field of oncology. Adam has consulting experience across the R&D value chain. He has delivered profitability strategies, future state operating model and process designs, market and business insights, and functional and user requirements. Adam has co-authored Deloitte’s recent white papers on executing open innovation in the biopharmaceutical industry and on establishing industry-leading medical affairs operations. He holds a BS in molecular biology from Cornell University and an MBA with a concentration in health sciences management from the University of Rochester.

14 Joint Deloitte/Lilly Manuscript

Page 14 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Sanjay Srivastava

an

us

cr

ip t

Sanjay Srivastava, PhD, is a senior manager at Deloitte Life Sciences Consulting. Dr Srivastava’s experience combines a scientific background in cell biology, DNA transfection and microbiology with his more than 15 years of experience building and managing consulting project teams, identifying value drivers and driving business strategies and operations. Dr Srivastava is actively involved in working with clients to address the value-based care paradigm and to develop capabilities in cellular therapies or diagnostic tools leveraging NGS sequencing data. Dr Srivastava holds a PhD in bio-organic chemistry from the University of Maryland with post-doctoral training at Johns Hopkins and Cornell Universities. He is also certified as a Lean Six Sigma green belt.

Ac ce

pt

ed

M

Marta Piñeiro-Núñez Marta Piñeiro-Núñez, PhD, is Executive Director, Lilly Open Innovation Drug Discovery. With close to 20 years of experience in the pharmaceutical industry, Dr Piñeiro-Núñez has accountability for the external innovation portfolio in the discovery chemistry organization at Lilly. After a decade as a medicinal chemist contributing to drug discovery projects in cognition, migraine and psychosis, Dr Piñeiro-Núñez led a variety of efforts to increase efficiency in pharmaceutical R&D and also drove the implementation and launch of the Lilly Open Innovation Drug Discovery program and website. Dr PiñeiroNúñez holds a PhD in organic chemistry from Indiana University, completed postdoctoral training at Colorado State University and is a certified Six Sigma black belt.

15 Joint Deloitte/Lilly Manuscript

Page 15 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Tatiana Vetman

ed

M

an

us

cr

ip t

Tatiana Vetman is a Consultant at Lilly Open Innovation Drug Discovery. Tatiana has over 15 years of experience in the pharmaceutical industry, including more than a decade as a medicinal chemist working in neuroscience and endocrine projects. Over her scientific career, Tatiana has contributed to the delivery of five clinical drug candidates, one of which is currently undergoing Phase I clinical studies. Tatiana recently transitioned from the bench to her current externally focused role within Lilly Research Laboratories, where she is responsible for scientific operations and external relationship management for the global discovery chemistry division, which includes portfolio management for the Lilly Open Innovation Drug Discovery program.

Chart 1: Within the R&D process there is a clear inflexion point at

pt

Candidate Identification (CI) with a marked differentiation between the

Ac ce

iterative “design-test-analyze” learning cycles that define the pre-CI testing phases and the linear and sequential “protocol-driven” development activities that occur after CI.

16 Joint Deloitte/Lilly Manuscript

Page 16 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Chart 2: A set of common metrics and a suitable executive dashboard for open innovation platforms, particularly for the OIDD program, focused

ip t

around four crucial aspects of an open innovation approach: a) investment,

Ac ce

pt

ed

M

an

us

cr

b) pipeline health, c) return and d) culture and capabilities.

17 Joint Deloitte/Lilly Manuscript

Page 17 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Figure 1: More than half of the follow-up collaborations signed have relied exclusively on some sort of in-kind contribution (leveraging only existing Lilly internal infrastructure and/or scientific capabilities). The remaining

ip t

balance, or incentive collaborations, comprises those cases where the

relevant experimental work (shown in the bar chart) is instead completed

2% 4% 14%

Chemical Scaffol

Sample Acquisit 79%

Material Scale-U

ed

M

an

us

cr

by the partner, after the transfer of financial support from Lilly.

Ac ce

pt

Compound Harv

18 Joint Deloitte/Lilly Manuscript

Page 18 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Figure 2: The original OIDD program infrastructure was established with IT and Business functions alone. As the program grew, and the volume and complexity of submissions increased, the composition of the dedicated

ip t

team evolved to an even distribution of IT, Business, and Scientific

Ac ce

pt

ed

M

an

us

cr

resources.

19 Joint Deloitte/Lilly Manuscript

Page 19 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Figure 3: Statistical analysis of OIDD inputs over the life of the program demonstrates that on average 500 new compounds are registered each month, resulting in a steady increase in the OIDD compound collection

ip t

over time. Since 2009 almost 2 million additional datapoints have been

Ac ce

pt

ed

M

an

us

cr

generated and leveraged to complement Lilly’s scientific research efforts.

20 Joint Deloitte/Lilly Manuscript

Page 20 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Figure 4: The distribution of the 1,800 confirmed actives identified across nearly 50,000 registered compounds represents an approximate 4% confirmed hit rate over all projects and reflects Lilly’s internal research

Ac ce

pt

ed

M

an

us

cr

ip t

priorities.

21 Joint Deloitte/Lilly Manuscript

Page 21 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Figure 5: Growth of the OI network is a positive indicator of the increasing diversity of potential opportunities. The proliferation of submitters within active affiliations can be directly attributed to the realized value of

Ac ce

pt

ed

M

an

us

cr

ip t

leveraging the OIDD offerings by the affiliated organizations.

22 Joint Deloitte/Lilly Manuscript

Page 22 of 23

Measuring the Effectiveness and Impact of an Open Innovation Platform

Revised Manuscript

Ac ce

pt

ed

M

an

us

cr

ip t

Highlights: • Open innovation (OI) is used across industries, including most pharmaceutical companies • Measuring OI effectiveness is challenging because of long drug discovery cycle times • Clinical attrition and outcomes do not assess effectiveness of early drug discovery • Measuring systems in drug discovery must focus on attributes of its proximal inputs and outputs • Meaningful efficiency metrics should consider the cyclical nature of the drug discovery process • A practical dashboard in early drug discovery incorporates performance and cultural metrics

23 Joint Deloitte/Lilly Manuscript

Page 23 of 23