- Email: [email protected]
Selecting winning new product projects: Using the NewProd system
34
J
PROD
INNOV
MANAG
1985;2:34-44
oc>oo
Selecting Winning New Product Projects: Using the NewProd System Robert G. Cooper
Separating probable winners from probable losers is the goal of the new product screening task, and Professor Robert G. Cooper has developed a model that does this with remarkable success. In this article, he reviews various approaches to new product screening and then presents the basics of the NewProd model. NewProd now has a history of use in industry that seems to be fulfilling its original research promise. Professor Cooper shows how managers can build their own screening models and outlines how such models can contribute in an important way to better new product selection decisions. Over the years, Professor Cooper has conducted a series of major research projects that have aimed at improvements in the new product process. Their hallmark has been managerial relevance and a sound theoretical foundation. This article, the third that Professor Cooper has published in JPIM, is in the same tradition.
Address correspondence to Professor Robert G. Cooper, Faculty of Business, McMaster University, Hamilton, Ontario, Canada L8S 4M4. This work was supported by a grant from the Office of Industrial Innovation, DRIE, Canadian Federal Government. 0
1985 Elsewer
52 Vanderbilt
Science
Ave.,
Publishing
New York,
NY
Co.,
Inc.
10017
New Products: A High-Risk Endeavor
New products, although central to many firms’ successes, are plagued by difficulties and uncertainties. Hopkins reports that for every 100 industrial new products launched, about 40 fail in the marketplace [ 151. Crawford estimates the failure rate to be about 35% [12]. Similarly, our research shows that for every 100 products that are fully developed, only 60 become commercial successes [ 111. The attrition rate of new product projects is equally disturbing: Booz Allen and Hamilton report that for every seven concepts that enter the new product process, only one becomes a commercial success; further, 46% of the resources that U.S. industry devotes to new products are spent on products that fail or are cancelled. New product project selection becomes a pivotal task in the desire to maximize returns from a firm’s new product program. There are far more new product ideas or projects conceived than resources to commercialize them [lo]. Moreover the great majority of these projects are unfit for commercialization. In an ideal new product process, management would be able to identify the probable new product winners in advance, and be able to allocate the firm’s development resources to these projects. As a result, failure rates would be low, misallocated resources would be kept to a minimum, and the return would be maximized. The idea screening stage is the first of these project selection stages in the new product process [ 10,161. It is the decision point at which more new projects are 0737-67X2/85/$3.30
SELECTING
WINNING NEW PRODUCT
J PROD INNOV MANAG
35
1985;2:34-44
BIOGRAPHICAL SKETCH Robert G. Cooper is Professor of Marketing at the Faculty of Business, McMaster University, Hamilton, Ontario, Canada, and also Director of Research of the federal government’s Canadian Industrial Innovation Centre, Waterloo, Ontario. He holds bachelor’s and master’s degrees in Chemical Engineering, and an M.B.A. and a Ph.D. in Business Administration. Dr. Cooper is a noted researcher, consultant, and lecturer in the field of industrial product innovation, and has published over two dozen articles and books on the topic. Many readers will be familiar with his widely acclaimed research study published in his book, Project Newprod: What Makes a New Product a Winner?, Montreal: Quebec Industrial Innovation Centre, 1980, from which part of this article is adapted.
‘killed” than at any subsequent stage. Moreover, it is a critical decision in the sense that, at this point, resources are first allocated to a project: too weak a screening procedure fails to weed out the obvious “losers” or “misfits,” with the resulting misallocation of scarce resources and the start of a creeping commitment to the wrong projects. In contrast, too “rigid” a screen results in many viable and worthwhile projects being rejected, and perhaps is even more costly to the firm in terms of lost opportunities.
New Product Screening: Current Approaches
A myriad of approaches and schemes have been proposed to aid in the initial or idea screening stage of the AUTHOR’S NOTE: The NewProd model presented in this article traces its roots back to earlier research undertaken by the author [6,8,9]. The idea for the model came about almost as an afterthought from research we undertook into why new products succeed or fail. The model, in conceptual form, was first presented in a theory article [lo]. Even though the model was not operational at that time, it attracted considerable attention, so much so that it was decided to operationalize the model, and computer software was written. Since that time, several institutes, including the Wisconsin Center for Product Exploration (University of Wisconsin, Engineering School) and the Industrial Innovation Centre-Montreal, have made NewProd available to local commercial users. To date, a number of major corporations have had experience with the NewProd model (or ProCon, as it is called at Wisconsin), and because of their interest and positive experiences, we felt the time appropriate to share some of these experiences with others, and hence this article. For more information on NewProd, contact the author.
new product process. But the selection or design of an appropriate screening tool must be made in the light of the following considerations.
A tentative commitment in a sequential process. The screening decision is only the first GO/KILL decision in a sequence of such decisions for the new product project under consideration [ 11. A GO decision is not an irreversible decision, nor is it a decision to commit all the resources needed for the entire project. Rejection and acceptance errors. The screening decision cannot be a perfect one, but must strive for a balance between errors of acceptance and rejection. It should not be overly conservative and rigorous, accepting only the “sure bets”; nor should it lead to the dissipation of resources over a large number of unwarranted projects. Uncertainty of information and absence offinancial data. The screening decision amounts to an investment decision, but one made in the absence of concrete financial data [lo]. The richest and most accurate data are typically found at the end of product development, or better yet, during commercialization [ 11. At the initial screening stage, however, financial data are often missing altogether, and where available, are probably very uncertain [22]. Even qualitative data are often subjective and fairly unreliable. Multiple objectives and evaluation criteria. The criteria used in the screening decision should reflect the corporation’s overall objectives, and in particular, its goals for its new product program. Not all of these criteria are quantifiable, nor are they necessarily internally consistent. Also, both the data available and the criteria are time variant, i.e., change or become better defined as the project progresses [2,7,14,19]. Realism and ease of use. The screening decision method must be amenable to implementation. The decision tool must be realistic, i.e., not require so many simplifying assumptions so as to invalidate its results. And the tool must be easy to use-data requirements, computational effort, and interpretation of results must be consistent with managers’ capabilities.
J PROD INNOV MANAG 1985;2:34-44
36
No one screening method available today meets all the requirements for an ideal screening tool as outlined above. Instead, a variety of methods have been developed, and each incorporates some of the desired features. The four main approaches to initial screening include [2,5,17]: 1. 2. 3. 4.
Benefit measurement models. Economic models. Portfolio selection models. Market research approaches.
Benefit measurement models require a well-informed respondent or group to provide subjective inputs regarding characteristics of the project under consideration [3]. Such models are systemmatic procedures for soliciting and integrating benefit data. They typically avoid conventional economic inputs, but rely more on subjective assessment of proxies for new product success and payoffs, and fit with corporate objectives. Included in this category are checklists, and their extension, scoring models. In the latter, ratings of a project’s attributes are sought and combined in a weighted fashion to yield a numerical project score. Economic models treat the idea screening decision much like a conventional investment decision. Computational approaches, such as payback period, breakeven analysis, return-on-investment, and discounted cash flow methods, are used. To accommodate the uncertainty of data, techniques of probability, including Monte Carlo simulation, risk analysis, and decision tree analysis, have been proposed. But at the idea screening stage, economic models suffer because they require considerable financial data as inputs when often relatively little is known about the project. Thus, such models are usually considered more relevant for “known” projects (line extensions, product modifications, etc.), or at later stages of the new product process. Portfolio selection models view the screening decision as part of the total resource allocation problem. Such techniques largely involve operations research constrained optimization methods, such as linear, integer, and dynamic programming [ 131. The objective is to develop a portfolio of new and existing projects in order to maximize an objective function, yet subject it to a set of resource constraints. Because these mathematical models require substantial data inputs, including financial data on all projects, timing information, resource needs, and availabilities, they are rarely used
PI.
Market research approaches are usually limited to
R. G. COOPER
relatively simple consumer products, such as packaged goods. Such techniques assume that the critical criteria for the GO/KILL decision solely involve the marketplace; technological and production issues are either obvious or simple to solve. Given a market-based screening decision, it makes sense to use a variety of market research techniques, ranging from consumer panels and focus groups to perceptual and preference mapping, to screen product ideas. Of all the dozens of screening models proposed, benefit measurement approaches are generally recommended for new product idea screening. Because only a tentative commitment is required, and since available information on the project is limited, benefit models become the logical screening tool [23]. For example, the Conference Board reports that about half the firms they studied have set forth writen guidelines or rules for project selection [ 161. Usually these are in the form of checklists or scoring models. Souder investigated 26 project selection models and found that managers rated scoring models best in terms of ease of use and cost to implement [23]. Souder concludes that scoring models are “highly suitable for preliminary screening decisions where only gross distinctions are required among projects. ”
Scoring Models: Their Shortcomings
In spite of their popularity, scoring models are plagued by difficulties [20]. Such models rely on the subjective ratings of managers, and hence data input may not be very reliable. However, at the screening stage, management opinion is often the only data available. Moreover, ratings from several evaluators together with confidence scores can be combined to yield a composite and more reliable value for each input variable. The premise here is that the “average” decision maker is near optimal; unfortunately, none of us is ‘ ‘average. ’ ’ Other criticisms tend to be of a technical nature. Often, scoring models are seen as oversimplifications, since they attempt to reduce a complicated decision situation to a product score [20]. A major deficiency is the arbitrariness of items or checklist questions used, and that importance weightings assigned to each criterion are also arbitrarily determined. The selection of these questions and weights are no doubt based on the judgment and past experience of the model developer.
J PROD INNOV MANAG 1985;2:34-44
SELECTING WINNING NEW PRODUCT
However, his or her experience may be limited to a handful of projects, while his/her ability to judge how important each item is and to translate these into numerical weights, may be limited. Freimar and Simon propose the use of linear discriminant analysis of a large number of past successes and failures in order to identify the weights to attach to each screening variable [20]. Another weakness is the fact that many of the variables or items are not independent of each other. For example, if one of the screening items is “compatibility with distribution channels,” then it certainly is not independent from the “compatibility with current products” measure. (Both items are taken from O’Meara’s model [ 181). The result is that certain items are double counted. Schocker, Gensch, and Simon note that factor analysis of the many screening variables to a subset of independent factors or dimensions could be used to eliminate the interdependence of ratings [21].
The Advantages In spite of these criticisms, the scoring model is perhaps the best idea screening tool available. Proponents argue that such a model has utility for a number of reasons: it helps make a highly judgmental decision somewhat more objective; it systematizes the review of projects; it forces managers to subject each project to a consistent and large set of review criteria; it focuses attention on the most relevant issues; it requires management to state goals and objectives clearly; it is easy to understand and use; and it is generally applicable.
The NewProd Model
The NewProd screening model is essentially a scoring model, but with several important differences. Like a scoring model, it is based on the premise that a project’s desirability, attractiveness, or eventual success can be predicted by examining the profile of the project. That is, there exists a “winning” or desirable profile that is fairly predictive of product outcomes. As in most scoring models, NewProd evaluators rate the project under consideration on a large number of criteria; zero-to-ten rating scales are used. These evaluators’ inputs are then combined mathematically to yield a product score, and in the case of NewProd, a likelihood of product success.
37
The difference is that NewProd was derived from a large number of past new product successes and failures. In short, the questions or items used and the weights attached to these items are based on a statistical analysis of almost 200 projects from 100 companies. And the model has been validated, yielding a predictive ability-ability to predict accurately a product success or failure-of approximately 84%.
The Development
of NewProd
Here’s how the NewProd model was derived. A total of 195 industrial new product cases from 102 firms was identified. Half of these were commercial successes, i.e., met or exceeded the acceptable financial return for this type of investment. The other half had been launched, but were subsequently rated as commercial failures. For each project, managers were asked to rate the project on each of 80 characteristics. Zero-to-ten rating scales were used. Of the 80 characteristics, a total of 48 were judged to be potentially useful screening criteria, since they would be known at the outset of the project. The 48 characteristics described such features of the project as: its marketplace; for example, market growth, level of competition, etc. the product advantage; for example, uniqueness, quality, superiority, etc.
size,
product
the project-company fit in a number of key areas ranging from distribution fit to technology resource compatibility. the newness of the project to the firm; for example, market newness, technological newness, etc. and the project itself; for example, its magnitude, complexity, determinateness, etc. Next, these 48 characteristics were reduced to 13 underlying factors or dimensions, which were independent of each other, yet captured the original 48 variables. Factor analysis was used, as suggested by Schocker, Gensch, and Simon [21]. These 13 factors are essentially composite variables comprised of the original 48 variables, each variable with its own weight or loading on each factor.
38
1PROD INNOV
MANAG
R. G. COOPER
1985;2:34-44
Next, an equation was developed between degree of commercial success (or failure) and these 13 underlying dimensions. l Of the 13 dimensions, eight were linked to product outcomes in a significant way. Both multiple regression and linear discriminant analysis were used, and yielded essentially the same results. The regression results were marginally better and were selected for use in the model. The eight key dimensions, their weights or coefficients in the success equation, and typical questions or items that load on each of the dimensions are shown in Table 1. The resulting model was validated using a crosssplit half method. The original sample was randomly split into two halves. One half was used to generate a model; the other half of the sample was used to test the model, i.e., see whether the model correctly predicted a new product success or failure. The procedure was repeated, but this time the halves of the sample were switched. Virtually identical models were obtained from both halves; therefore, one is justified in combining the sample halves and deriving a combined model (Table 1). Predictive abilities were excellent; the model accurately predicted successes 82.6% of the time and failures 86.0% of the time, for an overall predictive ability of 84%.
Product Advantage
Screening Criteria
Product Economic Advantage for the user-the second marketability factor-involves a lower price-for-value product and customer cost reductions. One other factor-namely product innovativeness-is not a dimension closely linked to success. Note that product innovativeness was a dimension identified in the factor analysis, but was not found to be tied, either positively or negatively, to project outcomes. Projects strong on this dimension involve highly innovative, new-to-market products, the first into the market with only a potential demand, and unique task products. But in spite of their innovativeness, such products fared no worse or better than less innovative products.
Market Criteria A total of eight factors or underlying dimensions were found to impact on new product outcomes (Table 1). Two factors describe the market opportunity or market attractiveness. The first of these captures the magnitude of the market opportunity: being in a high growth, large, and high need market. The second is entering a market where competitive intensity is low: few competitors, little price competition, few new products, and relatively static user needs. This latter factor gauges the ease of (or lack of resistance to) exploiting the market opportunity. Note that a third market descriptor uncovered in the factor analysis-the existence of a dominant competitor with a loyal customer base-is not decisive in determining product success or failure. ’ “Degree of commercial success” was defined from a financial standpoint: the degree to which the product’s profitability exceeded (or fell short of) the acceptable profitability level for this type. of investment. A “minus five” to “plus five” scale was used, where +5 = far exceeded the acceptable profitability and -5 = fell far short of the acceptable level.
The product’s marketability is also important to success. For marketability, it is critical to achieve a differential advantage either via the product’s design, features, and quality, or by virtue of the product’s economics to the end user. Product Superiority is the most important factor in the success equation, having a new product which: is superior to competitors’ meeting customer needs;
products in terms of
has unique features for the user, not available on competitive products; is higher quality than competitors’ products, e.g., more reliable, lasts longer, tighter specs, etc.; permits the customer to do a task he/she could not do with what was previously on the market; reduces the customers’
costs; and
is a highly innovative kind on the market.
product-
the first of its
Synergy Criteria Project-company fit dimensions were also important to product outcomes, Overall Project-Company Resource Compatibility was the more important, and included synergies in the areas of managerial skills, market research talents, sales force and distribution resources,
SELECTING
Table 1. NewProd
Screening
product superiority,
overall project/co.
quality,
resource
market need, growth,
advantage
F value
and uniqueness
1.744
68.7
product product product product product product
compatibility
1.138
30.0
a good “fit” between needs of project and company resource base in terms of: managerial skills marketing research skills salesforceidistribution resources advertising/promo resources financial resources engineering skills R&D resources production resources
0.801
12.5
high need level by customers large market ($ volume) fast growing market
0.722
10.2
product reduces customers’ costs product is priced lower than competing
(factor name)
and size
of product to end user
newness to the firm (negative)
technological
resource
market competitiveness
product
constant
scope
Model: Key Factors and Weights” Regression coefficient (weight of factor)
Key factors or dimensions
economic
J PROD INNOV MANAG 1985;2:34-44
WINNING NEW PRODUCT
compatibility
(negative)
-0.354
0.342
-0.301
0.225
0.328
“R2 = 0.420. Adjusted R* = 0.395. F(8,186) = 16.83. Std error = 2.73
Variables
or items loading on factor
is superior to competing products has unique features for user is higher quality than competitors does unique task for user reduces customers’ costs is innovative-first of its kind
for product
class
products
2.9
project new new new new new new new
takes the firm into new areas for firm such as: product class to company salesforceidistribution types of users’ needs served customers to company competitors to company product technology to firm production process to firm
2.5
a good “fit” between needs of project and company resource base in terms of: R&D resources & skills engineering skills & resources
2.0
intense price competition in market highly competitive market many competitors many new product intros into market changing user needs
0.9
a market-derived new product idea not a custom product, i.e., more mass appeal a mass market for product (as opposed to one or a few customers)
39
40
R. G. COOPER
J PROD INNOV MANAG 1985:2:34-44
advertising and promotion resources, financial resources, R&D and engineering talents, and production capabilities. Newness to the Firm is a negative factor for project outcomes. Projects involving a new product class to the company, new distribution and salesforce approaches, new types of customer needs, new advertising and promotion methods, a new clientele, new competitors, new product technology, or a new production process for the firm fare more poorly. The final important project-company fit dimension is Technological Resource Compatibility, a positive factor. Technologically compatible projects are those that fit well into the firm’s existing base in terms of R&D resources and skills, and engineering resources and skills. But note that production and technological newness (also uncovered in the factor analysis)-projects involving new production processes and technologies to the firm-has no impact on product outcomes.
Other Criteria A group of factors were identified that we later called ‘ ‘project descriptors. ’ ’ Most of these, including Product Technical Complexity and Product Determinateness (how clear the product specification and technical solution are at the outset of the project), have no link to product outcomes. Only one entered the success equation, but in a fairly weak manner, namely Product Scope: custom products aimed at one or a few customers fare more poorly than do mass-market, broad-appeal, more standardized new products.
Using the NewProd Approach
The NewProd model has been employed in a number of companies. Initial use has typically been to validate or to check out the use of the model and to determine its predictive ability. For example, in one firm, eight new product projects were scored by up to ten evaluators. All eight products were eventually launched, and their market results compared to the NewProd predictions. In all eight cases, the model accurately predicted the product’s outcome: strong success; marginal success; marginal failure; or strong failure. (Note that NewProd predicted a strong failure in one of the
cases, but commitment to the project was so strong that it was launched in spite of the negative prognosis; and the outcome was as predicted, i.e., a strong failure .)
Developing
Your Own Model
The methodology that we employed to derive the NewProd model can also be used by organizations in order to develop their own screening models. The basic steps are: 1. Develop a reasonable set of screening items or questions. The items we used in NewProd are probably a good place to begin. 2. Identify a sample of past new product successes and failures within the corporation. 3 Request one or more evaluators to rate each of these past projects on each of the criteria developed in item 1. 4. Using appropriate statistical techniques, e.g., factor analysis, multiple regression, and/or linear discriminant analysis, derive a subset of key underlying dimensions and a success equation. 5. Validate the model. Use either a cross split half method (described above) or better yet, some new test cases (as we have subsequently done). 6. Develop the computer software to handle the evaluator’s inputs. (Both batch and user friendly interactive programs for an IBM-PC have been written for NewProd.) 7. Establish a procedure within the firm to facilitate the use of the model, beginning with a teaching session in a seminar format. A number of companies are presently developing their own versions of the NewProd model, using the above procedure. While NewProd was derived for industrial products, such approaches also have applicability for other types of products. A major packaged goods firm, an organization involved in the production of television shows, and a pharmaceutical company, are at the early stages of similar model developments using the NewProd methodology. The NewProd model, as presented here, has also been operationalized for public use (see author’s note in box inset). The procedure is as follows: 1. Up to ten evaluators are selected to review a proposed new product project. Some of these evaluators are closely involved in the project; others are
SELECTING WINNING NEW PRODUCT
41
J PROD INNOV MANAG 1985;2:34-44
Table 2. Factor Scores: A Profile of the Project
Factor name 1. 2. 3. 4. 5. 6. 7. 8.
2.
3.
4.
5.
Product superiority/quality Economic advtg to user Overall company-project fit Technological compatability Newness to the firm Market need, growth, & size Market competitiveness Product scope
Evaluator
1
1.17 -0.25 -0.05 -0.20 -0.36 0.88 -2.05 1 .oo
chosen because they are well informed on the topic, but not directly connected to the project. Most users of the model attempt to put together a multidisciplinary evaluation team for each project. For example, one chemical company, which has had considerable experience with NewProd, brings together evaluators from Sales, Marketing, Product Management, Engineering, R&D, Production, and Finance. A preliminary briefing session is held, where the proposed product and project is described to the evaluators. This meeting is descriptive and not evaluative in nature, i.e., evaluators are permitted to ask questions about the project, but are discouraged from offering opinions about its pros and cons. Each evaluator then independently rates the project on each of the 48 items or questions. Items are presented in statement form, and the evaluator indicates whether the statement describes the project or not (disagree/agree: 0 to 10 scales). In addition, for each item, the evaluator indicates how confident he or she was in providing the rating (again a 0 to 10 confidence score). The model administrator collects the evaluation forms and forwards them for NewProd data processing . Upon receipt of the computer output, a debriefing meeting is held. Here, the team of evaluators review each others’ inputs, flagging points of disagreement. The scores on each of the important eight underlying factors are discussed, noting the positive or negative impact of each on the projects’ merits. Finally the product score and likelihood of success for each evaluator and the “average evaluator” are reviewed.
The following product screening case best illustrates this final debriefing or results interpretation session.
Evaluator
2
Evaluator
1.11 -0.44 0.16 0.04 -0.15 1.36 -2.11 1.05
3
1.33 -0.98 -0.61 -0.55 -0.22 0.01 -1.03 0.60
Mean weighted evaluator
Std. Deviation
1.19 -0.49 -0.16 -0.19 -0.24 0.88 -1.82 0.90
0.10 0.31 0.32 0.24 0.09 0.56 0.50 0.20
Example
The example used is the evaluation of a new A.C. motor control. 2 In this case, three evaluators answered the 48 screening questions and indicated their confidences on each of the 48. The evaluators’ input data was then displayed, together with the scores of a ‘ ‘mean weighted evaluator, ’ ’ based on the inputs of the three people (table not shown). Next, the evaluators’ 48 ratings were reduced to the eight dimensions or factors to yield a profile of the project (Table 2). Here, factor scores are shown on a scale, such that zero is neutral, and high values are plus or minus one. So from Table 2, and reading down the “mean evaluator column” we see that the new product: is truly a superior product, with real benefits to the user (factor 1: mean score = 1.19) is entering a very uncompetitive mean score = - 1.82)
market (factor 7:
is entering an attractive market-growth, potential, size, etc. (factor 6: mean score = 0.88) and so on. The “pros” and “cons” of the project are identified in Table 3. Here the impact of each element in the product’s profile is shown. For example, the strongest positive feature of this project is the product superiority (factor l), while its weakest facet is the lack of an economic advantage to the end-user (factor 2). 2Disguised project and names.
R. G. COOPER
1PROD INNOV MANAG
42
1985;2:34-44
Table 3. Factor Scores and Impacts: Factor Impact Table
Factor name 1. 2. 3. 4. 5. 6. 7. 8.
Product superiority/quality Economic advtg to user Overall company-project fit Technological compatability Newness to the firm Market need, growth, & size Market competitiveness Product scope
Mean evaluator 1.19 -0.49 -0.16 -0.19 -0.24 0.88 - 1.82 0.90
Table 4. Overall Project Ratings: and Probabilities
Project Scores
Evaluator
Name
Score
Probability
R. Booth C. Clayton G. Franks
3.71 4.12 1.55
91.3 93.5 71.5
Mean wt’d eval
3.32
88.8
(%)
Impact strong positive negative marginal (negative) marginal (negative) marginal (positive) strong positive positive marginal (positive)
Probability of Success Finally, in Table 4, the product scores are computed (based on the regression equation of Table 1). These scores are usually in the -5 to +5 range, where + 5 means “a great financial success.” From these scores, the percent likelihood of success is computed, again by calibrating against our past cases-the almost 200 known successes and failures. From Table 4 we see that all evaluators rated the project positively; that the evaluators 1 and 2 were extremely optimistic on the project (success likelihoods of 91% and 93%); that evaluator 3 was more pessimistic, with a resulting success probability of 7 1%; and that overall, the likelihood of success was 89%-a very positive result. Also in Table 4, the pros and cons are again displayed, this time in order of impact on the project. Diagnosis In a debriefing meeting, involving all evaluators, discussion focused on diagnostic questions, such as: .
Why did evaluators 1 and 2 differ so much from evaluator 3? (All three evaluators were present in the meeting. )
.
What are the main “pros” of the project? What is the relative impact of each? And, are we reasonably confident about our assessments of these positive features?
.
What are the “cons” of the project? How damaging are they? And most important, can we do anything to rectify, or at least, diminish them?
The various tables provided by the computer (not all shown) help in this diagnostic stage.
model
1 2 3 MN
Pros of the project (in descending order)
Cons of the project (in descending order)
1. Product superiority/quality 6. Market need, growth, & size 7. Market competitiveness
2. Economic of user
advtg
Marginal factors (from positive to negative) 8. Product scope 5. Newness to the firm 4. Technology compatability 3. Overall companyproject fit
The Model in Use
Besides yielding a product score and likelihood of success, a scoring model, such as NewProd, provides important diagnostic outputs. Indeed, some users claim that the diagnostics are more valuable than the product score and success liklihood. Experience in using NewProd shows that only a handful of projects are clearly black and white situations. Few projects are rated very negatively, wherein the great majority of factors are negative and the liklihood of success is 30% or less. Very positive projects are also in a minority: projects where all important factors are positive, where the success likelihood is 70% or better, and where the variance among evaluators is small. More often than not, projects receive a mixed evaluation, i.e. , some positive features, some negative features, a resulting success likelihood between 30% and 70%) and some variance between evaluators. And here is where the diagnostic capabilities of such a model become important. Some examples: .
In the debriefing meeting, evaluators look for areas of disagreement in the input variables. For example, in the most current version of the software, a list of input variables featuring high variances or disagreements among evaluators is dis-
SELECTING WINNING NEW PRODUCT
played. These high variance items become the topics of the meeting agenda. The meeting focuses on questions such as: why was there so much disagreement amongst us on item number X or why did evaluator A rate item Y so high, and evaluator B rate that item so low? Usually the reasons for the disagreement are readily identified. For example, in one evaluation of an industrial chemical, two of the evaluators assumed a positioning strategy and niche target market that avoided a head-on confrontation with major competitors; the other three evaluators rated the project assuming a nose-to-nose positioning strategy with competitors. The input ratings and resulting prognoses were quite different between these two groups of evaluators. But the reasons for these differences became quickly apparent, and a discussion on positioning strategy ensued. The product was eventually launched, targeted at the niche segment, and as predicted, became a success. In other cases, discussion of the inputs that feature high variances among evaluators reveals that one or more evaluators has had first-hand knowledge that he or she then shares with the group. So the debriefing session becomes a knowledge-sharing forum, with the group moving towards a common knowledge base. This sharing of experience is particularly fruitful when the group is multidisciplinary. l
The profile of the project on each of the eight key factors helps to identify and prioritize the pros and cons of the project. The relative impactspositive or negative-of these factors provide key insights into needed action. Three typical questions on a negative evaluation, for example, include the following:
Why did the project come out relatively negatively? What were the “killer factors”? And most important, can we do anything to improve these negative factors?
A Planning Tool The outcome of this impact analysis is very often a study or task to correct a highly negative feature of the project, or to confirm a critical positive feature. For example, in an evaluation of new building material, the product fared moderately positive on most of the eight screening factors. Only on Product Superi-
J PROD INNOV MANAG 1985;2:34-44
43
ority was the product rated poorly. And this one factor proved to be a damaging one to the overall assessment of the project. Discussions ensued and the R&D project manager confessed that it had been his goal to develop a product that would simply equal a competitor’s product. The outcome of the debriefing meeting was a recognition that product superiority was essential to the product’s success. And two studies were immediately commissioned to achieve this: an end-user interview study to identify weaknesses in the competitive product as perceived by potential customers; and in-house creativity sessions that focused on ways that the proposed new product could be significantly improved. Thus the output of a well-conceived screening model is not only a GO/KILL evaluation, but most importantly, an indication of what needs to be done next in the event of a GO or tentative project. A final benefit of a systematic screening procedure is the payoffs from involving evaluators from different functional areas within the firm. In one company, typical new product projects had been screened and reviewed by an R&D committee. Not surprisingly, there had been many complaints from the operating divisions about the types of new products that eventually found their way out the lab and into the divisions. The introduction of the NewProd system forced the involvement of people from the operating divisions in the screening decision, but in a relatively painless and efficient manner. Not only did better screening decisions result from these multidisciplinary inputs, but a thorny political problem was solved.
Learning from Experience
New product success can never be guaranteed. Indeed new product development is perhaps the riskiest endeavor of the modem corporation. But new product development must continue if the corporation is to prosper. Project selection is pivotal to effective risk reduction in new development. A scoring model, such as NewProd, can be a valuable tool in screening new product proposals. Moreover the diagnostics of such an evaluation help guide the project by identifying key questions, issues, and tasks to be undertaken. We can always learn from our past triumphs and disasters. An empirically derived screening model,
44
R.G. COOPER
JPRODINNOVMANAG 1985;2:34-44
based on these past experiences, is simply a convenient way of translating these experiences into a management decision tool, which promises to yield better results than traditional approaches.
References 1. Albala, America. Stage approach for the evaluation and selection of R&D projects. IEEE Transactions on Engineering Management EM-22:153-162 (Nov 1975).
11. Cooper, Robert G. The impact of new product strategies. Marketing Management 121243-256 (1984). 12. Crawford, C. Merle. New product failure rates-facts Research Management 9-13 (Sept 1979).
Zndustrial
and fallacies.
13. Gear, A. E., Lockett, A. C., and Pearson, A. W. Analysis of some portfolio selection models for R&D. IEEE Transactions on Engineering Management EM-1866-76 (May 1971). 14. Glotskey, G. R. Research on a research department: an analysis of economic decisions on projects. IEEE Transactions on Engineering Management EM-7:166-172 (Dee 1960). 15. Hopkins, David S. New Product Board Report #773, 1980.
Winners and Losers.
Conference
2. Baker, N. R. R&D project selection models: an assessment. IEEE Transactions on Engineering Management EM-2 1: 165% 17 1 (Nov 1974).
16. McGuire, E. Patrick. Evaluating New-Product Conference Board Report No. 604, 1973.
3. Baker, N. R. and Freeland, James. Recent advances in R&D benefit measurement and project selection methods. Management Science 21:11&1-1175 (1975).
17. Moore, John R. Jr. and Baker, Norman R. Computational analysis of scoring models for R&D project selection. Management Science 16:B212-232 (Dee 1969).
4. Baker, N. R. and Pound, W. R&D project selection: where we stand. IEEE Transactions on Engineering Management EM- 11: 124- 134 (1964).
18. O’Meara, J. T. Jr. Selecting profitable Review 83-89 (Jan-Feb 1961).
5. Booz, Allen and Hamilton. New Product Managementfor New York: Booz, Allen and Hamilton, 1982.
the 1980’s.
6. Calantone, Roger and Cooper, Robert G. New product scenarios: prospects for success. Journal of Marketing 45 (Spring 1981).
Proposals.
products.
New York:
Harvard Business
19. Rubenstein, A. H. Research project selection (an industrial researcher’s view). Research Management 71225-233 (May 1964). 20. Simon, L. S. and Freimar, M. Analytical Marketing. court, Brace & World, Inc., 1970.
New York: Har-
7. Charles, A. and Stedry, A. C. Chance constrained model for real-time control in research and development management. Management Science 12:B353-62 (April 1966).
21. Schocker, A. D., Gensch, D., and Simon, L. S. Toward the improvement of new product search and screening. In: 1969 Fall Conference Proceedings, Amer. Marketing Assoc., P. R. McDonald (ed.). 1969, pp. 168-175.
8. Cooper, Robert G. Identify industrial new product success: project NewProd. Industrial Marketing Management 8: 124- 135 (1979).
22. Souder, W. E. A system for using R&D project evaluation Research Management 21129-37 (Sept 78).
9. Cooper, Robert G. The dimensions of industrial new product success and failure. Journal of Marketing 43 (Summer 1979).
23. Souder, W. E. A scoring methodology for assessing the suitability of management science models. Chemical Engineering 123- 126 (Ott 28, 1963).
10. Cooper, Robert G. An empirically selection model. IEEE Transactions EM-28:54-61 (Aug 1981).
derived new product project on Engineering Management
methods.
J
PROD
INNOV
MANAG
1985;2:34-44
oc>oo
Selecting Winning New Product Projects: Using the NewProd System Robert G. Cooper
Separating probable winners from probable losers is the goal of the new product screening task, and Professor Robert G. Cooper has developed a model that does this with remarkable success. In this article, he reviews various approaches to new product screening and then presents the basics of the NewProd model. NewProd now has a history of use in industry that seems to be fulfilling its original research promise. Professor Cooper shows how managers can build their own screening models and outlines how such models can contribute in an important way to better new product selection decisions. Over the years, Professor Cooper has conducted a series of major research projects that have aimed at improvements in the new product process. Their hallmark has been managerial relevance and a sound theoretical foundation. This article, the third that Professor Cooper has published in JPIM, is in the same tradition.
Address correspondence to Professor Robert G. Cooper, Faculty of Business, McMaster University, Hamilton, Ontario, Canada L8S 4M4. This work was supported by a grant from the Office of Industrial Innovation, DRIE, Canadian Federal Government. 0
1985 Elsewer
52 Vanderbilt
Science
Ave.,
Publishing
New York,
NY
Co.,
Inc.
10017
New Products: A High-Risk Endeavor
New products, although central to many firms’ successes, are plagued by difficulties and uncertainties. Hopkins reports that for every 100 industrial new products launched, about 40 fail in the marketplace [ 151. Crawford estimates the failure rate to be about 35% [12]. Similarly, our research shows that for every 100 products that are fully developed, only 60 become commercial successes [ 111. The attrition rate of new product projects is equally disturbing: Booz Allen and Hamilton report that for every seven concepts that enter the new product process, only one becomes a commercial success; further, 46% of the resources that U.S. industry devotes to new products are spent on products that fail or are cancelled. New product project selection becomes a pivotal task in the desire to maximize returns from a firm’s new product program. There are far more new product ideas or projects conceived than resources to commercialize them [lo]. Moreover the great majority of these projects are unfit for commercialization. In an ideal new product process, management would be able to identify the probable new product winners in advance, and be able to allocate the firm’s development resources to these projects. As a result, failure rates would be low, misallocated resources would be kept to a minimum, and the return would be maximized. The idea screening stage is the first of these project selection stages in the new product process [ 10,161. It is the decision point at which more new projects are 0737-67X2/85/$3.30
SELECTING
WINNING NEW PRODUCT
J PROD INNOV MANAG
35
1985;2:34-44
BIOGRAPHICAL SKETCH Robert G. Cooper is Professor of Marketing at the Faculty of Business, McMaster University, Hamilton, Ontario, Canada, and also Director of Research of the federal government’s Canadian Industrial Innovation Centre, Waterloo, Ontario. He holds bachelor’s and master’s degrees in Chemical Engineering, and an M.B.A. and a Ph.D. in Business Administration. Dr. Cooper is a noted researcher, consultant, and lecturer in the field of industrial product innovation, and has published over two dozen articles and books on the topic. Many readers will be familiar with his widely acclaimed research study published in his book, Project Newprod: What Makes a New Product a Winner?, Montreal: Quebec Industrial Innovation Centre, 1980, from which part of this article is adapted.
‘killed” than at any subsequent stage. Moreover, it is a critical decision in the sense that, at this point, resources are first allocated to a project: too weak a screening procedure fails to weed out the obvious “losers” or “misfits,” with the resulting misallocation of scarce resources and the start of a creeping commitment to the wrong projects. In contrast, too “rigid” a screen results in many viable and worthwhile projects being rejected, and perhaps is even more costly to the firm in terms of lost opportunities.
New Product Screening: Current Approaches
A myriad of approaches and schemes have been proposed to aid in the initial or idea screening stage of the AUTHOR’S NOTE: The NewProd model presented in this article traces its roots back to earlier research undertaken by the author [6,8,9]. The idea for the model came about almost as an afterthought from research we undertook into why new products succeed or fail. The model, in conceptual form, was first presented in a theory article [lo]. Even though the model was not operational at that time, it attracted considerable attention, so much so that it was decided to operationalize the model, and computer software was written. Since that time, several institutes, including the Wisconsin Center for Product Exploration (University of Wisconsin, Engineering School) and the Industrial Innovation Centre-Montreal, have made NewProd available to local commercial users. To date, a number of major corporations have had experience with the NewProd model (or ProCon, as it is called at Wisconsin), and because of their interest and positive experiences, we felt the time appropriate to share some of these experiences with others, and hence this article. For more information on NewProd, contact the author.
new product process. But the selection or design of an appropriate screening tool must be made in the light of the following considerations.
A tentative commitment in a sequential process. The screening decision is only the first GO/KILL decision in a sequence of such decisions for the new product project under consideration [ 11. A GO decision is not an irreversible decision, nor is it a decision to commit all the resources needed for the entire project. Rejection and acceptance errors. The screening decision cannot be a perfect one, but must strive for a balance between errors of acceptance and rejection. It should not be overly conservative and rigorous, accepting only the “sure bets”; nor should it lead to the dissipation of resources over a large number of unwarranted projects. Uncertainty of information and absence offinancial data. The screening decision amounts to an investment decision, but one made in the absence of concrete financial data [lo]. The richest and most accurate data are typically found at the end of product development, or better yet, during commercialization [ 11. At the initial screening stage, however, financial data are often missing altogether, and where available, are probably very uncertain [22]. Even qualitative data are often subjective and fairly unreliable. Multiple objectives and evaluation criteria. The criteria used in the screening decision should reflect the corporation’s overall objectives, and in particular, its goals for its new product program. Not all of these criteria are quantifiable, nor are they necessarily internally consistent. Also, both the data available and the criteria are time variant, i.e., change or become better defined as the project progresses [2,7,14,19]. Realism and ease of use. The screening decision method must be amenable to implementation. The decision tool must be realistic, i.e., not require so many simplifying assumptions so as to invalidate its results. And the tool must be easy to use-data requirements, computational effort, and interpretation of results must be consistent with managers’ capabilities.
J PROD INNOV MANAG 1985;2:34-44
36
No one screening method available today meets all the requirements for an ideal screening tool as outlined above. Instead, a variety of methods have been developed, and each incorporates some of the desired features. The four main approaches to initial screening include [2,5,17]: 1. 2. 3. 4.
Benefit measurement models. Economic models. Portfolio selection models. Market research approaches.
Benefit measurement models require a well-informed respondent or group to provide subjective inputs regarding characteristics of the project under consideration [3]. Such models are systemmatic procedures for soliciting and integrating benefit data. They typically avoid conventional economic inputs, but rely more on subjective assessment of proxies for new product success and payoffs, and fit with corporate objectives. Included in this category are checklists, and their extension, scoring models. In the latter, ratings of a project’s attributes are sought and combined in a weighted fashion to yield a numerical project score. Economic models treat the idea screening decision much like a conventional investment decision. Computational approaches, such as payback period, breakeven analysis, return-on-investment, and discounted cash flow methods, are used. To accommodate the uncertainty of data, techniques of probability, including Monte Carlo simulation, risk analysis, and decision tree analysis, have been proposed. But at the idea screening stage, economic models suffer because they require considerable financial data as inputs when often relatively little is known about the project. Thus, such models are usually considered more relevant for “known” projects (line extensions, product modifications, etc.), or at later stages of the new product process. Portfolio selection models view the screening decision as part of the total resource allocation problem. Such techniques largely involve operations research constrained optimization methods, such as linear, integer, and dynamic programming [ 131. The objective is to develop a portfolio of new and existing projects in order to maximize an objective function, yet subject it to a set of resource constraints. Because these mathematical models require substantial data inputs, including financial data on all projects, timing information, resource needs, and availabilities, they are rarely used
PI.
Market research approaches are usually limited to
R. G. COOPER
relatively simple consumer products, such as packaged goods. Such techniques assume that the critical criteria for the GO/KILL decision solely involve the marketplace; technological and production issues are either obvious or simple to solve. Given a market-based screening decision, it makes sense to use a variety of market research techniques, ranging from consumer panels and focus groups to perceptual and preference mapping, to screen product ideas. Of all the dozens of screening models proposed, benefit measurement approaches are generally recommended for new product idea screening. Because only a tentative commitment is required, and since available information on the project is limited, benefit models become the logical screening tool [23]. For example, the Conference Board reports that about half the firms they studied have set forth writen guidelines or rules for project selection [ 161. Usually these are in the form of checklists or scoring models. Souder investigated 26 project selection models and found that managers rated scoring models best in terms of ease of use and cost to implement [23]. Souder concludes that scoring models are “highly suitable for preliminary screening decisions where only gross distinctions are required among projects. ”
Scoring Models: Their Shortcomings
In spite of their popularity, scoring models are plagued by difficulties [20]. Such models rely on the subjective ratings of managers, and hence data input may not be very reliable. However, at the screening stage, management opinion is often the only data available. Moreover, ratings from several evaluators together with confidence scores can be combined to yield a composite and more reliable value for each input variable. The premise here is that the “average” decision maker is near optimal; unfortunately, none of us is ‘ ‘average. ’ ’ Other criticisms tend to be of a technical nature. Often, scoring models are seen as oversimplifications, since they attempt to reduce a complicated decision situation to a product score [20]. A major deficiency is the arbitrariness of items or checklist questions used, and that importance weightings assigned to each criterion are also arbitrarily determined. The selection of these questions and weights are no doubt based on the judgment and past experience of the model developer.
J PROD INNOV MANAG 1985;2:34-44
SELECTING WINNING NEW PRODUCT
However, his or her experience may be limited to a handful of projects, while his/her ability to judge how important each item is and to translate these into numerical weights, may be limited. Freimar and Simon propose the use of linear discriminant analysis of a large number of past successes and failures in order to identify the weights to attach to each screening variable [20]. Another weakness is the fact that many of the variables or items are not independent of each other. For example, if one of the screening items is “compatibility with distribution channels,” then it certainly is not independent from the “compatibility with current products” measure. (Both items are taken from O’Meara’s model [ 181). The result is that certain items are double counted. Schocker, Gensch, and Simon note that factor analysis of the many screening variables to a subset of independent factors or dimensions could be used to eliminate the interdependence of ratings [21].
The Advantages In spite of these criticisms, the scoring model is perhaps the best idea screening tool available. Proponents argue that such a model has utility for a number of reasons: it helps make a highly judgmental decision somewhat more objective; it systematizes the review of projects; it forces managers to subject each project to a consistent and large set of review criteria; it focuses attention on the most relevant issues; it requires management to state goals and objectives clearly; it is easy to understand and use; and it is generally applicable.
The NewProd Model
The NewProd screening model is essentially a scoring model, but with several important differences. Like a scoring model, it is based on the premise that a project’s desirability, attractiveness, or eventual success can be predicted by examining the profile of the project. That is, there exists a “winning” or desirable profile that is fairly predictive of product outcomes. As in most scoring models, NewProd evaluators rate the project under consideration on a large number of criteria; zero-to-ten rating scales are used. These evaluators’ inputs are then combined mathematically to yield a product score, and in the case of NewProd, a likelihood of product success.
37
The difference is that NewProd was derived from a large number of past new product successes and failures. In short, the questions or items used and the weights attached to these items are based on a statistical analysis of almost 200 projects from 100 companies. And the model has been validated, yielding a predictive ability-ability to predict accurately a product success or failure-of approximately 84%.
The Development
of NewProd
Here’s how the NewProd model was derived. A total of 195 industrial new product cases from 102 firms was identified. Half of these were commercial successes, i.e., met or exceeded the acceptable financial return for this type of investment. The other half had been launched, but were subsequently rated as commercial failures. For each project, managers were asked to rate the project on each of 80 characteristics. Zero-to-ten rating scales were used. Of the 80 characteristics, a total of 48 were judged to be potentially useful screening criteria, since they would be known at the outset of the project. The 48 characteristics described such features of the project as: its marketplace; for example, market growth, level of competition, etc. the product advantage; for example, uniqueness, quality, superiority, etc.
size,
product
the project-company fit in a number of key areas ranging from distribution fit to technology resource compatibility. the newness of the project to the firm; for example, market newness, technological newness, etc. and the project itself; for example, its magnitude, complexity, determinateness, etc. Next, these 48 characteristics were reduced to 13 underlying factors or dimensions, which were independent of each other, yet captured the original 48 variables. Factor analysis was used, as suggested by Schocker, Gensch, and Simon [21]. These 13 factors are essentially composite variables comprised of the original 48 variables, each variable with its own weight or loading on each factor.
38
1PROD INNOV
MANAG
R. G. COOPER
1985;2:34-44
Next, an equation was developed between degree of commercial success (or failure) and these 13 underlying dimensions. l Of the 13 dimensions, eight were linked to product outcomes in a significant way. Both multiple regression and linear discriminant analysis were used, and yielded essentially the same results. The regression results were marginally better and were selected for use in the model. The eight key dimensions, their weights or coefficients in the success equation, and typical questions or items that load on each of the dimensions are shown in Table 1. The resulting model was validated using a crosssplit half method. The original sample was randomly split into two halves. One half was used to generate a model; the other half of the sample was used to test the model, i.e., see whether the model correctly predicted a new product success or failure. The procedure was repeated, but this time the halves of the sample were switched. Virtually identical models were obtained from both halves; therefore, one is justified in combining the sample halves and deriving a combined model (Table 1). Predictive abilities were excellent; the model accurately predicted successes 82.6% of the time and failures 86.0% of the time, for an overall predictive ability of 84%.
Product Advantage
Screening Criteria
Product Economic Advantage for the user-the second marketability factor-involves a lower price-for-value product and customer cost reductions. One other factor-namely product innovativeness-is not a dimension closely linked to success. Note that product innovativeness was a dimension identified in the factor analysis, but was not found to be tied, either positively or negatively, to project outcomes. Projects strong on this dimension involve highly innovative, new-to-market products, the first into the market with only a potential demand, and unique task products. But in spite of their innovativeness, such products fared no worse or better than less innovative products.
Market Criteria A total of eight factors or underlying dimensions were found to impact on new product outcomes (Table 1). Two factors describe the market opportunity or market attractiveness. The first of these captures the magnitude of the market opportunity: being in a high growth, large, and high need market. The second is entering a market where competitive intensity is low: few competitors, little price competition, few new products, and relatively static user needs. This latter factor gauges the ease of (or lack of resistance to) exploiting the market opportunity. Note that a third market descriptor uncovered in the factor analysis-the existence of a dominant competitor with a loyal customer base-is not decisive in determining product success or failure. ’ “Degree of commercial success” was defined from a financial standpoint: the degree to which the product’s profitability exceeded (or fell short of) the acceptable profitability level for this type. of investment. A “minus five” to “plus five” scale was used, where +5 = far exceeded the acceptable profitability and -5 = fell far short of the acceptable level.
The product’s marketability is also important to success. For marketability, it is critical to achieve a differential advantage either via the product’s design, features, and quality, or by virtue of the product’s economics to the end user. Product Superiority is the most important factor in the success equation, having a new product which: is superior to competitors’ meeting customer needs;
products in terms of
has unique features for the user, not available on competitive products; is higher quality than competitors’ products, e.g., more reliable, lasts longer, tighter specs, etc.; permits the customer to do a task he/she could not do with what was previously on the market; reduces the customers’
costs; and
is a highly innovative kind on the market.
product-
the first of its
Synergy Criteria Project-company fit dimensions were also important to product outcomes, Overall Project-Company Resource Compatibility was the more important, and included synergies in the areas of managerial skills, market research talents, sales force and distribution resources,
SELECTING
Table 1. NewProd
Screening
product superiority,
overall project/co.
quality,
resource
market need, growth,
advantage
F value
and uniqueness
1.744
68.7
product product product product product product
compatibility
1.138
30.0
a good “fit” between needs of project and company resource base in terms of: managerial skills marketing research skills salesforceidistribution resources advertising/promo resources financial resources engineering skills R&D resources production resources
0.801
12.5
high need level by customers large market ($ volume) fast growing market
0.722
10.2
product reduces customers’ costs product is priced lower than competing
(factor name)
and size
of product to end user
newness to the firm (negative)
technological
resource
market competitiveness
product
constant
scope
Model: Key Factors and Weights” Regression coefficient (weight of factor)
Key factors or dimensions
economic
J PROD INNOV MANAG 1985;2:34-44
WINNING NEW PRODUCT
compatibility
(negative)
-0.354
0.342
-0.301
0.225
0.328
“R2 = 0.420. Adjusted R* = 0.395. F(8,186) = 16.83. Std error = 2.73
Variables
or items loading on factor
is superior to competing products has unique features for user is higher quality than competitors does unique task for user reduces customers’ costs is innovative-first of its kind
for product
class
products
2.9
project new new new new new new new
takes the firm into new areas for firm such as: product class to company salesforceidistribution types of users’ needs served customers to company competitors to company product technology to firm production process to firm
2.5
a good “fit” between needs of project and company resource base in terms of: R&D resources & skills engineering skills & resources
2.0
intense price competition in market highly competitive market many competitors many new product intros into market changing user needs
0.9
a market-derived new product idea not a custom product, i.e., more mass appeal a mass market for product (as opposed to one or a few customers)
39
40
R. G. COOPER
J PROD INNOV MANAG 1985:2:34-44
advertising and promotion resources, financial resources, R&D and engineering talents, and production capabilities. Newness to the Firm is a negative factor for project outcomes. Projects involving a new product class to the company, new distribution and salesforce approaches, new types of customer needs, new advertising and promotion methods, a new clientele, new competitors, new product technology, or a new production process for the firm fare more poorly. The final important project-company fit dimension is Technological Resource Compatibility, a positive factor. Technologically compatible projects are those that fit well into the firm’s existing base in terms of R&D resources and skills, and engineering resources and skills. But note that production and technological newness (also uncovered in the factor analysis)-projects involving new production processes and technologies to the firm-has no impact on product outcomes.
Other Criteria A group of factors were identified that we later called ‘ ‘project descriptors. ’ ’ Most of these, including Product Technical Complexity and Product Determinateness (how clear the product specification and technical solution are at the outset of the project), have no link to product outcomes. Only one entered the success equation, but in a fairly weak manner, namely Product Scope: custom products aimed at one or a few customers fare more poorly than do mass-market, broad-appeal, more standardized new products.
Using the NewProd Approach
The NewProd model has been employed in a number of companies. Initial use has typically been to validate or to check out the use of the model and to determine its predictive ability. For example, in one firm, eight new product projects were scored by up to ten evaluators. All eight products were eventually launched, and their market results compared to the NewProd predictions. In all eight cases, the model accurately predicted the product’s outcome: strong success; marginal success; marginal failure; or strong failure. (Note that NewProd predicted a strong failure in one of the
cases, but commitment to the project was so strong that it was launched in spite of the negative prognosis; and the outcome was as predicted, i.e., a strong failure .)
Developing
Your Own Model
The methodology that we employed to derive the NewProd model can also be used by organizations in order to develop their own screening models. The basic steps are: 1. Develop a reasonable set of screening items or questions. The items we used in NewProd are probably a good place to begin. 2. Identify a sample of past new product successes and failures within the corporation. 3 Request one or more evaluators to rate each of these past projects on each of the criteria developed in item 1. 4. Using appropriate statistical techniques, e.g., factor analysis, multiple regression, and/or linear discriminant analysis, derive a subset of key underlying dimensions and a success equation. 5. Validate the model. Use either a cross split half method (described above) or better yet, some new test cases (as we have subsequently done). 6. Develop the computer software to handle the evaluator’s inputs. (Both batch and user friendly interactive programs for an IBM-PC have been written for NewProd.) 7. Establish a procedure within the firm to facilitate the use of the model, beginning with a teaching session in a seminar format. A number of companies are presently developing their own versions of the NewProd model, using the above procedure. While NewProd was derived for industrial products, such approaches also have applicability for other types of products. A major packaged goods firm, an organization involved in the production of television shows, and a pharmaceutical company, are at the early stages of similar model developments using the NewProd methodology. The NewProd model, as presented here, has also been operationalized for public use (see author’s note in box inset). The procedure is as follows: 1. Up to ten evaluators are selected to review a proposed new product project. Some of these evaluators are closely involved in the project; others are
SELECTING WINNING NEW PRODUCT
41
J PROD INNOV MANAG 1985;2:34-44
Table 2. Factor Scores: A Profile of the Project
Factor name 1. 2. 3. 4. 5. 6. 7. 8.
2.
3.
4.
5.
Product superiority/quality Economic advtg to user Overall company-project fit Technological compatability Newness to the firm Market need, growth, & size Market competitiveness Product scope
Evaluator
1
1.17 -0.25 -0.05 -0.20 -0.36 0.88 -2.05 1 .oo
chosen because they are well informed on the topic, but not directly connected to the project. Most users of the model attempt to put together a multidisciplinary evaluation team for each project. For example, one chemical company, which has had considerable experience with NewProd, brings together evaluators from Sales, Marketing, Product Management, Engineering, R&D, Production, and Finance. A preliminary briefing session is held, where the proposed product and project is described to the evaluators. This meeting is descriptive and not evaluative in nature, i.e., evaluators are permitted to ask questions about the project, but are discouraged from offering opinions about its pros and cons. Each evaluator then independently rates the project on each of the 48 items or questions. Items are presented in statement form, and the evaluator indicates whether the statement describes the project or not (disagree/agree: 0 to 10 scales). In addition, for each item, the evaluator indicates how confident he or she was in providing the rating (again a 0 to 10 confidence score). The model administrator collects the evaluation forms and forwards them for NewProd data processing . Upon receipt of the computer output, a debriefing meeting is held. Here, the team of evaluators review each others’ inputs, flagging points of disagreement. The scores on each of the important eight underlying factors are discussed, noting the positive or negative impact of each on the projects’ merits. Finally the product score and likelihood of success for each evaluator and the “average evaluator” are reviewed.
The following product screening case best illustrates this final debriefing or results interpretation session.
Evaluator
2
Evaluator
1.11 -0.44 0.16 0.04 -0.15 1.36 -2.11 1.05
3
1.33 -0.98 -0.61 -0.55 -0.22 0.01 -1.03 0.60
Mean weighted evaluator
Std. Deviation
1.19 -0.49 -0.16 -0.19 -0.24 0.88 -1.82 0.90
0.10 0.31 0.32 0.24 0.09 0.56 0.50 0.20
Example
The example used is the evaluation of a new A.C. motor control. 2 In this case, three evaluators answered the 48 screening questions and indicated their confidences on each of the 48. The evaluators’ input data was then displayed, together with the scores of a ‘ ‘mean weighted evaluator, ’ ’ based on the inputs of the three people (table not shown). Next, the evaluators’ 48 ratings were reduced to the eight dimensions or factors to yield a profile of the project (Table 2). Here, factor scores are shown on a scale, such that zero is neutral, and high values are plus or minus one. So from Table 2, and reading down the “mean evaluator column” we see that the new product: is truly a superior product, with real benefits to the user (factor 1: mean score = 1.19) is entering a very uncompetitive mean score = - 1.82)
market (factor 7:
is entering an attractive market-growth, potential, size, etc. (factor 6: mean score = 0.88) and so on. The “pros” and “cons” of the project are identified in Table 3. Here the impact of each element in the product’s profile is shown. For example, the strongest positive feature of this project is the product superiority (factor l), while its weakest facet is the lack of an economic advantage to the end-user (factor 2). 2Disguised project and names.
R. G. COOPER
1PROD INNOV MANAG
42
1985;2:34-44
Table 3. Factor Scores and Impacts: Factor Impact Table
Factor name 1. 2. 3. 4. 5. 6. 7. 8.
Product superiority/quality Economic advtg to user Overall company-project fit Technological compatability Newness to the firm Market need, growth, & size Market competitiveness Product scope
Mean evaluator 1.19 -0.49 -0.16 -0.19 -0.24 0.88 - 1.82 0.90
Table 4. Overall Project Ratings: and Probabilities
Project Scores
Evaluator
Name
Score
Probability
R. Booth C. Clayton G. Franks
3.71 4.12 1.55
91.3 93.5 71.5
Mean wt’d eval
3.32
88.8
(%)
Impact strong positive negative marginal (negative) marginal (negative) marginal (positive) strong positive positive marginal (positive)
Probability of Success Finally, in Table 4, the product scores are computed (based on the regression equation of Table 1). These scores are usually in the -5 to +5 range, where + 5 means “a great financial success.” From these scores, the percent likelihood of success is computed, again by calibrating against our past cases-the almost 200 known successes and failures. From Table 4 we see that all evaluators rated the project positively; that the evaluators 1 and 2 were extremely optimistic on the project (success likelihoods of 91% and 93%); that evaluator 3 was more pessimistic, with a resulting success probability of 7 1%; and that overall, the likelihood of success was 89%-a very positive result. Also in Table 4, the pros and cons are again displayed, this time in order of impact on the project. Diagnosis In a debriefing meeting, involving all evaluators, discussion focused on diagnostic questions, such as: .
Why did evaluators 1 and 2 differ so much from evaluator 3? (All three evaluators were present in the meeting. )
.
What are the main “pros” of the project? What is the relative impact of each? And, are we reasonably confident about our assessments of these positive features?
.
What are the “cons” of the project? How damaging are they? And most important, can we do anything to rectify, or at least, diminish them?
The various tables provided by the computer (not all shown) help in this diagnostic stage.
model
1 2 3 MN
Pros of the project (in descending order)
Cons of the project (in descending order)
1. Product superiority/quality 6. Market need, growth, & size 7. Market competitiveness
2. Economic of user
advtg
Marginal factors (from positive to negative) 8. Product scope 5. Newness to the firm 4. Technology compatability 3. Overall companyproject fit
The Model in Use
Besides yielding a product score and likelihood of success, a scoring model, such as NewProd, provides important diagnostic outputs. Indeed, some users claim that the diagnostics are more valuable than the product score and success liklihood. Experience in using NewProd shows that only a handful of projects are clearly black and white situations. Few projects are rated very negatively, wherein the great majority of factors are negative and the liklihood of success is 30% or less. Very positive projects are also in a minority: projects where all important factors are positive, where the success likelihood is 70% or better, and where the variance among evaluators is small. More often than not, projects receive a mixed evaluation, i.e. , some positive features, some negative features, a resulting success likelihood between 30% and 70%) and some variance between evaluators. And here is where the diagnostic capabilities of such a model become important. Some examples: .
In the debriefing meeting, evaluators look for areas of disagreement in the input variables. For example, in the most current version of the software, a list of input variables featuring high variances or disagreements among evaluators is dis-
SELECTING WINNING NEW PRODUCT
played. These high variance items become the topics of the meeting agenda. The meeting focuses on questions such as: why was there so much disagreement amongst us on item number X or why did evaluator A rate item Y so high, and evaluator B rate that item so low? Usually the reasons for the disagreement are readily identified. For example, in one evaluation of an industrial chemical, two of the evaluators assumed a positioning strategy and niche target market that avoided a head-on confrontation with major competitors; the other three evaluators rated the project assuming a nose-to-nose positioning strategy with competitors. The input ratings and resulting prognoses were quite different between these two groups of evaluators. But the reasons for these differences became quickly apparent, and a discussion on positioning strategy ensued. The product was eventually launched, targeted at the niche segment, and as predicted, became a success. In other cases, discussion of the inputs that feature high variances among evaluators reveals that one or more evaluators has had first-hand knowledge that he or she then shares with the group. So the debriefing session becomes a knowledge-sharing forum, with the group moving towards a common knowledge base. This sharing of experience is particularly fruitful when the group is multidisciplinary. l
The profile of the project on each of the eight key factors helps to identify and prioritize the pros and cons of the project. The relative impactspositive or negative-of these factors provide key insights into needed action. Three typical questions on a negative evaluation, for example, include the following:
Why did the project come out relatively negatively? What were the “killer factors”? And most important, can we do anything to improve these negative factors?
A Planning Tool The outcome of this impact analysis is very often a study or task to correct a highly negative feature of the project, or to confirm a critical positive feature. For example, in an evaluation of new building material, the product fared moderately positive on most of the eight screening factors. Only on Product Superi-
J PROD INNOV MANAG 1985;2:34-44
43
ority was the product rated poorly. And this one factor proved to be a damaging one to the overall assessment of the project. Discussions ensued and the R&D project manager confessed that it had been his goal to develop a product that would simply equal a competitor’s product. The outcome of the debriefing meeting was a recognition that product superiority was essential to the product’s success. And two studies were immediately commissioned to achieve this: an end-user interview study to identify weaknesses in the competitive product as perceived by potential customers; and in-house creativity sessions that focused on ways that the proposed new product could be significantly improved. Thus the output of a well-conceived screening model is not only a GO/KILL evaluation, but most importantly, an indication of what needs to be done next in the event of a GO or tentative project. A final benefit of a systematic screening procedure is the payoffs from involving evaluators from different functional areas within the firm. In one company, typical new product projects had been screened and reviewed by an R&D committee. Not surprisingly, there had been many complaints from the operating divisions about the types of new products that eventually found their way out the lab and into the divisions. The introduction of the NewProd system forced the involvement of people from the operating divisions in the screening decision, but in a relatively painless and efficient manner. Not only did better screening decisions result from these multidisciplinary inputs, but a thorny political problem was solved.
Learning from Experience
New product success can never be guaranteed. Indeed new product development is perhaps the riskiest endeavor of the modem corporation. But new product development must continue if the corporation is to prosper. Project selection is pivotal to effective risk reduction in new development. A scoring model, such as NewProd, can be a valuable tool in screening new product proposals. Moreover the diagnostics of such an evaluation help guide the project by identifying key questions, issues, and tasks to be undertaken. We can always learn from our past triumphs and disasters. An empirically derived screening model,
44
R.G. COOPER
JPRODINNOVMANAG 1985;2:34-44
based on these past experiences, is simply a convenient way of translating these experiences into a management decision tool, which promises to yield better results than traditional approaches.
References 1. Albala, America. Stage approach for the evaluation and selection of R&D projects. IEEE Transactions on Engineering Management EM-22:153-162 (Nov 1975).
11. Cooper, Robert G. The impact of new product strategies. Marketing Management 121243-256 (1984). 12. Crawford, C. Merle. New product failure rates-facts Research Management 9-13 (Sept 1979).
Zndustrial
and fallacies.
13. Gear, A. E., Lockett, A. C., and Pearson, A. W. Analysis of some portfolio selection models for R&D. IEEE Transactions on Engineering Management EM-1866-76 (May 1971). 14. Glotskey, G. R. Research on a research department: an analysis of economic decisions on projects. IEEE Transactions on Engineering Management EM-7:166-172 (Dee 1960). 15. Hopkins, David S. New Product Board Report #773, 1980.
Winners and Losers.
Conference
2. Baker, N. R. R&D project selection models: an assessment. IEEE Transactions on Engineering Management EM-2 1: 165% 17 1 (Nov 1974).
16. McGuire, E. Patrick. Evaluating New-Product Conference Board Report No. 604, 1973.
3. Baker, N. R. and Freeland, James. Recent advances in R&D benefit measurement and project selection methods. Management Science 21:11&1-1175 (1975).
17. Moore, John R. Jr. and Baker, Norman R. Computational analysis of scoring models for R&D project selection. Management Science 16:B212-232 (Dee 1969).
4. Baker, N. R. and Pound, W. R&D project selection: where we stand. IEEE Transactions on Engineering Management EM- 11: 124- 134 (1964).
18. O’Meara, J. T. Jr. Selecting profitable Review 83-89 (Jan-Feb 1961).
5. Booz, Allen and Hamilton. New Product Managementfor New York: Booz, Allen and Hamilton, 1982.
the 1980’s.
6. Calantone, Roger and Cooper, Robert G. New product scenarios: prospects for success. Journal of Marketing 45 (Spring 1981).
Proposals.
products.
New York:
Harvard Business
19. Rubenstein, A. H. Research project selection (an industrial researcher’s view). Research Management 71225-233 (May 1964). 20. Simon, L. S. and Freimar, M. Analytical Marketing. court, Brace & World, Inc., 1970.
New York: Har-
7. Charles, A. and Stedry, A. C. Chance constrained model for real-time control in research and development management. Management Science 12:B353-62 (April 1966).
21. Schocker, A. D., Gensch, D., and Simon, L. S. Toward the improvement of new product search and screening. In: 1969 Fall Conference Proceedings, Amer. Marketing Assoc., P. R. McDonald (ed.). 1969, pp. 168-175.
8. Cooper, Robert G. Identify industrial new product success: project NewProd. Industrial Marketing Management 8: 124- 135 (1979).
22. Souder, W. E. A system for using R&D project evaluation Research Management 21129-37 (Sept 78).
9. Cooper, Robert G. The dimensions of industrial new product success and failure. Journal of Marketing 43 (Summer 1979).
23. Souder, W. E. A scoring methodology for assessing the suitability of management science models. Chemical Engineering 123- 126 (Ott 28, 1963).
10. Cooper, Robert G. An empirically selection model. IEEE Transactions EM-28:54-61 (Aug 1981).
derived new product project on Engineering Management
methods.