Venture planning and analysis for large-scale energy technology developments

TcchndogyInsocicy,

Vd3,pp.

0160-791X181/040387-21$02.00/0

387-407(1981)

Copyright D 1981 Pergamon Press Ltd

Printed in the USA. All tights reserved.

Venture Planning and Analysisfor Large-Scale Energy Technology Developments A Policy Perspective C. Lawrence Meador and Philip J. Pyburn

The Federal government and private industry must develop policies which guide and support both public and private organizations in the creation and use of major new energy technologies. Ideally, these policies should maximize the probability that adequate energy resources will be available to support future demands while keeping the costs and risks of development and implementation to a minimum. The policy analysis problem - broadly defined - is, therefore, the evaluation of the economics of programs which could provide energy for the future. Economics, as defined by Samuelson, is “the study of how men and society end up choosing, with or without the use of money, to employ scarce productive resources that could have alternative uses, to produce various commodities and distribute them for consumption, now or in the future, among various people and groups in society. It analyzes the costs and benefits of improving patterns of resource allocation.“‘The specification of the program and the evaluation of its economics (including financial considerations, as well as positive and negative economic externalities and incommensurables, such as pollution, quality of life, etc.) is what is meant by the term “venture analysis.” Multiple views and definitions of venture analysis have been proposed which may conflict with the definition of the term. The authors hope that proponents of other perspectives will bear C. Luwrence Meador, a lecturer in MIT’s School of Engineering, is a speciakt in computer-based information end decision support systems, and serves as president of Decision Support Technology, Inc., and director of Reseach & Planning, Inc. Dr. Meador has served as consultant to the US Departments of Energy and Defense and as as&ant director of the Centerfor Information Systems Reseamb at MIT’s Sloan Schoolof Management. He ir editor of Computer Communications and Communication e Informatica. Philip J. Pyburn is Assistant Professor of Management Information Systems andDirector of the Information Systems Reseanh Center at the Boston University Scbooi of tinagement. He ri the autbor of severalartzk~es that ded with theplanning and management of complex information systems, and has extensive consuhing experience in tbe management aspects of other technology-based systems. He received his muster’s degree from MIT’s Sloan School of Management and his doctorate from the HarvardBusiness School.

387

388

C. Luwrence Meador and Philip J. Pybum

with them in examining the particular characteristics that will be emphasized here. This paper develops the outline of a model for assessing research and development ventures for major new and improved energy technologies. As such, it draws upon work in several areas, including Policy Analysis, New Product Planning, Technology Assessment, and Technology Forecasting. The attempt here is not, however, to review in detail all the literature that might be relevant to the analysis of new energy technologies and the allocation of R&D resources to these technologies. Instead, an overall framework is proposed which captures the major areas to be considered in the evaluation of proposed energy research and development . Historically, the evaluation of proposed research and development has focused on the narrow perspective of the specific technology in question. Froomkin suggests that the narrow focus of most analyses leads to one of two possibilities.* Either the R&D proposal is rejected because of a sense that “something is missing,” or the proposal is accepted and the ultimate results bring with them unexpected side effects. He proposes four major difficulties with the present mode of public policy analysis (of which R&D resource allocation is a subset), including: 1) limited objectives; 2) absence of an overall model of the process to be investigated; 3) estimates which consider only specific “target groups” of users; and 4) absence of analysis of the preconditions to success. McGlauchin notes that the planning and evaluation of research and development is inherently difficult because feedback on performance is long delayed, and research plans can be rendered obsolete by events elsewhere.3 The sometimes counterintuitive complexities of planning and problems encountered in implementation of large-scale technology development efforts is further emphasized and supported in examples from space exploration and energy by Seamans and 0rdway.4 These problems notwithstanding, Baker and Freeland indicate the limitations of current venture analysis techniques, including:’ 1) inadequate treatment of risk and uncertainty; 2) inadequate treatment of multiple, often interrelated, criteria; 3) inadequate treatment of project interrelationships with respect to both value contribution and resource utilization; 4) no explicit recognition and incorporation of the experience and knowledge of the R&D manager; 5) the inability to recognize and treat nonmonetary aspects such as establishing and maintaining balance in the R&D program (e.g., balance between basic and applied work, between offensive and defensive activity, between product and process effort, between in-house and contracted projects, between improvement and breakthrough orientation, and between high risk/ high payoff and moderate or low risk/moderate payoff opportunities); 6) perceptions held by the R&D managers that the models are unnecessarily difficult to understand and use: and 7) inadequate treatment of the time-variant property of data and criteria and

Phzning and Analysis for Large-Scale Technology Developments

389

the associated problem of consistency in the research program and the research staff. The model suggested in Figure 1 represents an attempt to deal with each of the issues noted above by developing an evaluative framework for the special case of energy R&D. It is important to note that the model includes processes that are not strictly required for the analysis of an individual venture. However, only when venture analysis is considered in the overall context of a process for strategically selecting and controlling required resources can the specific analysis be useful. The remainder of this paper is organized around Figures l-2, 4-6 and 7-8; each of the facets of a venture analysis will be discussed in more detail, citing some of the relevant literature as appropriate. Other materials and references can be found in the bibliography.

Strategy The discussion of strategy creation in this section focuses on those issues that are critically relevant to venture analysis (see Figure 2). The framework developed here is extended from Cohen and Cyert, who suggest a nine-step strategy planning process:6 1) formulate goals; 2) analyze environment; 3) qualify values for goals; 4) develop objectives of subunits; 5) analyze gap between sum of subunit objectives and overall goals; 6) strategically search for alternatives; 7) specify portfolio; 8) implement strategy; and 9) monitor results. While it is certainly true that the determination of “what business the organiztion should be in” and the total value of resources provided to those businesses are the ultimate objectives of strategy planning, there are several other issues which are of specific relevance to the allocation of energy R&D. Of particular importance are the characteristics of the “portfolio” of opportunities required. It is important to specify in some degree of detail how the pieces of this portfolio interact, especially in terms of the contribution of each piece relative to its cost. Some of the concepts of financial portfolios apply to portfolios of R&D projects in the

FIGURE 1. S&ctioa,

Analysis and Impkmentation of Energy R&D.

C. Lawrence Meaa’or and PhiLipJ. Pybzm

390

IMPLEMENTATION

EXPERIENCE J I

STRATEGY 0

Kinds of innovation to be sought

0

Resources

0

Characteristics -

to be provided of project

value contribution

-

resource

-

overall

of each project

utilization level

“portfolio”

of each project

VENTURE SELECTION

of risk

l

Timing

of returns

0

Degree

of control required

0

Mi’x of direct technologies

and indirect

FIGURE

creation

2.

of

Strategy.

sense that risk can be reduced, if more than one project is considered. Beyond a certain point, however, the effect of adding additional independent projects to the portfolio is minimal. In the assessment of energy technologies it is critical that the timing of expected returns be detailed explicitly, Unlike a financial portfolio, the management of an energy R&D portfolio cannot as easily trade very large, long-run results for less effective results in the short run. For example, while fusion technologies may provide the greatest return for money invested when viewed over a very long time span, it is infeasible to structure the energy R&D portfolio with a total bias in this direction because of short-term requirements. New fossil fuel and other more conventional technologies may contribute little to the overall portfolio in the long term, but the likelihood of short-term payoff is so high that investments in these “suboptimal” strategies will probably have to be made. Unlike the development of a private R&D portfolio, however, the government must additionally determine how it will intervene in the energy R&D process. Abernathy and Chakravarthy suggest that this intervention can take two general forms:? 1) TecbnoLogy Push: Direct support of R&D to develop or modify technologies; and 2) Technofogy P&h Indirect efforts which motivate the private sector to create or modify technologies (e.g.,regulation, price, subsidies, tax relief, etc.). It seems likely that some mixture of both strategies will be required for the

Planning and Analysis for Lurge-Scaie Technology Developments

391

development of new and improved energy sources where the technology must not only be developed, but also commercialized. The appropriate mix of Technology Push/Technology Pull activities is probably also influenced by the degree of control the government is likely to have over the end product of the R&D. Abernathy and Chakravarthy’ suggest that this control increases as the process moves from basic research to prototype development and as the R&D focuses more on product development and less on information collection. Because a venture analysis in the energy systems area must consider not only the consumer market (utility, business, or individual), but also the supplier market, technology push strategies by the government can affect either. Figure 3 shows these relationships and details the several possible combinations of strategy options available. Table 1 summarizes some of the issues that are relevant for each strategy option mix of an energy venture analysis. In summary, the strategy portion of the overall process conditions to a large extent the venture analysis to come. The result is a “screen” that filters the plethora of possible actions and focuses on those that look most promising. Further,

TIKIINOI

I’uslI

OGY (Tl’) 4

SP-TP-MP

SUPPLY PUSH (SP)

MARKET PULL (MP)

MIXED SP-MP

0

PRIMARY MMED

STRATEGIES

(COMBINATION)

STRATEGIES

FTGURE3. Strategy Definitions.

C. Lawrence Meador and Phi@ J, Pyburta

392

TABLE1. Detailed Characteristics of Strategies Characteristic

Strategy

Technology

Push

Deploy extensive Research, Development, Testing, and Evaluation Solve all significant technical problems Develop production designs (detailed) Leave development of production capability to manufacturers Provide R&D incentive programs (by R&D organizations, but not manufacturers)

Supply Push

Develop manufacturing capabilities and facilities Initiate substantial purchases by government to develop manufacturing experience (“prime” pump) Leave Research, Development, Testing, and Evaluation and market research to manufacturers and purchasers Provide manufacturers with incentives to establish capability to produce energy systems in production quantities

Market Pull

Develop extensive market research (characteristics, response, requirements) and market stimulation Specify product performance requirements in detail Leave development of production capability and Research, Development, Testing, and Evaluation to manufacturers Provide purchasers with incentives to procure energy systems in quantity

Supply Push I Technology Push

Stimulate development of manufacturing capability Support major Research, Development, Testing, and Evaluation effort Solve major technical problems Develop one (or a small number of) production designs Provide modest initial purchases by government to develop manufacturing experience Leave development of detailed market requirements to purchasers Provide manufacturers and R&D organizations with incentives

Technology Push I Market Pull

Support major Research, Development, Testing, and Evaluation effort Support major market research effort Solve major technical problems Develop one production design Identify major product performance requirements and market characteristics Provide R&D organizations and purchasers with incentives

Supply Push I Market Pull

Stimulate development of manufacturing capability Support major market research effort Provide initial purchases by government to develop manufacturing experience Identify major product performance requirements and market characteristics Provide manufacturers and purchasers with incentives

Planning and Analysti for Large-Scale Technology Developments

393

strategy

Characteristic

Supply Push I Technology Push I Market Pull

DOE program commitments to reflect emphasis on each of the three primary strategies Development of manufacturing capabilities Research, Development, Testing, and Evaluation program Market research program Provide manufacturers, R&D organizations and purchasers with incentives

strategy provides the benchmark for review to ensure energy R&D is moving in the right direction.

that the balance

of all

Venture Andyszl~ “The wise allocation of resources implies knowledge of which actions will yield the highest net benefit.‘“’ The difficulty with venture analysis of energy technology proposals is that “net benefit” is extremely hard to measure and the determination of “highest” assumes that each proposal can be measured against a clear set of objectives. In the first part of this section the setting of objectives is discussed (see Figure 4). In the second section, project analysis, the process of gathering required data for “net benefit,” is covered, while the final section reviews the mechanisms and considerations for calculating “net benefit .” Objectives Campbell and Nichols argue that the major problem with the analysis of public policy is that “means and ends are seldom clearly distinguished.“‘O The process of

VENTURE

ANALYSIS

OBJECTIVES .

Determine whether the government should be involved in this R&D activity

VENTURE

.

Develop overall operational objectives for venture

SELECTION

.

Develop specific goals and expected outputs

.

PerIornl

“gap analysis” to determine required

technologies .

Set priorities

among objectives

FIGURE 4. Venture Analysis -

Objectives.

PROJECT ANALYSIS

394

C. Lawrence Meador and Pbi(ip J. Pybum

setting objectives for a specific venture focuses directly on the “ends” of the R&D. It is important to develop objectives that are both clear and responsive to the ultimate consumer. In the energy area, for example, objectives for a venture should focus on measures of energy and cost (e.g., ergs, watts, dollars, etc.) rather than on units of technology. The ultimate consumer is little concerned with “how” energy is developed, but he is most definitely concerned with whether it is available and cost-effectively delivered. Perhaps the most common mistake made in the analysis of new energy ventures is the setting of objectives that are narrowly defined in terms of a specific technology. As will be seen in the Evaluation Section, this makes analysis much more difficult. Having developed objectives that focus on tlse rather than on technology, McGlauchin suggests that technological goals be established and expected outputs be projected.” It is likely that this process will lead to several technologies that could conceivably meet the objectives of the R&D, so an iterative review which links technologies to objectives is recommended. Utilizing the judgment of the researcher, it should also be possible at this point to establish the technological developments required to meet the objectives. The final step in the objective setting process is usually ignored in venture analysis: the setting of priorities among various objectives. Developing a priority is a critical mechanism for focusing research on really important objectives, however, and it provides measures of relative “success” when objectives are only partially achieved. Campbell and Nicholls” discuss the use of multidimensional scaling as a device to set priorities on objectives that are often contradictory and measured in a variety of units. Prcject Analysis To effectively analyze an R&D project it is necessary to forecast the availability of the technologies that will be required to meet research objectives (see Figure 5). These forecasts “do not necessarily need to predict the precise form technology will take in a given application at some specific future date. Like other forecasts, the purpose is to help evaluate thepmbabifi~ and significance of various possible future developments.“‘3 For purposes of evaluation, however, it is important to determine whether, and with what probability, a new technology will be developed. Gerstenfeld suggests several techniques for assessing the potential for technology movement, including Delphi, trend fitting, PATTERN, timedependent comparisons, and PERT.14 Quinn offers additional techniques such as demand assessment, theoretical limits test, parameter analysis, systems analysis, scientific surveys, and analysis of competing technologies.” A state-space modeling approach for assessing dimensionally incompatible objectives and subobjectives in such projects has also been proposed. I6 Having determined the potential for developing the required technologies, it is necessary to assess the proposed technology of the venture. The breadth of this assessment is critical to the success of the venture analysis, because it determines

Planning and Analysis for Large-Scale Technology Developments

VENTURE

PROJECT

OBJECTIVES

ANAI.YSIS ANALYSIS

.

Porecast

.

Assess

.

Determine -

availability

of required

technologies

technology costs

economic

-

noneconomic

-

determine

how costs

are distributed

-

.

.

Determine -

economic

noneconomic

(societal,

-

social

-

environmental

-

political

Assess

EVALUATION

-

benefits

-

-

395

buyer,

risk of not reaching

technical

and supplier)

objectives

risk

-

competitive

-

sensitivity

risk of coats and benefits

.

Determine

rate of diffusion

.

Determine

timing of availability

to risk

of proposed

FIGURE 5. Venture Analysis -

technology

of technology

Project Analysis.

which subjects need to be considered and which do not. A seven-step process has been proposed to ensure an adequate assessment, including: 1) define assessment ; 2) describe relevant technologies; state-of-the-art assumptions (identify and describe critical 3) develop nontechnical factors influencing technologies); 4) identify impact areas (identify societal areas most impacted by application of technologies); 5) make preliminary impact analysis (trace and integrate process by which technology impacts society); 6) identify action options (develop alternate ways to utilize technologies); and impact analysis (determine how each option affects the 7) complete technological impact on society). I7 The assessment methodology proposed implicitly reflects the “impact” of project costs and benefits, because so much of the evaluation portion of the model is concerned with comparing costs and benefits. These factors are isolated below for further review. costs Costs are the negative impacts associated with a new technology. While the venture analysis is concerned with minimization of these consequences, it is impor-

396

C. Lawrence Meaa’or and Philip J. Pybum

tant to recognize that “cost” is, in some logical sense, in the eye of the beholder. Depending on the objectives for the technology, an item could be viewed either as a cost or a benefit (depending on the audience). FisherI suggests that the compilation of venture costs include three areas of negative impact: economic costs, noneconomic costs, and distribution costs. I9 Economic costs are the easiest to measure, because they are essentially the value of benefits given up by choosing a particular venture. Resources spent working on wind-energy conversion systems cannot be spent on fast-breeder reactor development. In a sense, economic costs are the inputs required to reach the venture objectives. Noneconomic cost, on the other hand, is related primarily to the negative results of a particular venture (often referred to as economic externalities). The fact that noneconomic costs are hard to evaluate and quantify (see Evaluation section) makes them no less relevant for a complete analysis. In fact, noneconomic energy system costs (e.g., environmental pollution, consumer safety, etc.) are likely to be critical decision criteria when a venture is evaluated. Distribution costs are those impacts of a venture which serve to redistribute wealth among groups. For example, subsidies to utilities for their use of a new energy technology may merely involve a transfer payment from one sector of the economy (general taxpayer) to another (utility stockholders). While such transfers have no direct overall economic impact, redistribution of this sort may well be considered an important decision criterion. In determining which costs are relevant, the “world with/world without” principle is useful. Only those costs incurred explicitly because of the venture need be considered. In other words, the cost associated with the “world without” the venture (including past, or sunk costs) need not be considered. Only the marginal costs of the venture in question should be discussed. It is also important to note that dollars are not often a good proxy for true cost, because of the assumption inherent in the use of dollars that all resources are substitutable. In the near term this is almost certainly never true, so it is advisable to use real resources whenever possible in the discussion of cost. When evaluating cost it is important to attempt where possible to match costs and benefits temporarily. Therefore, if benefits are expected to be stated in terms of commercialization of a venture, the costs of research, development, production, implementation, and use must be considered.

Benefits are the positive impacts of a new technology and, like costs, they are dependent on the objectives of the venture. If objectives have been properly set, they are measures of the degree to which the goals of the venture will be achieved. Unfortunately, benefits are also difficult to measure because positive impacts can be economic, noneconomic, or distributional. Diffenbach notes that three separate types of economic factors are relevant to the commercialization of new energy technologies: societal, utility, and supplier.” It is important, therefore, to consider at least these three levels of economic benefit. Noneconomic benefits should include social, political, and environmental impacts, as well as effects on income/ wealth distribution.

Planning and Analysis for Large-Scale Technology Developments

397

A major difficulty in the development of venture benefits is the problem of valuing the degree to which objectives are achieved. Quade suggests several approaches to the valuation of benefits, including:2’ 1) Equating venture costs with venture benefits. Although this is usually an absurd measure, it is surprisingly prevalent in policy analyses (i.e., educational quality measured by cost per pupil). 2) Equating venture work load with venture benefits. This measure is reasonably common, though inadequate, because it measures only the amount of work being done, not how effective the work is. 3) Reducing all venture benefits to a common measure, typically dollars. As suggested in the cost discussion, many imports are not economic but need to be considered nonetheless. 4) Valuation of benefits based on explicit statements by experts. 5) Valuation of benefits based on the value of past decisions. 6) Valuation of benefits based on the imputed economic value of the venture to society or some smaller group. 7) Valuation of benefits based on individual preferences for various outcomes, measured through the use of a lottery describing alternate results. These cost and benefit factors, when combined with the timing and diffusion assessments, begin to establish the likely commercial potential of new energy ventures . 22It should be recognized, however, that significant risk is inherent in most energy R&D. beyond the obviously required assessment of technical risk (i.e., the impossibility of meeting technological requirements within time and budget constraints), Diffenbach also notes that competitive risks directly affect the progress toward commercialization. These risks include the direct competition for research funding, intratechnology competition between institutions, and intertechnology competition, whereby one technology preempts another. Evaluation Clearly the most critical portion of the venture analysis is the evaluation of costs (what is given up if this venture is carried out) and benefits (how well the objective is achieved). It is here, moreover, that the most serious difficulties come to bear. These problems are the result of the highly imprecise nature of the terms “cost” and “benefit” and the difficulty in developing useful quantitative measures of R&D program economies (see Figure 6). Ideally, the venture analysis should develop a measure of rid-adjusted net benefit so that resources can be allocated wisely. In practice, however, this measure is most often a set of criteria (often conflicting) which in the aggregate “provide a statement about cost, effectiveness, risk, timing, etc., by which the alternatives can be ranked . . . In principle, the needed criterion is clear enough: the optimal system is the one that yields the greatest excess of positive impacts (attainment of objectives) over negative impacts.“23 Unfortunately, in practical problems, no such simple subtraction is acceptable. Quade suggests several broad analytical approaches to this evaluation problem, including Operations Research, Systems Analysis, Cost-Effectiveness, and Cost-

C. Lawrence Meador and Phi&) J. Pybum

398

VENTURE

ANALYSIS

EVALUATION .

Determine -

.

Determine strategic

PROJECT ANALYSIS

.

.

risk-adjusted

multicritcria

-

costs

-

benefits

-

timing

-

risk

of technology

project

effects

on characteristics

of

“portfolio” -

VENTURE ACCEPTANCE

Evaluate

government

-

direct

creation

-

indirect

Determine

net benefit

evaluation

intervention

strategies

for

project

creation funding

level

and timing

FIGURE 6. Venture Analysis -

Evaluation.

Benefit Analysis. Operations Research (OR) is essentially the application of scientific methods to the problems of management.‘* Schlesinger suggests that the term OR be confined to problems involving the efficient allocation of resources so that it is distinct from Systems Analysis, which is concerned with “optimal choice of objectives.“2’ In any case, both OR and Systems Analysis involve the application of rigorous, logical thinking to the evaluation problem. While OR tends to be more quantitative and computer-oriented, both approaches rely heavily on the development of “models” of the underlying economic, social, and political processes . Cost-effectiveness is a more specialized form of analysis where the resource costs of a project are compared to the effectiveness of the project in meeting stated objectives. As mentioned earlier, the measurement of effectiveness is almost always difficult and usually requires the use of proxy measures. According to Quade,26 the best approach to cost-effectiveness is to evaluate relative effectiveness for a faed level of cost, or to determine the minimum cost for a fued level of effectiveness. In this way, values of effectiveness and cost that are not measured in the same units can be compared. The major drawback with the cost-effectiveness approach is that - while it allows ranking of alternatives for the same goal - it cannot compare alternatives which seek different goals. This is currently of serious practical concern, because most energy R&D proposals utilize very different objectives (and thus measures of effectiveness). To overcome the problems with cost-effectiveness analysis, it is occasionally possible to measure costs and benefits in the same units. “In practice, this means expressing both benefits and costs in monetary units, dollars for example, a process that often must be done very arbitrarily and that leads to neglect of certain benefits and costs. “27 Ideally, all costs and benefits should be identified and converted to dollar equivalents, including side-effect costs and intangible benefits. The aim of cost-benefit analysis is to maximize “the present value of all benefits less that of all costs, subject to specific restraints2* but its major drawbacks are

Planning and Analysis for Large-Scale Technology Developments

399

that the approach is dangerously misleading when important benefits or costs are not adequately captured and quantified. The question of cost and benefit distribution throughout society is especially a problem in the evaluation of energy systems, where various ventures may tend to distribute benefits to utilities, business, and consumers in a highly disproportionate way. Regardless of the analytical approach taken, however, it is clear that some set of decision criteria must be assembled. While many criteria have been suggested as being useful evaluative tools, in general, they can be grouped into the following categories: economic valuation criteria; model output criteria; and nonquantitative criteria. The following discussion briefly surveys several of the important techniques.

Economic Vduation Criteria2” Net Present Value. The Net Present Value (NPV) Criteria is based upon the economic principle that money has “time value”; i.e., benefits and costs associated with future points in time should be valued the same as identical costs and benefits associated with the present. The NPV method uses a discount rate (d) to adjust the value of costs and benefits at different points in time before summing them together. The principal problems with the NPV criterion is the establishment of the discount rate (d), and the fact that NPV does not discriminate magnitudes of benefits and costs. 3o Znternaf Rate of Return. The Internal Rate of Return (IRR) is defined to be the discount rate (d in the NPV formula above) that equates an initial cost and the discounted sum of all future benefits. The essential problem with IRR is that a unique solution occurs when the discounted cumulative flow of funds changes sign only once. When multiple changes in sign occur (i.e., when significant net costs are incurred periodically through the project life), use of IRR can be misleading. Benefit-Cost Ratio. The benefit /cost ratio (B/C) is the ratio of the sum of the benefits (appropriately discounted by d) to the sum of the discounted costs. While this criteria is quite popular, it is not entirely satisfactory in the context of the venture analysis model which suggests explicit consideration of the “portfolio effects” of a specific project. This is because B/C calculates net benefit per dollar of cost. Therefore, smaller projects will often have a higher B/C than equivalent (or even superior) large projects. On the other hand, B/C in combination with NPV, can be useful for judgmentally ranking projects where there is a capital constraint. Minimum Average Unit Cost. While this criterion purports to address the optimum size of a project, by suggesting the scale that minimizes average cost, it is incomplete in its exclusive focus on costs.

400

C. Lawrence Meador and Phi’lipJ. &burn

Pay-Back Period Utilizing this criterion, projects are selected based upon the most rapid recovery of initial costs. The difficulty with this criterion for the assessment of alternative energy sources is the length of time required to recover initial research, development, and production costs (though actual return over the life of the energy system may be well in excess of costs). This criterion does not reflect any benefits achieved by the technology after the pay-back point. Other Crijeti. Other less useful criteria include Cut-off Period, Net Average Rate of Return, and Annual Value.“’ Quantitative Muhtittrihzte Modeiti2 Optimization Modeh. Optimization models include representations of real relationships that can be captured with formal mathematical constructs. These models use techniques such as linear (and nonlinear) programming, dynamic programming, queueing theory, network theory, or game theory. As criteria for evaluation, they require relatively rigid assumptions about the form of the relationships between costs and benefits which may not exist for a highly complex problem, such as energy R&D venture analysis. Simufations. Simulation models represent relationships numerically, but without necessarily using all of the formal analytical techniques of mathematical models. The “solution” to simulation is calculated by iterating through the simulation model on a case-by-case basis. One describes what one thinks are the appropriate relationships; then the model is used to “experiment” with various values to determine predicted interactions. Because of their relatively unstructured general form, simulations may be useful as criteria for the evaluation of relatively complex nonlinear systems. 33 Operationa! Games. Operational games (not to be confused with game theory) put individuals in positions where they “act out” the role of policy-makers. Because the individuals can make decisions based on “intuitive” as well as “scientific” grounds, this technique is likely to provide useful criteria for the evaluation of political costs and benefits. 34 A summary of literature discussing modeling techniques useful for evaluating R&D projects is shown in Table 2 for additional reference. Nonquantitative Cnjeria3’ Scenarios. Scenario generation involves the specification of costs and benefits in the future that result from a sequence of hypothetical events. Scenarios seem likely to provide insights into many of the nonquantitative aspects of energy venture analysis. For example, several performance and cost possibilities could be hypothesized for wind energy conversion systems, and then a scenario could be written describing the events that might lead consumers to put windmills on their roofs.

Souder ( 1972)4”. 4’ Linear Nonlinear Zero-one Scoring Profitability Utility index

Project scoring Project index Math programming Utility models Descriptive

(1970)‘”

Alboosta/Holzman

Decision theory Economic analysis Operations research

Scoring models Economic models Risk analysis Constrained optimization

(1967)”

Economic analysis Operations research

Decision theory

Techniques

Moore/Baker ( 1969)j8

CetronIMartinoIReopcke

Baker /Pound (1964)”

source

model

Scoring models used to evaluate representative models from each class Uses data from 30 actual projects to perform comparative analysis of four models designed to represent main categories

Brief discussion of each model type Identification of critical factors not included in most models

Brief discussion of each model type Summary of some accepted descriptive insights. First empirical data relating output from different forms

Features which desctibe input and output characteristics Ease of use: data Areas of applicability Table summary for 30 models

Discussion of general descriptive nature of project selection material Characteristics of 30 representative models. Brief description of areas of application and data tequitement5

TABLE 2. Summary of Modeling Techniques for R&D Evaluation

402

C. Lawrence Meador and Phihip J. Pybum

Obviously, the disadvantage of scenario writing as an evaluative bias that can be entered into the analysis by the analyst.

criterion

is the

Delphi. Delphi is an iterative technique for eliciting the opinions of experts, generally through the use of questionnaires. Typically, this technique calls for the anonymity of respondents and the iterative feedback of results to the entire group. In this way, the group begins to focus its analysis and - over several iterations - comes to a consensus. The result of a Delphi evaluation is particularly useful because it provides a statistical picture of not only the media response, but also a range of responses. 42The evaluation of costs and benefits is thus left to the opinions of experts with this technique, and it may therefore be useful in highly complex, nonquantifiable areas (e.g., the cost of pollution which kills noncommercial fish species). To complete the discussion of the evaluation portion of a venture analysis, it is necessary to deal with two issues: “portfolio effects” and uncertainty. The “portfolio effects” are essentially those characteristics which make a particular project fit well or poorly with all of the other energy projects in progress. In spite of adverse relationships between costs and benefits, for example, it might be necessary to accept development of an energy source because it fills a time gap before the availability of a superior technology. While factors such as this should obviously be considered as part of the formal analysis and evaluation of costs and benefits, they are mentioned here because it is important to keep in mind the concept that there is no one “best” system, only ventures that fit well with each other and the overall strategy. The evaluation of uncertainty is a more difficult issue to deal with, however. Quade suggests several techniques including:43 1) defer decision until better information is available; 2) obtain additional information; 3) build flexibility into the design as a hedge against uncertainty; 4) analyze the decision under “worst case” and “break-even” conditions; 5) use decision-theory (Bayesian) techniques44; and 6) perform sensitivity analysis to determine how sensitive the results are to variations in data or assumptions. (This analysis can selectively isolate key variables for further analysis or it can include complete enumeration of all possibilities.) As suggested at the outset of this section, the ideal evaluation would entail the generation of a single measure of risk-adjusted net benefit. The practice of venture evaluation, however, requires that a large dose of judgment be used when evaluating the costs (negative impacts) and benefits (positive impacts) of a project. Decision Support Systems It should be noted that decision situations requiring

a combination

of analysis and

Planning and Analysis for Large-Scale Technology Developments STRATEGY

403

MODIFICATION t

IMPLEMENTATION .

Develop project control tools

VENTURE

.

Periodic forecast of technology

ACCEPTANCE

.

Periodic assessment of technology

.

Evaluate implications of “breakthroughs” -

project technology

-

competttive technology

-

COMMERCIALIZATION

-

FIGURE 7. Implementation.

judgment raise the question of how the analytic process should be integrated into the larger decision process. This is indeed a very complex problem and involves the further question of what form of information system technology should be designed to support the analysis process. Decision Support Systems (DSS) are a class of computer-based information systems, which have evolved only recently and which are designed to address consequential, semi-structured, problem environments where no programmable decision rules are available for “solving” the decision problem in closed form. Such problems are found in environments which include policy analysis and policy formulation, as well as strategic and long-range planning, to name a few where the DSS approach has been usefully applied. In this approach, there is an emphasis who is dealing with a on improving the “effectiveness’ of a decision-maker, complex-problem domain, rather than improving the “efficiency” with which structured routine tasks are performed. This represents a dramatic departure (in both focus and techniques) from the traditional approach to computer-based applications in transaction processing and reporting environments such as inventory control, accounts receivable, sales analysis, or payroll. One particularly interesting application of the DSS concept in energy technology policy has been undertaken by a large international banking organitation, which provides funding for various types of electric utility construction to client organizations. In this DSS application critical questions of electric-energy demand and production gaps were examined along with alternative productionequipment technology mixes, using a very high-level computer modeling language called EXPRESS. It should be emphasized here that the analysis activities generated only a part of the information necessary to come to final decisions, since the problem-setting included important variables such as legal and regulatory issues, environmental implications, and political realities which had to be factored into the overall decision process. Readers interested in the DSS area should refer to Keen and Scott Morton, Blackman, Little, Urban, and Meador and Ness for different perspectives on the evolution of the field.

C. Lawrence Meador and Phi@ J. Pybum

404

COMMEIICLALIZATION

IMPLEMENTATION

-m

0

New Starts

0

Regulation

0

Training

l

Taxation

a

Monitoring

FIGURE8. Commercialization.

Swnmary Venture analysis of a major energy R&D program involves a process whose objective is to provide insights into the policy question: “Should the government be in this particular energy-technology R&D business ?” It is critical to recognize the broader policy formulation and analysis process within which venture analysis seeks to provide insight. Although several venture evaluation approaches have been suggested which incorporate muitiattribute decision criteria, they suffer from either: a) the weakness of collapsing the various dimensionally incompatible decision criteria (monetary, environmental, energy, etc.) into a single index or numeraire; or b) they fail to take decision criteria other than energy and dollars into account at all. Both approaches are inadequate for effective venture analysis. The “portfolio problem” of multiple competing/interacting energy-technology options at the policy-decision level suggests that venture analysis results for specific technologies should be provided with complete assessments of various relevant decision criteria in their original (and most appropriate) metrics. Costs and benefits measured in dollars, ergs, environmental damage units, plant safety units, and so on should not be collapsed into a single ranking or evaluation criterion at the venture analysis stage. To do so will totally muddle and confound the higher-level portfolio policy-decision process which compares the effects of multiple energy-technology developments and alternative wealth distribution and implementation approaches. It should be noted, moreover, that while this paper has focused on the specific policy-level issues of R&D venture analysis for energy technologies, it appears that this methodology may be appropriate for a variety of public-policy decisions. Certainly most of the notions presented could be usefully applied to other public sector R&D efforts. Additi.onally, however, many of the concepts apply to non-R&D policy decisions where the emphasis is on the selection of a unified, time-phased program from a range of possible alternatives4’ Notes and References 1. P. A. Samuelson,

Economics

2. J. Froomkin, “Needed: 1976.

(New York: McGraw-Hill,

A New Framework

1973). p, 3.

for Analysis of Government

Programs,”

Pohcy Andysir,

Spring

Planning and Analysis for Large-Scale Technology Developments 3. L. D. McGlauchin.

“Long Range Technical

4. N. Baker and J. Frccland, M%egemcnt

Harvard Business Review, July-August

Planning,”

“Rcccnt Advances in R&D Bcncfit Mcasurcmcnt

Science, June

405

1968.

and Project Selection Methods.”

1975.

5. Ibid. 6. K. J. Cohen and R. M. Cycrt, “Strategy: Blrrinerr. July

Formulation,

1973. See also W. H. Grubcr,

Management

Association,

7. W. J. Abernathy Framework,”

October

and Monitoring,”

The Jormaf

of Corporate

R&D,” American

Integration

of

1981.

and B. S. Chakravarthy,

Graduate

Implementation.

“The Strategic

“Government

School of Business Administration,

Intervention

and Innovation

Harvard

University Working

in Industry:

A Policy

Paper Series. 78-4.

8. Ibid 9. Campbell

and Nichols,

“Setting

Priorities Among

Policy Amu’y~is. Fall 1977.

Objectives,”

10. Ibid. op. cit.

11, McGlauchin, 12. Campbell

and Nichols,

13. J. B. Quinn.

“Technical

14. A. Gcrstcnfcld,

15. Quinn, op.

op. cit. 197 1, p. 1.

The Journaf of Business. January

Forecasting,”

1971.

cit.

16. C. L Mcador and A. C. Parthc, plications”

Harvrrrd Brcsinerr Review, March-April

Forecasting,”

‘Technological

in F. P. Davidson,

Engineering Revisited (Boulder, 17. The MITRE Corporation,

“Managing

Macro-Development:

C. L. Mcador and R. Salkcld, CO: Wcstvicw

Policy, Planning

Press, 1980).

Tecbnofogicaf AssEIIment Methodo(ogy (Springfield,

18. G. Fisher, “Cost Considerations

in Policy Analysis,”

19. Many authors class noneconomic

and Control System Im-

cds.. How Big and StU Beat&&f? Macro-

and distribution

VA: NTIS, 1971).

PO/icyAna/yris, Winter

1977.

costs as social costs.

Evaluation of New Energy Technoiogies: Factors Affecting Progreu Toward Commerci&e-

20. J. Diffcnbach, tion, Harvard

University.

unpublished

dissertation.

Analysis for Public Decirions (New York: American

21. E. S. Quadc.

Elscvicr Publishing

Company,

I975), pp.

107-114. 22. There is a large body of literature technologies

specific technological 23. Quadc,

on the subject of technological

diffuse in the commercial

environment

diffusion

geometrically,

and transfer.

In general,

new

but the actual rates arc dependent

on

factors not relevant to this general discussion.

op. cit.. pp. 91-92.

24. Ibid... pp. 22-27. 25. J. R. Schlesinger,

“Quantitative

26. Quadc,

op. cit., p. 25.

27. Quadc,

op. cit., p. 26.

Analysis and National

Security,”

World Por’itics, 1963, p. 298.

28. A. R. Prcst and R. Tuvcy. “Cost Benefit Analysis: A Survey,” The EconomicJournal, 29. R. P. Zimmcr

et a/. , Benefit-Cost Methodology Stndy, DOE/NASA

1965. 15, p. 683.

7827-75-l.

30. The reader should see Benefit-Cost Methodology Study (29). pp. 73-91, for an excellent discussion of the sctting of a “social opportunity”

discount

rate.

31. See Benefit-Cost Metbodo/ogy Study (29) for a complete 32. Quadc.

description

of these techniques.

op. cit., pp. 150-154. WorldDynamics (Cambridge.

33. See J. W. Forrester, of a large-scale

simulation

model whose solutions

Massachusetts:

Wright-Allen

Press, 1971) as an example

arc used to evaluate policy altcrnativcs.

34. M. Shubik and G. D. Brewer, MO&S, Simtdatz~ns and Games - A Survey (Santa Monica, CA: The Rand Corporation, 35. Quadc,

1972), R-1060

ARPAIRC.

op. cit., pp. 153-154.

36. N. R. Baker and W. Pound, ing Management, EM-11.

“Rand D Project Sclcction: December

37. M. J. Cctron, J. Martin0 and L. Rocpckc, titative Methods,”

“The Sclcction of R and D Program

IEEE TransactiOns on Engineering Management, EM-14,

38. J. R. Moore and N. R. Baker, “Computational Management Science. Vol. 16, Dcccmbcr at the 11th Institute 1972.

of Management

“Comparative

Analysis of Scoring

Content

-

Survey of Quan-

March 1967.

Models for R&D Project Selection,”

1969.

39. Chester A. Alboosta and Albert G. Holzman, 40. W. E. Sounder,

Where WC Stand,” IEEE Transactions on Engineer-

1964.

“Optimal

Science Meeting,

funding

of an R and D Project Portfolio,”

Los Angeles,

Analysis of R&D Investment

CA, October

Models.”

presented

1970.

AIIE Transactions, Vol. 4. March

C. Lawrence Meaa’or and Phi@ J. Pybum “A Scoring

41. W. E. Sounder,

Methodology

Management Science, Vol. 18. June

for Assessing the Suitability

The De/phi Method(Santa

42. N. C. Dalkey,

of Management

Monica, CA: The Rand Corporation),

43. Quade. op. cit.. pp. 218-220. 44. R. Schlaifer. Anaf’ysir of Decisions Under Uncertainty (New York: McGraw-Hill, 45. For example,

Mckenney et a/.

School Working

Science Models.”

1972.

, “An Analysis of the 1979 Planning and Budgeting

Paper 7949 (1979). and L. Gremillion,

RM-5888-PR. 1968). Process,” Harvard Business

J. L. Mckenney and P. J. Pyburn,

Pz&icAdmintitra-

tion Review (in press), discuss many of these same notions in the context of project and program another

selection in

public setting.

Bibliography Abernathy,

W. J.. and B. S. Chakravarthy,

Framework,” Alboosta,

Graduate

C. A., and A. G. Holzman,

stitute of Management Ansoff,

“Optimal

Science Meeting,

H. 1. and J, M. Stewart,

(November-December

Intervention

Funding

Los Angeles,

“Strategies

Harvard

and Innovation University,

Working

of an R&D Project Portfolio,” CA (October

for a Technology-Based

in Industry:

A Policy

Paper Series 78-4. presented

at 1 Ith In-

1970). Business,”

Harvard Bushers Review 45

1967), pp. 71-83.

Baker, N., and J. Freeland,

“Recent Advances

Management Science, June

in R&D Benefit Measurement

and Ptoject Selection

Methods,”

1975. “R and D Project Selection: Where We Stand,” IEEE Transactionr on Engineer-

Baker, N. R., and W. H. Pound,

ing Management, EM-11 (December Blackman,

“Government

School of Business Administration,

A. W., “New Venture

1964).

Planning:

The Role of Technological

Forecasting,”

Technological Forecasting

and Social Change 5 (1973), pp. 25-49. Cambell

and Nichols.

Cetron,

“Setting

Priorities Among

Objectives,”

Pohcy AnaLyk, Fall 1977.

M. J., J. Martin0 and L. Roepcke, “The Selection of R and D Program

Chambers,

J., et al,

August Cohen,

Content

- Survey of Quantitative

IEEE Transactions on Engineeting Management. EM-14 (March 1967).

Methods,”

“How to Choose the Right Forecasting

Harvard Business Review 49 (July-

Technique,”

1971), pp. 45-74.

Kalman J., and R. M. Cyert, “Strategy:

Formulation,

Implementation,

and Monitoring,”

The /our&of

Business, July 1973. The Conference Dean,

Board, “Appraising

the Market for New Industrial

Products”

N. C., The Delphi Method, (Santa Monica: Rand Corporation),

Dalkey,

B. V., “Evaluating,

Management Diffenbach. Harvard Edelman,

Selecting and Controlling

Association,

J., “Evolution University, 1969)

of New Energy Technologies: J.. “Venture

Progress Toward Commercialization,” and Risk,” FinancialExecutive 37

Analysis: The Analysis of Uncertainty

in Policy Analysis,”

Forrester, J. W., World Dynamics (Cambridge, Frank, E. W., “Business Evaluation Froomkin. J., “Needed:

A., “Technological

W. H., “The Strategic

PO/icy Analysis, Winter

Massachusetts:

of Research,”

A New Framework

Gee, S., “The File of Technology Gruber,

Factors Affecting

dissertation.

pp. 56-62.

Fisher, G., “Cost Considerations

Gerstenfeld,

R&D Projects” (Research Study 89) (New York: American

1968).

unpublished

F., and Greenber,

(August

(SBP No. 123).

RM-5888-PR.

Transfer in Innovation,” Integration

Press, 1971).

Financzhf Executive 32 (July 1964).

for Analysis of Government

Forecasting,”

1977.

Wright-Allen

Programs,”

The Journaf of Business, January of Corporate

PO/icyAnalysiJ, Spring 1976.

Research Management 17 (November R&D,” American

1974).

I97 1.

Management

Association,

October

1981. Hanan,

Mack. “Corporate

Growth Through

ary 1969). Hitch, C. J.. and R. N. McKean,

Venture Management,”

Harvard Business Review 47 (January-Febru-

The Economics of Defense in the Nuclear Age (Cambridge:

Harvard University

Press, 1960). Keen, P. G. W., and M. S. Scott Morton, Wesley,

1978).

Decision Support Systems: An OrganizationaL Perspective (Addison-

Planning and Analysis for Large-Scale Technology Developments Little, J., “Models and Managers: McGlauchin. McGuire,

L. D., “Long-Range

P. E., “Evaluating

The Concept of a Decision Calculus,” Technical

New-Product

McKean, R. N., Eficiency in Government

Planning,” Proposals,”

Management Science 16:8 (April 1970).

Harvard Business Review (July-August The Conference

407

1968).

Board (No. 604), 1973.

Through Systems Andysir (New York: John W. Wiley & Sons, 1958).

Meador, C. L., and A. C. Parthe, “Managing Macro-Development: Policy, Planning and Control System Implications,” in F. P. Davidson, C. L. Meador and R. Salkeld, eds., How Big andSti/(Beautiful? Macro-Engineering Revisited (Boulder, CO: Westview Press, 1980). The MITRE Corporation, “A Technology Assessment Methodology” (Springfield, VA: NTIS, 1971), pp. 22-120. Montgomery, D. B., and G. L. Urban, Management Science in Mudefing (Englewood Cliffs, NJ: Prentice-Hall, 1969). Moore, J. R. and N. R. Baker, “Computational Analysis of Scoring Models for R&D Project Selection.” Manugement Science 16 (December 1969). Pessemier, E. A., “New Product Ventures,” Business Horizons 11 (August 1968) pp. 5-19. Quade, E. S., Anafyrirfir Pub/h Decisions (New York: American Elsevier Publishing Co., 1975),pp. 22-30. Quinn, J. B., “Technological Forecasting,” Harvard Business Review (March-April 1967), p. 1. Schlaifer. R., Arzdyh of Decisions Under Uncertainty (New York: McGraw-Hill, 1968). Schlesinger, J. R., “Quantitative Analysis and National Security,” Wodd Politics (1963). p. 298. Schwartz, J. J., “How an Organization Decided to Innovate,” Wharton Quutierly 7 (Spring 1974). Seamans. R. C. and F. I. Ordway, “The Apollo Tradition: An Object Lesson for the Management of Large-Scale Technological Endeavors,” InterdirciplinaT Science Reviews 2:4 (1977). Shubik, M. and G. D. Brewer, Mode/s, Sirnufations a&Games - A Survey (Santa Monica: Rand Corporation. 1972). R-1060 ARPAIRC. Sivern, D. H.. “Product Opportunity Analysis,” Management Accounting 50 (February 1969) pp. 44-46. Souder, W. E., “Comparative Analysis of R&D Investment Models,” AIEE Tramactions, Vol. 4 (March 1972). Souder. W. E., “A Scoring Methodology for Assessing the Suitability of Management Science Models,” Manugement Science 18 (June 1972). Twiss, B., Managing Tecbnologicd Innovation (London: Longman Group Ltd., 1974). Utterback, J. M., “Innovations in Industry and the Diffusion of Technology,” Science 183 (February 1974), pp. 620-626. Zimmer, R. P.. et d.., Benefit-Cost Metbodo/ogy Study, DOE/NASA 7827-75-l.