Methods to predict performance in major program acquisition

O),IEG,4 The Int. Jl of Mgmt Sci. Vo[. II. N o 2. pp. 155-173, t983 Printed tn Great Britain. ,All rights reserved

0305-0483 83 020155-[9$03.00 0 Copyright !~ 1983 Pergamon Press Ltd

Methods to Predict Performance in Major Program Acquisition AJ ROWE University of Southern California, USA

IA SOMERS Hughes Aircraft Company, USA (Receiced January 1982, in revisedform October 1982)

Numerous instances have been documented of major acquisitions where costs far exceeded the original estimates. With increasing levels of system complexities, limited resources, constant changes in scope and advances in technology, the probability of cost growth is a critical aspect of the acquisition process. Even where technology is considered as being current state-of-the-art, programs are not immune to cost growth. Considerable effort has been expended in attempting to find why most acquisitions have cost overruns. The research reported here builds on prior work in the field and provides an approach for dealing with the uncertainty involved in acquisitions. The objective of this paper is to develop an approach to predicting cost uncertainty which recognizes that variability cannot be eliminated but that there are trade-offs available to decision makers predicated on formulating cause and effect models. Thus, inherent in the approach presented here is system dynamics, interdependencies, variability and uncertainty in the acquisition process. The approaches used to achieve desired objectives are also described.

T H E ACQUISITION PROCESS ANY DESCRIPTION of the acquisition process is, at best, only a static representation of an extremely complex set of interdependent activities. For our purpose, we will use two basic diagrams to aid in understanding the process. The first, the matrix shown in Fig. 1, describes the kinds of uncertainty associated with acquisitions and attempts to describe the bases of causality. Thus, internal control assumes all things are known and controllable with estimates based on past data, procedures, designs, suppliers, etc. The other three categories, however, represent the reality in major acquisition, even though management does not have complete control or ability to predict outcomes. It is this realm of uncertainty that has significant impact on cost and is the principal emphasis of this paper.

The second approach to understanding problems in the acquisition process and the inherent cost overruns is represented by the diagram shown in Fig. 2 where the factors, the interdependencies and the processes involved in acquisition management are illustrated. Although it does not reflect the dynamics and interactions that occur in an on-going organization. Fig. 2 does illustrate a number of key concepts that will be developed in this paper. The linkages between the four basic uncertainty variables and acquisition management help to define the processes and activities or variables that contribute to the uncertainty of acquisition management. The four basic uncertainty variables considered are: (1) organ&ational s l a c k - - a measure of the organization's ability to perform the task requirements; 155

156

Route. Somers--Per[ormance in .14ajor Program Acquisition UNPREDICTABLE

Technological uncertainty

LEVEL OF UNCERTAINTY INVOLVED IN ACQUISITION

Environmental uncertainly

Advanced technology Inadequate task definition Concurrency of design and production Design changes

Unpredictable events Failures Disruptions Force majeure Regulations

/nternal control

Customer uncertainly

Project controls Resource allocation Schedules Estimates Budgeting

i i ] i i

Contractual requirements Scope changes Time compression Stretch-out Constraints Inadequate funding

PREDICTABLE

CONTROLLABLE

UNCONTROLLABLE DECISION MAKER'S ABILITY TO CONTROL ,ACTIVITIES

FIG. I. AcquL~ition uncertainty.

(2) customer urgency--the time compression, concurrency, or degree of overlap between phase of development and production, including changes in scope;

Considering the whole diagram, one can see that the variables and linkages define a network of interdependencies which ultimately contribute to the uncertainty and the consequent problems in acquisition.

(3) technological uncertainty--a measure of the state-of-the-art and the degree of interdependency among system components;

DEFINING THE PROBLEM OF COST OVERRUNS

(4) em, ironmental uncertahuv--the factors that cause disruption, delays, shortages, failures, etc. that are not under the control of management in the acquisition process.

In its report to Congress, the General Accounting Office (GAO) [45] claims that major weapons' cost growth since World War II far exceeds the rate of inflation and that various

ORGNZTONL

COOS'TON Jt "°'MENT

'OENCY COUSTOM.

",,,cost

SCHEDULE

FIG. 2. Acquisition management process.

/

Orneea, Vol. II. No. 2

NIGH

157

HIGH

d

0 hI.Z 0 L>

Z z u o_

It.

0 t03 0

ERRORSj

~ - C

(P)

0

.\\\\\

LOW LOW

0 L)

"-<2_

LOW HIGH

DEGREE OF CONTROL

(a)

LOW

HiGH

DEGREE OF PLANNING

(b)

FIG. 3. Effects 0/" let'els o[ control on net benefit and total cost.

efforts to restrain costs seem unlikely to achieve really substantial cost reductions. This statement by the G A O goes counter to the general belief that has been held which contends that closer scrutiny of the acquisition process, such as tighter control and stricter acquisition policies on the part of the government, will significantly reduce the levels of cost growth. However, control alone will not solve the problem since the cost of additional controls can be considered part of a cost growth--adding to the problem rather than the solution. Figure 3a illustrates the effects of various levels of control on total cost. The net benefit of control curve (value less cost on a point-by-point basis) shows the dependence of cost on level of control. In this context, net benefit is equivalent to the economic concept of net value which considers changes in value as a function of changes in cost. This curve shows the optimal value and how changes in control (increase or decrease) affect the net benefit. Figure 3b illustrates a similar approach using degree of planning as a function of cost of planning and cost of errors (lack of complete planning). In this case, the total cost curve (cost of planning plus the cost of errors) will have some optimal value from which a change in degree of planning will increase the total cost. An alternative to the use of control as the sole means for reducing cost is an approach based on predicting the likelihood of disruptions occurring during performance of a contract and thereby permitting cost reductions. Thus, based on a relationship between cost growth and likelihood of disruption, the cost of control can be reduced. Knowing when and where disruptions might occur (with a defined proba-

bility), the acquisition manager can develop a basis to reduce the likelihood of disruption and thereby lower the level of cost growth. In addition, other approaches can be applied in advance of a probable disruption which also will lower the impact on cost growth due to the effects of disruptions. 'Disrupt' means cause disorder or turmoil. In our context, the term disruption refers to the disorder and turmoil created in a program plan and in related production procedures. Hence, disruption cost is the difference between the actual cost for a program on the one hand, and the cost 'reasonably required' to perform the task in the configuration finally delivered on the other. The 'should-cost as-built" estimate includes the estimated cost of all changes incorporated in the final deliverable product. It does n o t include penalties caused by the late incorporation of changes (retrofit, rework, lost learning, etc.). Figure 4 graphically illustrates these differences. The existence of a sizeable overrun always raises the question as to the adequacy of the ACTUAL COST

ISRUPTION COST

SHOULO COST AS BUILT CHAN

S

H•NGES

N FIG. 4.

Cost due to disruption.

15S

Ro~e. SO,ler~--Perfi)rrnance in J.Ia/or Progrctm .4cqu~sltion

original estimate used as the basis to determine the degree of overrun. The overrun may reflect a poor assessment or errors in the estimating process, both leading to a low estimate: such a discrepancy itself could well be disruptive. We shall use should-cost based on a realistic estimate for each organization as an "objective" definition of disruption. This approach suggests using estimating methods which consider fundamental characteristics of the product, such as parametric cost estimating. Cost growth is not limited exclusively to the acquisition of weapon systems. Literature is replete with examples of cost growth on various projects. For example, the Roman Aqueduct had a 10030 overrun and the Suez Canal had a 200°;; overrun. Another example of cost overruns is described for energy processing plants [31J--the overruns ranged from less than 200% to almost 700!',~ for the actual cost compared with the initial estimates. When programs such as the Apollo Space Program (with a 200%0 overrun) and the Space Shuttle Program are added, the types and levels of technologies spanned is diverse. Both military and nonmilitary projects suffer from cost growth. Furthermore, cost growth is not limited to programs that have advanced state-of-the-art technology. One of the classic cases of cost overrun, the F-l I l, is described in the paper by Roesch & Sage [37], who report that unprecedented cost growth of $3 million per unit in 1966 to almost $15 million per unit in 1970 was caused by ineffective program management on the government's part, poor DOD/contractor relationships, over-centralization of acquisition management in the Office of the Secretary of Defense and poorly defined operational requirements. As a counter-example to purely military projects, Cochran [10] examined a number of nonmilitary overruns. Based on an analysis of the annual cost growth, which ranged from 4 to 18%, inflation accounted for the major portion of the overrun in six out of 14 programs. Overall, cost overruns ranged from 124.3 to 387°/(, The GAO in its January 1979 report [44], indicated that inflation accounts for approximately 50% of the cost growth in military projects. Thus, both military and non-military programs suffer from the effects of inflation which, in the era of double digit inflation, are becoming acute.

FACTORS CAUSING COST OVERRUNS There have been many studies attempting to identify the causal/'actors involved in cost overrun. In this section, we will summarize findings from a number of sources and use these results. and the cost trade-off curves in the next section. as the basis for development of a model to predict likelihood of disruption and the concomitant cost growth. The GAO and RAND Corporation have done the most extensive analysis of cost growth. In a report to Congress [46], the GAO identifies the military's desire for maximum perIbrmance, high technology weapon systems along with instability of congressional funding and constraints as the major contributors to cost growth. They also identify low rates of production, absence of price competition, lack of motivation tbr contractors to reduce cost, impact of government controls, reduced R&D expenditures and lowered productivity as additional elements contributing to cost growth. Their conclusions on how to reduce cost were: to provide program stability that helps recover investments, to put more emphasis on invested capital than on production costs and to provide greater flexibility in meeting changes in priorities and needs. In its June 1980 report to Congress [47], the GAO identified the issues that had a direct impact on weapon system's mission effEctiveness. The majority of the issues were concerned with operational or performance limitations, survivability or vulnerability, availability, meeting requirements and reliability. In major program acquisitions, the issues were system affordability, requirements of data reporting, concurrency of production and development, inadequate testing, cost-effectiveness assumptions, lack of qualified personnel, indecision on system urgency and technical risks. Since the late 1950s, the RAND Corporation has conducted studies on the inherent uncertainty of the development process. From early 1960 to date, it has issued a number of increasingly specific reports to show that design and production concurrency accompany large overruns, which were avoided when development work substantially preceded production. Quantitative measures of state-of-the-art demonstrated impressive correlation between degree

Or~ze~a. ~'ol. /1. ~o. 2

of state-of-the-art and the proportion of cost overrun for cases of design and production concurrency'. As a result, R A N D researchers recommend reducing uncertainty in advanced technology programs by a return to "incremental" product d e v e l o p m e n t - - a procedure which has long been used by manufacturers of. commercial products with stable design [42]. The R A N D studies also show that cost and schedule problems are larger as the degree of state-of-the-art advance increases. Sophisticated management planning and control programs and incentive contracting have had little effect. In addition, improvement in cost estimating and monitoring of cost growth does not provide substantial cost reduction. Even the use of concurrency may not reduce the development time. In short, the main problem lies with the uncertainty that affects development and production costs in the presence of urgent time schedules. In describing system acquisition experience, Perry [35] points out that initial estimates tend to be overly optimistic and not to consider, or to understate, technological difficulties actually encountered in program development. They found that, in nearly all cases, renegotiated contracts were much closer to actual performance requirements and that this was reflected in adjusted costs. Thus, the earlier a cost estimate is made, the greater the expected uncertainty of actual cost. In general, in the early conceptualization stage, the required technological advances and eventual system configuration are not well defined. Concerning cost growth and performance faults, they found that they were principally caused by changes in program scope which were outside of the contractor's control--these factors generally accounted for the difference between predicted cost of the original program and the final cost of the program actually delivered. The report by Large [24] addressing bias in cost estimates illustrates another dimension of the problem of cost overruns. In an analysis of the comparison of initial bid with final cost, he found that where the technological advance required is not fully understood, the final cost can be off by a factor of two. He cites statements made by contractors which indicate clearly that initial estimates are designed to win the competition, an approach known as "buying-in'. Martin et al. [29] in their paper on the re-

159

lationship bet~veen cost and cost estimation indicate that a cost estimate is. at best. a reasoned guess about a future outcome, which requires judgment and therefore is subjective in nature. Furthermore. because cost estimates are probabilistic in nature and are valid only as long as the assumptions on which they were based remain the same, there is a need to revise estimates in consideration of the uncertainty which exists. They point out that the level of uncertainty is very high in the early phases of an acquisition, in part due to the vagueness of system specifications: it is reduced as information is obtained from testing and evaluation. The conclusion drawn is that mistakes are made using the wrong approach for the given phase of the acquisition cycle. This conclusion is supported by studies which show that initial estimates seldom reflect the final cost. Davis, in his analysis of uncertainty associated with cost estimating [12], contends that lack of information contributes to the uncertainty of cost estimates and that diffusion of authority and responsibility for cost estimating throughout the acquisition cycle complicates the process. Furthermore, short-fuse requirements impose severe time constraints and also contribute to estimating errors. He concludes that there is a need for greater flexibility in the acquisition of major systems in order to cope with program uncertainties, which could be achieved by cost range, instead of single point, estimates. The problem of cost estimating is poignantly described in Allen's paper on a theory of cost growth [1]. It describes cost as a 'dependent" variable that reflects actions taken in response to contractual requirements. Thus, if cost becomes uncoupled from requirements, then the numbers are meaningless--point estimates speed up this uncoupling. A number of reasons why costs become uncoupled from reality are: (1) budgets do not anticipate technical requirements, but instead react in a lagged manner to changes in requirements; (2) there is a serious gap between those who are knowledgeable about costs and those having technical knowledge; (3) annual budgetary considerations tend to dominate the incremented acquisition process;

160

Rowe, Sorners--Per/i)rmance in 3,1a/or Program Acquisition

(4) the sheer magnitude of numbers of people involved in government functional areas proliferates the problem and reduces flexibility: (5) there are differential effects of inflation on given program components; (6) competition causes cost estimates to decline in the pre-contract award period: (7) monopsony--a market condition where the customer controls demand and tries to minimize cost. The recommendation that was made, then, was to change these practices so that the acquisition process more nearly follows normal competitive practice, which would avoid cost growth based on unrealistic initial bids and unreasonable budgeting and estimating practices during the acquisition cycle.

RESEARCH ON CAUSES OF COST G R O W T H In a stud,', conducted at the University of Southern California [38], six major programs were analyzed to determine the primary causes for cost growth, schedule slippage and performance degradation. Twenty-six factors were identified as specifically contributing to cost overruns. These were placed into the four basic categories described in the beginning of this paper. The research found that the importance of the factors depends on specific environmental conditions and that rank-ordering has no meaning for general application. The relevance of each factor to the six programs is shown in Table I. The result was that every one of the programs had cost overruns and all encountered schedule slippage, with BART and the F - I l l having pec'~Ormance degradation. Customer urgency had the most pervasive impact on all programs, technological uncertainty was second, with organizational slack third and envi-

TABLE I. CAUSESOF COSTOVERRUNS ,4. Organizational slack (1) Lack of incremental (i.e. milestone) development. /2) Lack of control of the entire project. (3) Overlapping development of interdependent projects. (,-1) Split and dispersed organizational control of key elements in a high risk project (i.e. fragmentation of responsibilities). (5) Inadequate consideration given to trouble areas (i.e. subsystem dependencies) which could delay the program. (6) Incomplete preplanning. (7) Lack of organizational cohesiveness and continuity. (8) Lack of manpower to deal with design changes. (9) Incongruent personnel career objectives. B. Customer urgency (10) Inadequacy (or incompleteness) o1" task definition at the time of contract and poor DoD/contractor relationship. (l I) Contracting simultaneously for cost, time and technical performance. (12) Contract provisions (e.g. elimination of contingency provisions, total package concept) and negotiation techniques. (13) Mutual acceptance of unrealistic prospective cost estimates of product and delivery schedules (i.e. cost optimism syndrome, schedule and risk optimismL (14) Overlap of development and production phases (concurrency). C. Environmental uncertaint)' (15) Lack of financial strength for Iarge, long-term risky projects. (16) Low bidding, while lacking the resources to finish the job. (17) Political/economic pressure to win the competition at any cost. (18) Economic pressures for a general reduction of expenditures. (19) Inflation, regulation and poor cost estimates. (20) Optimistic promises concerning schedule, cost and technical performance. D. Technological uncertaint)" (21) Underestimation ol"the degree of technological breakthrough required in the state-of-the-art o1 product development, while under a fixed and tight time and performance constraint. (22) Pushing technology too last. (23) Lack of prototype development. (24) Performance requirements beyond state-of-the-art. (25) Inadequate test program. (26) Major design or scope changes.

Omega. ~'ol. 1 l, .Vo. 2

161

TABLE 2. MAJOR CAUSAL FACTORS FOR SIX PROGRAMS

Cause

RB2ll Engine

.4. Organizational slack (I)

(2) (3)

Program BART Train F-I I l Aircraft

x

x

x x

SRAM Missile

C-5A Cargo

X

X

X

x

(4)

(5)

LHA Craft

X X

x

×

x

X

X

*(

X

X

K )<

×

(6)

(7) (8)

x

x

(9)

x

x

110)

x

x

X

X

X

X

(It)

x

x

X

X

;'(

X

(12) (13) (14)

x x x

x

X

X

K

X

x

B. Customer urgency

X

x

X

X

×

X

x x

X

X

X

X

X

X

C. Encironmental uncertainty (15) (16) (17)

x x x

(18) (19)

x

(20)

X

D. 7~chnological uncertainty (21) (22)

x x

(23)

x

(24)

x

(25) (26)

x x

x

X

X

X

X

X

×

X

X

X

x x

ronmental uncertainty having the smallest impact. The specific events or factors contributing to the effects shown in Table I are summarized in a matrix (Table 2). In retrospect, it is little wonder that these projects, which span civilian and military, high and moderate technology, all experienced varying degrees of difficulty in meeting cost estimates and contractual commitments. (As shown in Table 2, not all factors occurred in each of the projects analyzed--some factors occurred in all projects, such as factors 10, I I and 12; others occurred only in a single program, such as factors 15, 16 and 17 for the RB211 engine and factor 20 for the F - I l l aircraft.) Examination of the data from other studies conducted reveals that there are no simple answers to reducing cost growth when dealing with the uncertainty inherent in the acquisition of programs. Advances in state-of-the-art and changes in design to meet requirements that are determined during system test and evaluation are generally not under the program manager's control. Furthermore, there will inevitably be some degree of concurrency during development, as well as overlap of authority because

X X

X X

×

of multiple organizations involved in the process. There is little doubt that over-optimism in new designs leads to design changes and ultimately to cost overruns. Inflation, changing political and customer influences and environmental catastrophies will continue to plague the acquisition process. Low bidding, poor or inappropriate estimates, improper budgeting and cost control--all contribute to the problem. When these factors are compounded with interrelatedness, delays, disruptions, concurrency and extreme variability, it is little wonder that no definitive answer has emerged to solve the cost overrun problem. What is needed is an approach to the "management' of risk and uncertainty that will minimize the occurrence of acquisition disruptions.

D E V E L O P M E N T OF AN A P P R O A C H TO T H E M A N A G E M E N T OF A C Q U I S I T I O N UNCERTAINTY In the sections that follow, patterns of disruption will be examined and the risk and

1~2

Ro~e..S',mi¢'r.~--Pertbrmance in .tltgor Prec,.ram Acquisition

uncertainty, as they pertain to causality models. ~ill be identified. Peck & Scherer's comprehensive analysis [34] of the weapon systems acquisition process defined risk as the level of consequences of a wrong prediction. They operationally defined uncertainty as the relative unpredictability of an outcome of a contemplated action. They categorize uncertainty as either internal or external where internal uncertainty related to the possible incidence of unforeseen technical difficulties in the development of a specific weapons system. Examples of internal uncertainty include development time of interrelated technologies, substitutable technologies and performance to specification. External uncertainty covers factors external to a given project, but that affect the course and outcome that can be expected. Examples include rate of technological change in weaponry, changes in strategic requirements and shifts in government policy. Harrison [22] defines risk, certainty and uncertainty as follows: (a) risk--a common state or condition in deci-

sion making characterized by the possession of incomplete information related to a probabilistic outcome; (b) certainty--an uncommon state of nature characterized by the possession of perfect information related to a known outcome; (c) uncertainty--an uncommon state of nature characterized by the absence of any information related to a desired outcome. Harrison further contends that "'genuine uncertainty is as common as complete certainty". The more common state of nature is incomplete or imperfect information, which means that the expected outcome contains an element of risk for the decision maker. There is no situation that deals with the future that can be completely known when the acquisition process lasts anywhere from two to twelve years. How can a program manager possibly forecast events that far in the future with any meaningful degree of accuracy? Beverly et al. [3] describe uncertainty in systems acquisition as the lack of knowledge in

development requiring state-of-the-art technolog,v; the,," apply risk based on historical phenomena for which probabilities can be established. On the other hand, certainty or uncertainty deals with the existence of knowledge: uncertainty is greatest when kno~Iedge is at its lov~est level and would describe the situation where a new system is being developed which involved advanced state-of-the-art technology. Lack of knowledge inevitably leads to errors in estimating, in design and, ultimately, in cost control; these errors lead, in weapons acquisition, to three kinds of uncertainty: in design and technology, in scheduling and in cost. The authors point out that there is a conflict of goals because reduced design.technology uncertainty enhances performance, while cost minimization tends to adversely affect both performance and schedule goals. McNichols [30] presents a means for estimating the distribution of cost uncertainty where actual costs differ significantly from original cost estimates. He contends that cost overrun is a meaningless concept because all cost estimates rely heavily on subjective judgments and are subject to considerable uncertainty. He considers four basic requirements in the treatment of uncertainty: (1) generation of probability distributions for individual cost elements; (2) generation of total cost by additive distributions; (3) combining probability density functions to form a compound distribution; and (4) consideration of dependence or degree of correlation between cost elements. The problem of dealing with uncertainty is the difficulty in determining the likelihood of achieving cost goals. The descriptions of risk and uncertainty presented above illustrate the variety of approaches that can be taken. The relevant question, however, is how best management can deal with the problem of uncertainty in the acquisition process. Although it is commonly assumed that any major overrun signifies poor management, this premise fails to recognize that uncertainty is inherent in acquisition and that managers operate under severe time and resource constraints. In view of this inherent uncertainty, managers must consider: (a) did the project warrant the resources and risks involved; (b) were the alternatives rationally defined;

163

Omc'4a, ~,,d. 11, .'Vo. 2

(c) were considerations given to the probable resuRs of or alternatives to the failure of the project: (d) were suitable steps to prevent failure or to hold cost to a minimum made available: (e) was there adequate monitoring of events to detect deterioration of the situation at an early stage, and to limit losses. Clearly, managers take risks whenever they proceed in the face of unknown situations. Unfortunately, the degree of uncertainty and the potential effects of unfavorable events are often difficult to determine since many situations involve potential events whose probability is not measurable. All too often the term ~calculated risk" refers to a decision reached in recognition of factors known to be unfavorable but "not susceptible to calculation'. The future may even involve events of a totally unexpected or unknown type. Thus, in dealing with complex problems under limited time and resources, substantial errors in judgment can occur in unpredictable ways.

A C A U S A L BASIS F O R DEFINING UNCERTAINTY Although uncertainty is defined as a lack of knowledge about specific effects, it can be examined in terms of the factors that contribute to disruption (shown in Table 3), in an attempt to understand the causal relations that lead to cost increase. (The premise is that control of the variables contributing to uncertainty is an effective means of controlling cost growth. This is analogous to queueing theory, where a knowledge of queue behavior and sequencing rules permits servicing the maximum demand with TABLE 3. FACTORS IN DISRUPTION 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Delay--gaps in carrying out a program. Interruption--short delay. Stretch-out--slow down of program. Interference--~elay by stoppage. Redesign---change scope, redo previous work. Work stoppage--partial interruption. lnterdependencies--indirect delays. Shortages or errors--delays due to rework. Overlap--interferences and delay due to concurrency. Redirection of eflbrt---disruptive effect of reorganization.

available resources. Delays are not eliminated: rather, they are reduced by adding capacity or modified by changing priority ru!es.I Two factors that cause disruption in the acquisition process are concurrency and technological uncertainty. Concurrency is a strategy which requires the overlap of the development,-test. production and operation activities in order to reduce the 'conceptual to operational" lead time: it is most often a result of customer urgency in attempting to meet tight deadlines. Delivery urgency, enlbrced by competitive conditions, exerts strong pressure on suppliers to commit themselves to delivery dates which are inherently optimistic or based on the assumption that no serious problems will develop. The plan becomes even more urgent when this situation is combined with some degree of technological uncertainty.

TECHNOLOGICAL

UNCERTAINTY

As used here, technological uncertainty refers to two conditions: (1) the highly abstruse demands at the very forefront of scientific knowledge or state-of-the-art; (2) a major gap between an organization's area of expertise and what it is required to perform effectively. Rapid technological change can have a major, perhaps catastrophic, financial impact on an organization, which can be termed a "technical disruption'. Uncertainty therefore occurs simply because it is difficult to make valid judgments, when even experts have incomplete knowledge. Managers, in turn, must rely on the recommendations of technical people, and yet they must be able to detect errors and inconsistencies. Hence, the manager's incomplete knowledge can be a limiting factor. At the same time, the amount of time used for testing and proving a new concept is held to a minimum during a period of rapid technological change, when there is usually intense competition since a major advance by one enterprise reduces the business potential available to others. Factors that determine the state-of-the-art (shown in Table 4) include the newness of technology as well as the design requirements. Thus, state-of-the-art for a given organization can be construed as the ability to produce a given design, in addition to the newness of the technology involved. Uncertainty due to teeh-

R~me. Sarner~--Per)~)rrnance in .~[ajor Pro,¢ram Acquisition

~6-.t

TABLE 4. Fa.CTORS DETERMINING THE STATE-OF-THE-ART 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Size--number of interrelated components, physical volume. Complexity---difficulty in meeting performance requirement. Newness of technology---experimental state of technology. Percentage of proven technology~degree of newness. Experience in the field--work on similar programs. Percentage of new components--test and evaluation requirements. Interdependency of subsystems--types of linkages. Degree of precision--quality or cleanliness requirements. Special resources--testing or tooling requirements. Definitive specifications--clarity in meeting requirements.

I I. Design flexibility--tolerancelevel, substitutes available. 12. Required theoretical analysis--need to support proposed design. 13. Degree of differencefrom existing technology--life cycle of technology. 14. Available knowledge--amount of experimentation required. 15. Infrastructure required--degree of dependency on vendors.

nological change often arises from the concurrence of development and production. The perceived necessity of initiating the ponderous and involved processes of production before there is real certainty as to the stability of the product design places these at the mercy of any delays or changes which may occur in the design. Such delays or changes are more likely to occur as the degree of concurrency increases. During rapid technological change, the pressure to take on new projects is greatest, but so too are the risks in doing so, Unfortunately, there is no warning that this risk will turn out to be unacceptable, The effects are pervasive and far reaching, ranging from the ineffectiveness of familiar technical skills and operational procedures to the impact of recondite scientific laws on the design, construction, operation and maintenance of a new product.

INTERDEPENDENCY If the degree of state-of-the-art is a driver of technological uncertainty, then interrelatedness is a major multiplier of'the cost of development and production. Interrelatedness of design results in a cascade of changes because each component or subsystem affects many others. Interrelatedness can also affect production and vendor activities, since a change in production methods or delivery cycle in one area or component (generated by design delays and changes) may affect production of other components or work in other areas. Again the process can become very complex with many ripple effects. In general, a product operating in

a more advanced area of technology will be more subject to interrelatedness effects. O R G A N I Z A T I O N A L SLACK Two other factors compound the disruptive effect of concurrency: (1) the level of resources effectively available to the project and (2) the degree of external control over events. The level of resources comprises all types of resources-technical, managerial, financial, etc.--and their adequacy is measured by what might be termed "organizational slack', which relates to the organization's experience in the basic technology involved. It provides an invaluable fund of knowledge and skill in handling the unexpected problems which arise and which could overwhelm an inexperienced organization. A second problem is the degree to which the task at hand fully engages all resources available over the time-frame of the project, since lack of availability may leave inadequate reserves for use on unexpected problems. This inadequacy can become a critical flaw, given the intense time compression inherent in concurrent design and production. Organizational slack is used to define the level or degree of unknowns that are internal to the system rather than the external exigencies. Factors related to internal uncertainty could be measured using dimensions such as: (l) the organization's ability to respond to new or unforeseen requirements; (2) the flexibility that has been built into the organization;

Omega, Vol. I/.

(3) prior experience with the given technology: (4) number of linkages of subsystem or interaction with other projects: (5) percentage of the project's subsystems being developed that are at the state-of-theart of the technology:

,Vo. 2

165

but deals with the question of whether the organization is able to cope with problems as they arise or to anticipate problems. In turn, the amount of slack or flexibility in the organization determines the ability to respond to uncertain requirements. If management is operating with minimum slack, then any disruption will cause a large delay.

(6) the amount of time compression or tightness of schedules;

ORGANIZATIONAL PERFORMANCE

(7) availability and/or access to resources; (8) maturity in the planning and control of operations, including computer systems and organization structure; (9) amount of overlap of development, design and implementation: (I0) number of contractors or organizations involved in the project. These factors contribute to a measure called "organization response capacity'--that is, management's ability to cope with uncertainty. In turn, the delay, disruption or slippage that can be anticipated would be shown by the relationship of this capacity to customer demand, as in Fig. 5. Expected disruption is an exponentially increasing function which is dependent on the organizational response capacity, which in turn depends on the level of concurrency. Thus, when the level of concurrency approaches the response capacity, the delay increases. This formulation does not deal with uncertainty p e r se,

HiGH

---f

)RGANIZATiONAL SLACK

a 0

__. c-

/

Another aspect of organizational slack relates to expected performance. Cochran [9] has identified key factors which contribute to disruption as a result of task variability-and which management can review in order to achieve more effective control: (a) inadequate definition of product specification; (b) underestimating the degree to which stateof-the-art must be advanced; (c) poor cost engineering or organization planning; (d) not allowing for the degree of uncertainty in meeting plans: (e) not anticipating the "backup' activities required in case the main approach fails. Cochran [9] also described the S-curve patterns of labor hours (also referred to as the S learning curve) as a cause of disruption leading to substantial cost overruns when development of a

CAPACITY TOIRESPONO

PECTED RUPTION

LOW SERIAL

OVERLAP CUSTOMER DEMAND REPRESENTED BY DEGREE OF CONCURRENCY

FIG. 5. Impact q/" concurrency on disruption and response capacity.

166

Ro~e. Somer~--Perlornlance in ~!,lalor Pr¢*vram .4cqut~ittnn

major new design is concurrent with production and under severe deliver,', pressure. Labor cost reflects the impact of design delays, design growth and changes in the production function. The disruption caused by the S-curve effect generally continues well beyond the first units produced, because of the way in which production operates. The procedures, tools and methods established during the start-up period inevitably carry forward to subsequent periods and costs follow accordingly. Managers generally acknowledge that it is harder to revise engrained organization practices than it is to start from scratch, since design growth and changes cause revisions in production methods and sequencing and in facilities usage. Therefore, if a change is introduced after production has been established, considerable time is required to fully implement the program; and, if design changes occur after the affected components have been completed, this requires rework and reinstallation, which involves extra cost (the cost of such work is dependent on the degree to which it is different from the position or sequence normally assigned to the original task). This work also creates extensive interference with other on-going tasks, which can involve correspondingly greater cost penalties.

D E T E R M I N I N G A PATTERN OF D I S R U P T I O N The ability to define causal relations among variables in disruption and uncertainty is a first step in predicting cost overruns and in determining which actions a program manager could take to avoid cost growth. For example, Augustine [2] proposed using additional planning funds based on an assessment of risk. He contends that even the most capable program manager is not able to forecast all the problems that will be encountered in a development program spanning anywhere up to ten years, but that it is, however, quite possible to forecast the "probability' that additional funds will be required. He recommended the use of T R A C E (total risk assessing cost estimate) as the basis for justifying the additional funding. One of the early attempts to deal with uncertainty was proposed by Marschak et al. [28]; who indicated that where "component' interrelatedness is defined, one can predict the effects

that are likely to occur. Under conditions of uncertainty, low slack heightens interrelatedness and substantially increases the risk of redesign. Furthermore, the risk of redesign is sensitive to (a) the degree that design reaches beyond the former state-of-the-art and (b) any requirements to use existing components, which can strain the designer and lead to suboptimization. When there is a high degree of close coupling or interrelatedness, the likelihood of design change is substantial: when there is loose coupling and engineering slack, which permits deviation to occur so that when components are redesigned the deviation does not influence the other components, there is less propensity to redesign. It is argued that the tightness of component interrelatedness can be traded off against uncertainty, and thus achieve more effective control.

RISK MODELS Many causal relations currently utilize risk rather than uncertainty to predict possible outcomes. Figure 6 shows the relationship between risk and uncertainty as related to causality. Models of known phenomena provide a more certain basis for prediction than random events which are used for estimating probabilities. Uncertainty, on the other hand, covers those areas that are ill-defined or where there is lack of knowledge of effects. Risk and cost under conditions of certainty, moderate uncertainty and high uncertainty can be described by Fig. 7. Merrow [31] developed a relationship, shown in Fig. 8, between cost growth and 'months

CE~T~,N

N

--

UNCERT~.rN KNOWN

~&NOOM

(MODELS)

{ PROB.~81LITY OF ERROI'3S)

UNKNOWN (ILL-DEFINED)

INFLUENCES ON C~,US*'~LiTY

FIG. 6. Taxonomy of causality and uncertainty [39].

Omega, Vot. 1I, ,Vo. 2

I67

REGION OF LJNCERTAINTY

a. A -16 0

0 p-

0

COST

COST

{ Q ) NO UNCERTAINTY

( b ) MOOERATE

h

UNCERT.&INTY

-2~ O

t.or, _~

(O

i~,._REGIO N O F ~

= A;4toS"

I I 1 [ I [ 120 IOO 80 60 40 20 O MONTHS BEFORE INITIAL OPERATIONAL DELIVERY

0 COST

(V'cost growth with program duration and level o/" technical d(fficuhy [31].

FIG, 8. Variation

( C ) HIGH UNCERTAINTY

FIG. 7. Rehttion,~'hip.~"between cost, risk and uncertainty.

before initial operational delivery' as a function of state-of the-art. The curves in Fig. 9 relate the level of technological uncertainty to the program life cycle and advances in state-of-theart by showing the probable impact of a program stretch-out on technological uncertainty. Figure 10 [37] attempts to relate state-of-the-art to interdependency and level of concurrency and the likelihood of disruption is shown as a function of varying levels of concurrency. Another version of the likelihood of disruption developed by Cochran & Rowe [8] is shown in Fig. 11 as a function of delivery urgency and technological uncertainty under differing levels of resource application. The higher the level of resource application for a given delivery urgency and technological uncertainty, the lower the likelihood of disruption. A number of examples abound where alternative

designs are produced in parallel to reduce the chance of failure. Expanding capacity, adding personnel or increasing the level of testing and evaluation are other examples of resource expenditures used to reduce uncertainty. With regard to technological uncertainty, Duvivier [15] recommends the use of technological forecasting to assess the risk in meeting the demand for increasingly advanced technology. He postulates that advances are extrapolations of current knowledge and that breakthroughs are rare. Even when breakthroughs do occur, such as the laser, it takes eight to twelve years to incorporate them in new systems. He shows examples of engine weight, lift and fuel consumption all following smooth curves, Thus. the cost and benefit of new technologies can be based on an extrapolation of technology growth curves.

UNCERTAINTY VS PHASES HIGH

.J

LOW PROGRAM PHASES

FIG.

9.

STRETCH OUT -

hnpact O/'streteh-out (adaptedfrom

[2]).

Route, Somer.~--Pertbrmance in Major Program Acquisition

168

NIGH CONCURRENCY

COMPLEX Fr~ r ~J I j.{ LL

MODERATE CONCURRENCY

O I

CONCURRENCY

hi F-

SIMPLE FEW (low Irnpccf)

MANY {high impoct}

INTERDEPENDENCIES IN COMPONENTS

FIG. 10. Relationship O]'state-of'the'art to concurrency.

Regression models have contributed significantly to the understanding of causality in the acquisition process. For example. Leech & Earthrowe [25] have shown that the ratio of actual costs to estimated expenditures can be predicted based on a regression with actual size of the job. Using a sample of 64 jobs, they developed a regression curve, where r = 0.955 + 0.0092( and X = actual job size in man-hours. As they point out. in every case a commitment was made to the customer based on an initial design. However. where the job is large and requires considerable technical innovation, or where the quantity ordered is small (no opportunity for learning), the design and development costs contribute significantly to the final cost. They recommend an investment portfolio approach to minimize the risk associated with design uncertainty. New technologies are seldom used for an entire system design. Rather, they represent a

small percentage of all the subsystems and components. Where the manager maintains control over those components which utilize new technology, he is in a better position to effect the reduction of cost overruns. The use of the Pareto law as a basis for determining which components contribute most to technological uncertainty is shown in Fig. 12. Point C on the absicissa represents the subset of components that contribute most to technological uncertainty in terms of system impact. Typically, 20~ of the components contribute 80~0 of the problems encountered. The Pareto law can be used as a basis for parametric estimating and provides a useful tool to control technological uncertainty. C O M P U T E R A P P R O A C H E S TO ACQUISITION M O D E L I N G This portion of the paper examines specific computer models that have been used in the acquisition process. The selection of the models W Z

T ~' Z

I00

0 l,--- 8O

g

W L9

J C I

roj ~d FIG. I 1, Expected disruption.

20

I

iOO

PERCENTAGE OF" COMPONENTS

FIG. 12. Pareto law.

Ome~.a. Vol. I1. ),%. 2

chosen for analysis is not intended to be exhaustive, but rather indicative of what has been used or is proposed for use. For our purpose the types of models will be grouped into two major categories: (1) probabilistic stochastic models and 12) simulation models. The extension of the two groups of models leads to a proposed approach of this paper--the causal-integrative model (CIMI--which deals with factors beyond the capability' of many of the current models. 1. One approach that has gained considerable acceptance as a basis for estimating costs is parametric cost estimation, which uses performance variables such as speed, weight, range, power, etc. to predict costs, because estimates of these parameters are usually known early in the design phase. These estimates are based on historical data of previous or similar systems and utilize statistical relationships between cost and perfbrmance parameters of these systems. 2. A second approach to risk analysis is a model developed by Freeman [17] which allows various alternatives or systems to be objectively compared based on aggregate risk assessment. The process begins by segmentation of the various program functions into categories reflecting the schedule, cost and performance variables. Risk distributions, represented by utility functions, are used to determine utility values as each of the variables are changed. Following this is the development of a risk matrix where the options (or alternative systems) are compared with the criteria used for choice. The summary risk or probability tbr each system/alternative can then be determined on a quantitative basis. Computer-based simulation models are used to analyze complex systems. Simulation models are based on the following premises: --management can be treated as information feedback systems; --intuitive judgments are generally unreliable regarding how these systems will change, even when knowledge of individual subsystems is available; --sufficient inlbrmation is available for experimental model-building;

169

- - t h e main structure of controlling policies and decision streams of an organization can be represented by the simulation model: - - t h e model is structured so that external forces interact with internal decisions: --policy' and structure changes can be examined in terms of economic behavior tbr describing system performance. A main advantage of computer simulation is that it forces managers to tbrmatize the decision-making process which, in turn, can aid in gaining insights into program performance.

Causal-integratice model ( CIM) Figure 2, as noted previously, suggested that the acquisition management process can be described by the four factors: organizational slack, technological uncertainty, environmental uncertainty and customer urgency. Although the graphic does not reflect the dynamics and interactions that occur during acquisition, it does illustrate the static linkages. A system of exterior linkages is needed to give a greater degree of representation of the acquisition process. Expansion of Fig. 2 to include the internal and external linkages in the process is the basis for the development of the CIM. An extension of the computer simulation approach is the CIM shown in Fig. 13. This model describes the processes, flows, variables, feedback loops, delays, exogenous variables and key decisions related to the four basic variables shown in Fig. 2. As noted earlier, acquisition models currently being used do not address all of these variables and thus each model lacks some degree of completeness. Referring to Fig. 13, the CIM can be used, for example, to determine how a change in economic uncertainty affects the level of environmental uncertainty which, in turn, affects mission, scope and funding. These changes perturb the system to effect changes in the other variables through the pervasive network of interdependencies. Changes in a key variable thus have an impact on the acquisition cycle in ways that are not intuitively obvious without the aid of a dynamic model to point out the causal relationships.

@ ~L

r

I ~J

F

I

i

J

__

I

J ! i

i

7

~

I

®

F

o©

Ome'.,a. Vol. I I. 3,'o. 2

Representative module o/ ttle CIM To illustrate the CIM. a modular approach is used which involves developing one module at a time and interfacing it with the remainder of the model. After the first module is developed and the interfaces are validated, a second module is developed and integrated into the expanded model. The interfaces of the two modules are tested for proper operation with the same approach used for the single module. This process is repeated until all the modules are completely integrated into one model with many submodels or modules. Data from projects can then be used to test the final model for variations in predicted values. The aggregate variable organizational slack (OS) can be used to demonstrate this modular approach. From Fig. 13, the subvariables for organizational slack are those shown in Table 5 along with measures used to quantify these subvariables. As an initial step, the development of the OS module starts with a system dynamics representation of the events and processes that make up this aggregate variable. Once this step, and the programming effort, is completed, test data for the subvariables are used to check the module operation. Data inputs from outside of the module can be of either table "look-up' type or of" functional relationships (curves). Thus, when the module 'needs' external values, they are developed from computations utilizing the functional relationships or from a data matrix. With program test data in the module and empirical data used in the rest of the CIM, the output variables can then be observed as functions of changes in the organizational slack module. For example, output curves of cost performance can be generated by varying the level of resource allocated (REAL). Sensitivity

171

studies could be made regarding cost performance and the impact of variations in the organizational slack subvariables on this output performance measure. The direction proposed by this approach includes the following: - - d e v e l o p m e n t of a complete computer-based model: --testing of the model using data from the completed program; --validation of the model using current programs; --implementing the model for policy level decisions in acquisition management. The direction of research in acquisition management utilizing this tool has progressed to the point that: --testing of the model using completed programs can be done; --validation of the model with current programs can be examined; --implementation of the model for policy level decisions in acquisition management is feasible. CONCLUSION The material presented in this paper has attempted to highlight important advances that have been made in improving the acquisition process. Because of the pervasiveness of the

TABLE 5. SUBVARIABLF_SFOR ORGANIZATIONAL SLACK

I. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

Organizational capacity (ORCA)--people. experience levels. Organizational demand (ORDEr---hours required for task. Subcontract slippages (SUSl,b---percentage delay time. Change in scope (CHSC)--variation from contract, percentage change. Level of resources allocated (REAL)---people, budget levels. Learning curve (LECUI---rate of learning effects. Key personnel turnover (KPTOI---percentagechanges. Key equipment delays {DEKE)--percentage time. Subcontractor GFE delays (DESE)--percentage time. Changes in technological uncertainty (CHTU)--state-of-the-art advances, complexity, number of components. Changes in rate quantity (CHRQ)---production required concurrency. Level of competition (LECO)---price levels in real percentages.

1"2

R:,~te, Sortter~--Per/brmance in Major Program .qcquisitton

subject, of necessity not all relevant research or applications could be included. Rather, what has been presented here can be considered as indicative of the current state-of-the-art in acquisition management and a baseline approach for future developments. At the outset, the paper has emphasized the need for a causal basis for understanding the factors that affect cost overruns. A number of examples were presented that clearly illustrated that cost growth is a phenomenon related to the acquisition of complex projects, both civilian and military: and that four primary variables contribute to cost: environmental uncertainty, technological uncertainty, customer urgency and organizational slack. It was pointed out that control, as currently practiced, is not sufficient to avoid cost overruns. Prior research was reviewed which identified reasons for cost overruns, including an extensive study done at the University of Southern California which identified 26 contributing factors, among which the key contributors were the four primary variables in the acquisition process described above. Having established a basis for understanding why cost overruns occur, the report dealt with the risk and uncertainty aspects of the problem. An attempt was made to define the sometimes abstruse terminology used in the field. This material provided a foundation for the section on a causal basis for defining uncertainty. A number of studies were presented to help understand what causes uncertainty and how to approach it in the acquisition process, For example, it was pointed out that uncertainty and disruption cannot be eliminated, but that it can be controlled if there are causal models such as the R A N D study relating cost to advance in state-of-the-art. Based on the material presented, a set of causal relations among variables in disruption and uncertainty were examined in order to establish a "pattern of disruption'. This was followed by the section on current approaches to acquisition modeling, including ones used for risk analysis. Finally, a causal-integrative model was presented which illustrates the complex relationships that exist among the variables affecting the acquisition process. Although this is a preliminary model, it provides a basis for integrating the approaches to date to managing the acquisition process. It includes many causal

sub-models, such as concurrency, learning curve, disruption, etc. and also covers the dynamic interdependencies that exist and the treatment of risk and uncertainty as integral parts of the model. Acquisition managers utilizing more sophisticated tools can improve the effectiveness of their performance and thereby achieve the maximum potential cost control. The causal approach described in this report offers the potential for achieving this goal.

REFERENCES I. ALLEN WH (1972) Should cost ',,,ill cost must cost: a theory on the cause of cost grov,th. Office of the Assistant for Cost Anal',sis. US Arm..~ Safeguard System Office, Arlington. Virginia, June. 2. AUGUSTINE NR (1979) Augustine's laws and major system development. Dql~'nsc Sv~'lclrts ;$[Wnt Re~'. Spring, 50--76. 3. BEW.RLV JG, BONELLO FJ & DA','LSO>,Wl 11981) Economic and financial perspectives on uncertainty in aerospace contracting. Paper presented at 2nd symposium on management of risk and uncertainty in the acquisition of major programs. USAF Academ~, Colorado, February. 4. BRowx RA (1978) Probabilistic models of project management with design implications. IEEE Trans. Engng ~[~mt EM-25, May. pp. 43-49. 5. BURT JM (1977) Planning and dynamic controt of projects under uncertainty. Mgmt Sci. 24, pp. 249-258. 6. CHACO,taN CB (1979) Large engineering project risk analysis. IEEE Trans. Enffng 3lgmt E3,1-26, pp. 78-86. 7. COCHRAN EB, PATZ AL & ROWE AJ 11978) Concurrency and disruption in new product innovation. Calif Mgmt Rec. XX! (1), 21-33. 8. COCHRAX EB & ROWE AJ 11978) The sources of disruption to project cost and delivery performance. Prac. 8th DoD Symposium on Acquisition Research. West Point, New York. 9. COCHRAN EB (1980) Measuring and predicting production disruption costs due to design uncertainty and delivery urgency. Proc. 9th Annual DoD FA1 Acquisilion Research Symposium. Annapolis. Marl, land, June. 10. COCHRAX" EB (1980) Unpub. manuscript supplied to authors. 11. COYLE RG (1977) Management S.vstem Dynamics. John Wiley, London. 12. DAVIS GW (1976) The dilemma of uncertainties associated with cost estimating in the pro.i~t management office. Report LD-36420A, Defense S~stems Management College, Fort Belvoir. Virginia. 13. DIEXEr,tAXN PF (1966) Estimating cost uncertaint.~ using Monte Carlo techniques. RM-485.4-PR. RAND Corporation, Santa Monica. California. 14. DRAKE HB (1970) Major DoD procurements at war with reality. Hart'. Bus. Rev. 48(-II. 119-140. 15. DuvtvlE~t G F (1971) Technological ad:ances and program risks. Technol. Forecasting & Social Change 2(3,4), 277-287. 16. FORR~TER JW (1961) hl&~strial Dynamics. M IT Press, Cambridge, Massachusetts. I7. FREE~,t,',N RG (1979) The basic approach to the model was presented at the Conl~'rence on .~[anagement oJ" Uncertainty in ,!,[ajor Progrant Acquisitions. University of Southern California, February.

Omega, Vol. 11, .Vo. 2 18. GR.-'CfSONCJ JR {1960} Decisions Under Uncertainty. Har',ard University. Boston. Massachusetts. 19. H,,~E EJ (1977) How useful are historical costs? Air Force Conptroller Jul',. pp. 34-36. 20. HARMAN AJ (1970) A methodolog~ for cost factor comparison and prediction. RM-6269-ARPA. RAND Corporation, Santa Monica, California. 2 [. HARMANAJ ( 1975 ) Measurement of technological innovation by firms. P-5496. RAND Corporation, Santa Monica, Calilbrnia. 22. HARg.lSON EF (1975) The Managerial Decision-making Process. Houghton Miffin, Boston, Massachusetts. 23. JONESH & Twlss BC (1978) Forecasting Technologyjbr Planning Decisions. Petrocelli Books, New York. 24. LARGEJP ([974) Bias in initial cost estimates: how low estimates can increase the cost of acquiring weapon systems. R-1467-PA'E, RAND Corporation, Santa Monica. California. 25. LEECH DJ & EARTHROWE DL (1972) Predicting design costs. Aeronaut. Jl September. 26. LOCHRYRR, FLAMMERP, SMITH DR. HEAD RG, HENRY I'M, HUDSON WR, NELSON" EB, CARSON GG & Gt,'ILMARTIN JF (1971) Final report USAF Academy risk analysis study team. US Air Force Academy, Colorado. 27. MARKS KE (t978) An appraisal of models used in lit'e cycle cost estimz,tion for USAF ~,ircraft systems. R-2287-AF. RAND Corporation. Santa Monica, California. 28. MARSCHAK TA, GLENNON TK. SUMMERS R (1967) Strategy for R&D: studies in the microeconomics of development. RAND Corporation, Santa Monica, California. Pub. Springer, New York. 29. MARTINMD, GLOVERWL & LENZJO (1977) The status on uncertainty measurement for the development contract. Nat. Estimating Soc. Jl, Fall. 30. McN~CHOLSGR (1976) On the treatment of uncertainty in parametric costing. Unpub. PhD dissertation, George Washington University, Washington. DC. 31. MERROWEW (1979) A review of cost estimation in new technologies: implications for energy process plants. R-2481 DOE, RAND Corporation, Santa Monica, California. 32. MOELLER WG {1980) Affordability for major system acquisitions. Logistics Management Institute, Washington. DC. 33. OSTW-~LD PF (1974) Cost £~'timating JOT Engineering and Management. Prentice-Hall, Englewood Cliffs, New Jersey. 34. PECK MJ & SCHERER FM (1962) The weapons acquisition process: an economic analysis. Division of Research, Grad. Sch. Bus. Admin., Harvard University, Boston, Massachusetts. 35. PERRY R et at. (1971) Systems acquisition strategies.

36.

37.

38. 39.

40.

41.

42.

43.

44.

45.

46. 47.

48.

173

R-733-PR ARPA. RAND Corporation. Santa Monica. California. l:~v_,ssSJ (19751 Methodology for subjective assessment of technological assessment. R-1375, RAND Corporation, Santa Monica, California. ROESCH N & SAGE AP {1980) Systems engineering methodology for defense system acquisition. Proc. 9th Annual DoD FAI Acquisition Research Symposium. Annapolis. Maryland. June. ROWE AJ (principal investigator) {1977) Disruption research project. Unpub. paper. University of Southern California. April. ROWE AJ & SOMERSIA (1982) Methods to predict cost overruns in the acquisition process. Federal acquisition research symposium, Washington, DC. Ma). SIMPSON FS (1968) Measurement of technical performance in weapon system development programs: a subjective probability approach. RM-5207-ARPA. RAND Corporation, Santa Monica, California. STANLEY WL & MILLER MD (1979) Measuring technolo~cal change in jet fighter aircraft. R-2249-AF, RAND Corporation, Santa Monica, California, SU~,tMERSR (1965) Cost estimates as predictors of actual weapons costs: a study of major hardware articles. RM-3061-PR. RAND Corporation, Santa Monica. California. US Congress (1979) Inaccuracy of Department o[" Defense weapons acquisition cost estimates (report of the Committee, 96th Congress. Ist session, November 16). House Report 96-656. US Government Printing Office, Washington, DC. US General Accounting Office (1979) A range of cost measuring risk and uncertainty in major programs--an aid to decision-making. PSAD-78-12, Washington, DC. US General Accounting Office (I 979) Financial status of major federal acquisitions September 30, 1978. PSAD-79-14. Washington, DC. US General Accounting Office (1979) Impediments to reducing the costs of weapons systems. PSAD-80-6, Washington, DC. US General Accounting Office (I 980) Issues identified in 21 recently published major weapon system reports. PSAD-80--43, Washington, DC. US Gocernment Office Management and Budget (1976) Major system acquisitions. OMB circular A-109. Washington. DC.

CORRE,SPONDENCE: ProJessor AJ Rowe, Department of Management and Policy Sciences. School of Business Administration, University of Southern California. Los Angeles, CA 90007. USA.

ADDRESS FOR