- Email: [email protected]
Software cost modeling: Some lessons learned
Software
Cost Modeling:
B. W. Boehm
and R. W. Wolverton
This paper summarizes certain lessons we learned recently in developing a software cost estimation model for TRW. With respect to general needs in the cost estimation field, it was found particularly important and useful to address the following three needs: to develop a set of well-defined, agreed-on criteria for the “goodness” of a software cost model; to evaluate existing and future models with respect to these criteria; to emphasize “constructive” models that relate their cost estimates to actual software phenomenology and project dynamics. This paper presents a set of criteria found to be particularly Important in practical software cost estimation, together with examples of their importance.
Some Lessons Learned
Con.st,‘lrc.til.ene.ss. Can a user tell why the model gives
the estimates it does‘? Does it help one understand the software job to be done? Detail.
Does the model easily accommodate the estimation of a software system consisting of a number of subsystems and units‘? Does it give (accurate) phase and activity breakdowns?
Stubilif_v.
Do small differences in inputs produce small differences in output cost estimates?
Scope. Does the model cover the class of software projects whose costs you need to estimate’! Errsr c?f’ USC. Are the model
to understand
inputs
and options
easy
and specify?
P~o.sprc.ti1,~nrs.s. Does the model avoid the use of in-
CRITERIA FOR THE GOODNESS SOFTWARE COST MODEL
formation that will not be well known project is complete‘?’
OF A
Dc$/lition.
Has the model clearly defined which costs it is estimating, and which costs it is excluding‘?
Are the estimates close to the actual expended on the projects?
Objec~tilYt>~. Does the model
the
P~~~:sitnony. Does the model
Our initial set of criteria reflected our primary objective in developing the new TRW Software Cost Estimating Program (SCEP): to serve as an aid in developing cost estimates for competitive proposals on large government software projects. Other objectives for developing a cost model-e.g., to evaluate the impact of using new techniques-may require a somewhat different set of criteria. We found the following list of criteria important in our context:
Fidrlity.
until
costs
avoid allocating most of the software cost variance to poorly calibrated subjective factors (e.g., complexity)? That is, is it hard to jigger the model to obtain any result you want‘?
redundant factors, ciable contribution
avoid the use of highly or factors that make no appreto the results?
In the process of developing the TRW SCEP model and evaluating existing models [ I- IO]. we found each of these criteria important in terms of lessons learned involving software cost modeling. Results ofour evaluation and model development with respect to these criteria are described in the following section. EXAMPLES
OF THE UTILITY
OF THE CRITERIA
Definition. One of the first questions a proposal manager asks whenever we run a software cost model to support a cost proposal is, “Does this estimate include the cost of management? Requirements analysis? Training‘? Computer operations?” Surprisingly, the documentation for most existing cost models does not answer these questions satisfactorily. For the
‘Clearly, this criterion is specific to the objective of cost ,wc’tlkfio~~. For other objectives, such as technology impact assessment. a retrospective model could be appropriate.
The Journal of Systems and Sofware I 195-201 (1980) ti” Elsevie~ North Holland. Inc.. 1980
B. W. Boehm and R. W. Wolverton
196
SUBSYSTEM . * l
I SUBSYSTEM
SSX ACTIVITIES
SXl SSX
CODE AN0 UNIT TEST
l
SX411 UNIT TEST PLAN
PRE-ACCEPT.
TEST
MGMT.
SX4121
PLANS
SX4122
PROCEDURES
SX4123
INTEGRATION
SX4124
TEST
5X4125
REPORTS
l
SX413 ACCEPT. TEST
8X222 CONTRACT
SX2111
FCNL. ROTS.
:
1X223 END ITEM ACC. PLAN
SX4131
PLANS
SX2112
PERF. RQT?..
:
SX224 CUSTOMER
5X4132
PROCEDURES
SX2113
I/F RQTS.
:
SX225 OFFSITE
SX4133
ACC. TEST
SX2114
TEST RQTS.
:
SX226 SUBCONTRACT
SX4134
REPORTS
SX2115
RQTS.VAL.
SX212
S/W SSX DESIGN
SX2121
ARCH. DESIGN
$X2122
IIF DESIGN
SX2123
DB DESIGN
SX2124
DES. VEA.
SX2125
PDR
SX2126
CDR
MGMT.
I/F
OFF. MGMT. t&GMT.
$X227 DES. REVIEW BD.
SX414 TEST BEDS
SX228 TEST REVIEW
SX415 TEST TOOLS
BD.
: . . . l
SX213 PROJ. STDS. SX214 USER MANUALS MGMT.
SX2151
CCB
SX2151
CM0
. . ..**.*.**.~*.*~*~.~..**
SX2153
PROG. SUP. LIB.
: INCLUDED *IN STANDARD ~S~FTWARE ~0s~
SX217 STUDIES
Figure 1. Work hierarchy.
& ANAL
breakdown
.. .. .
l
l
l
SX216 OUAL ASSUR
* .
,............*........**.... :. SX42 POST-ACCEPT. TEST .. .. SX421 TECH. EVAL. SUPP. .
.
SX215 CONFIG.
l
SX416 TEST DATA
SX229 AUDITS
:
: i .. : i ‘ * i .. i . . l
i
l
. . . . : .
DOC’N
S/W RQTS.
i * SX2116 SRR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
: . . :
SX41
SX33
SX221 COSTISCHIPERF. SX211
: l * . . . . : .
ENGR. DATA MGMT. DATA
SX412 INTEG. & TEST
.I.*..~~.*...*~~~***~.**~*.***.
: l * . . . . . . .
SX31 SX32
ESTIMATES
.*.......................
structure
(WBS)
.. .. . ... .. .. :. .. l
. . :
+ .
activity
TRW SCEP model, we found that the best solution was to define a standard work breakdown structure (Figure 1) and use it to define those costs that were included in our estimates and those that were excluded. We found that some existing models seemed to work well for some classes of software and not very well for others. For example, Figure 2 presents the results of our analysis of the Boeing [ 91 and Putnam [ 101 estimation models when applied to their own (Boeing and U.S. Army Computer Systems Command) data. The Boeing model appears to estimate small projects reasonably well, but gives extreme overestimates on large projects; the Putnam 1978 model does just the reverse.2 It would be highly Fidelity.
Tutnam has recently developed a new model called SLIM which evidently eliminates this problem.
l
SX422 OPER. EVAL.
SUPP.
SX5 TRAINING SX6l
CONVERSION
SX62
INSTALLATION
t*..***.**....* :. NOT INCLUDED :. ...............
valuable for the field if similar information were available with respect to other models and other software project attributes. For comparison, Figure 3 indicates the performance of the TRW SCEP model on 20 completed TRW projects. The sequence of activities in developing, calibrating, and evaluating the model was to: 1. Survey the current state of the art in cost estimation. Table 1 shows one of the result of the survey, a summa~ of the factors used in the major current cost estimation models. 2. Develop a baseline model. The phase-sensitive nature of the model is based on a model developed earlier by the authors to evaluate the cost impact of new software technologies; it is functionally similar to the Boeing model in this regard. 3. Refine the model parameters via a two-round Delphi exercise involving 10 experienced TRW line and project managers. 4. Calibrate the model parameters using an initial sample of seven completed projects. 5. Evaluate the model using a sample of 20 completed
197
Table 1. Factors Used in Various Cost Models GROUP
SDC
FACTOR
TRW 72
65 SIZE
SOURCE
PUTNAM 78
X
NO.
ROUTINES
X
NO.
DATA
NO.
REPORTS
RCA
DOTY
IBM
X
X
INST. INST.
OBJECT
T
X
X
X X
ITEMS
X
COMPLEXITY
X
X
X
X
X
LANGUAGE
X
RE-USE
X -__
____ HARDWARt ATTRIB
TIME
H/W
CONSTRAINT
H/W
PROJECT
PERSONNEL
ATTRIB.
PERS. CONTINUITY
DEV.
TOOLS
X
X
X
___ X X X X _X X
X X
X
X
X
~-
c
X
X
X X
X
X
EXPER.
LANGUAGE
X
X
QUALITY
EXPERIENCE
APPLIC.
X
X
CONFIGURATION
CONCURRENT
H/W
X
CONSTRAINT
STORAGE
L / t
X X
X
X
ATTRIB.
X
X
NO. PERSONNEL TYPE
[
X
X
DOCUMENTATION
PROGRAM
TRW 78
BOEING
X X
X
X
EXPER. X
&TECHNIQUES
ENVIR.
CUSTOMER
ATTRIB.
ROTS.
DEFINITION
ROTS.
VOLATILITY
X
I/F
X
X X X
SCHEDULE SECURITY COMPUTER
X
ACCESS
X
REQUIRED
X
X
TRAVEL/REHOSTING QUALITY
7 RESULTS
A
ON BOEING
PUTNAM
1978 RESULTS
USA-CSC
PROJECTS
PROJECTS
ON
OD
6
0
A
A
Figure 2. Estimator Boeing and Putnam
fidelity vs project 1978 models.
size:
E
CARMOCS
aCOMIS
t E F f
A MPAS
*ACS
A SAAS 2
aMPMIS ,, COSCOM
I
5
10
20
50
loo ACTUAL ACTUAL
200
500
MAN-MONTHS
0
MAN-YEARS
a
1000
2000
B. W. Boehm and R. W. Wolverton
PROJECT
SIZE:
MEDIUM
TO VERY
PROJECT
TYPE:
C3, AVIONICS,
ACTUAL
Figure 3. Example performance.
of
the model produce any cost estimate desired, simply by modifying the subjective complexity factor. (In fact, PRICE S has a mode called ECIRP that will do this for you automatically.) Although modifying complexity may solve a user’s short-term pricing problem, it does not produce any objective and meaningful cost estimation function for the user. In developing the TRW SCEP model, we found that although we were unable to avoid including a complexity factor, we have made the complexity rating an attribute of each individual unit in the software, and have provided users with a set of scales for calibrating the complexity of different types of unit. Table 2 shows the complexity scale for computational/numerical analysis type units as an example: such a scale makes the complexity rating a much more objective, verifiable attribute.
LARGE
SENSOR,
ANALYTIC,
SUPPORT
MAN-MONTHS
TRW
software
cost
model
projects, including the initial seven. This evaluation was done by an independent third party. The high value of R2, the square of the correlation coefficient, should be interpreted with considerable care. For example, the sample consists of only medium to very large TRW government contract software efforts; we are not sure how it performs on other types of software effort. Further, the calibration is on past projects. We have included a factor to cover the impact of software technology improvements; however, until we get more experience in comparing SCEP estimates with the resulting project actuals, we must consider the model still in an experimental state.
Constructiveness. We use the TRW SCEP model as a means of cross-checking separate cost estimates developed by project personnel in different ways. Inevitably, this leads to differences between the various estimates, and a need to understand why one was higher or lower than the other. Similarly, project personnel want to know in project terms why different factor ratings give them different cost estimates. To help answer these questions, and to promote accurate factor ratings by users, we have provided a
Figure 4. Effect of complexity model (September 1977 version). 260,
factor
I
in RCA PRICE
S
I
I
Objectivity. Figure 4 shows one of the results of our analysis of the RCA PRICE S model in September 1977,” indicating the extreme sensitivity of the 1977 model to the subjective complexity (SDCPLX) factor. If a project may be described as “hard,” the model will produce a cost estimate that is six to seven times higher than an “easy” project; this is a huge source of variation for a parameter that is entirely subjective. Experience indicates that this means a user can make
“Subsequently, PRICE S has incorporated a set ofguidelines for using and adjusting the complexity factor which reduce the magnitude of this problem.
-0
1.0
2.0 PLATFORM.
PLTFM
Software
Cost Modeling
199 Table 2. Example of Complexity 1
VERY LOW
COMPUTATIONS EXPRESSIONS:
2
LOW
USE OF STANDARD MATH AND STATISTICAL ROUTINES AND BASIC MATRIX-VECTOR OPERATIONS
3
AVERAGE
USE OF BASIC NUMERICAL ANALYSIS SKILLS E.G., MULTIVARIATE INTERPOLATION ORDINARY DIFFERENTIAL EQUATIONS
4
HIGH
DIFFICULT BUT STRUCTURED NUMERICAL ANALYSIS E.G., NEAR-SINGULAR MATRIX EQUATIONS, PARTIAL DIFFERENTIAL EQUATIONS
5
VERY
*SPECIAL
OPTION
HIGH
Table 3. Example of Factor/Rating
I. REc3”IRED OVALITY
LITTLE
LITTLE
UNSTRUCTURED NUMERICAL E.G., FAST, HIGHLY ACCURATE OF NOISY, STOCHASTIC DATA
ment by a certain factor, which may vary by phase. Table 3 thus tells the user why costs have increased, in terms of the impact of a very high required quality on the project activities. In general, we have found that such tables do help the users of the cost model better understand the software job for which they are preparing.
Definitions: TRW Cost Model
REOUIREMENTS ANALYSIS
MANY
DIFFICULT, ANALYSIS ANALYSIS
TO EVALUATE SIMPLE A = B + C * (D - E)
AVAILABLE.
table for each of the factors in the TRW SCEP model indicating the impact on project activities of the various factors and their ratings. For example, the table for the “required quality” factor is shown as Table 3. When a system is rated as having a “very high” required quality, the SCEP model will increase the cost of performing the various phases of system develop-
FACTOR
Scale: TRW Cost Model
DETAIL TBD’S “ALID’N
PRELlMlNARY DESIGN
LITTLE MANY
DETAIL TBD’S
DESIGN
INFO.
MINIMAL
UOF’S
CODE & UNIT
MlNlMAL
TEST
INTEGRATION AND TEST
MINIMAL
U.T. PLAN
TEST PLANS
NO TEST PROCEDURES
MINIMAL
OA. CM
MINIMAL
DA. CM. STDS.
MINIMAL
CDR
MlNlMAL
OA. CM
MlNlMAL
OA, CM
MlNlMAL
POR
NO DRAFT
MINIMAL NOMINAL
I/O AND OFFTESTS
MINIMAL NOMINAL
STRESS,OFFTESTS
MlNlMAL
USER MAN
MINIMAL DOCU.
AS-BUILT
LlTTLE
BAS,C
“ERIF’N
INFO.
VERIF’N TED’S
BASIC DA.. CM. STOS BASIC
PDR
MlNlMAL
FULL
USER MAN.
USER MAN
FREDUENT
MODERATE
DETAIL
BAS,C “DF’S. CM. CDR MlNlMAL
PA,
USER MAN
TRW POLlCfES
BASIC
BASIC
OA. CM. U.M.
FULL
TRW POLICIES
DETAILED “ERIF’N DA. CM. STDS. CDR DOC’N
FULL
I/O AND TESTS
MANY
RDTS.
UNTESTED
BASlC TEST PLANS
U.T. PLAN
PARTlAL FCL’S, PATH TEST, STDS CHECK
PARTIAL NOMINAL
USER MAN.
DETAILED VERIF’N. OA, CM, STOS. PDR, OOC’N
I”&” INTERFACE ISUPPORT. RESPONSE)
BASIC
DESIGN
MlNlMAL FCL’S. PATH TEST, STDS CHECK
NO DRAFT
DETAILED “ALID’N. PLANS. SRR
DETAIL
OFF-
MINIMAL FREQUENT UNTESTED
ROTS
BASIC OA. CM, U M. PARTIAL NOMINAL
TRW POLICIES
TEST PROCED.
FULL
STRESS, TESTS
OFF
TRW POLICIES
DETAILED U.T. PLAN, FCL’S, OA. CM. DOC’N CODE WALKTHRUS
DETAlLED TEST PLANS. PROCEDURES. OA, CM, DOC’N
EXTENSIVE OFFNOMINAL TESTS
EXTENSlVE STRESS, OFF-NOMINAL TESTS
DETAILED “ERIF’N OA. CM. STDS, PDR. DOC’N
DETAILED VERIF’N. DA, CM. STOS. CDR. DOC’N
DETAILED U.T. PLAN, (FCL’S. OA, CM. DOC’NI CODE WALKTHRUS
VERY DETAILED TEST PLANS. PRDC’S, OA, CM. DOC’N
IV&”
IV&V
VERY EXTENSlVE NOMlNAL TESTS
VERY EXTENSlVE STRESS.OFF-NOMINAL TESTS
,NTERFACE
,NTERFACE
IV&V
INTERFACE
OFF-
IV&”
INTERFACE
B. W. Boehm and R. W. Wolverton
200 Detail. We have also found from experience that models requiring greater detail tend to produce more accurate estimates, mainly because:
the gathering of greater detail tends to increase people’s understanding of the job to be done; and ifthe added detail results in the overall estimate being the sum of some smaller individual estimates, the law of large numbers tends to work to decrease the variance in the estimate. Stability. Figure 5 shows one result ofour analysis of the Doty model [7,8] ; it has a severe discontinuity in the neighborhood of 10,000 source instructions. In that neighborhood, small differences in sizing can lead to large differences in estimated cost. Such a lack of stability is an important thing to know about a cost model. Scope. In our comparative cost model analysis to date, we have found that some existing models seems to work well for some classes of software and not very well for others. For example, we have found that the TRW SCEP model overestimates costs on projects that are less than 5 person-years in total effort, indicating that one of our linearity assumptions holds only
Figure 5. Doty model:
suggested
estimating
relationships.
over a certain range of project sizes. It is currently well calibrated over the range 60-2000 m&n-months. Ease of use. A cost model will be less useful in general to the extent that it relies on inputs that are difficult to understand and obtain, or to the extent that it makes it difficult for the user to perform common cost estimation functions such as the combination of the estimates of two subsystems into an estimate for the overall system. (We have encountered the latter difficulty on several existing models.)
One current problem with the use of Halstead model [ 121is that it relies on detailed properties (i.e., number of operaoperands) are generally unknown at the time estimate is needed.
Prospectiveness.
practical knowing tors and the cost
Parsimony. A model will be less useful to the extent that it requires a number of inputs where one would suffice. An example is the Walston-Felix model [S] which has separate entries for the use of different modern programming practices (MPPs) such as top-down development, structured code, and use of inspections. For the original research purpose of the Walston-Felix model, it was important to distinguish these entries, but for practical estimation of projects using modern management guidelines, a single “use of MPPs” factor would suffice.
CONCLUSIONS 500 -
We have found the 10 criteria discussed above extremely valuable in evaluating alternative software cost estimation methods and in developing the TRW SCEP model, which has been used routinely over the past 18 months in estimating and tracking the costs of over $100 million worth of software on over 50 projects and proposals. We have particularly found that its constructive nature has helped our staff (both hardware and software people) become more knowledgeable about and sensitive to software cost drivers.
P .” / / .’ .’ .’ .‘ .’
REFERENCES lo/ 5-
/’
/
/’
-.-NOMINAL
/’ ---NOMINAL
SOURCE INSTRUCTIONS
OR
1. E. A. Nelson, Management Handbook for the Estimation of Computer Progrumming Costs, SDC, ADA648750, 31 October 1966. 2. R. W. Wolverton, The Cost of Developing Large-Scale Software, IEEE Trmns. otz Comprrters (June), 615-636 (1974). 3. L. H. Putnam, A General Solution to the Macro Software Sizing and Estimating Problem, IEEE Trans. Software Engineering (July), 345-361 (1978).
Software Cost Modeling
201
4. J. K. Herd. J. N. Postak. W.E. Russell, and K. R. Stewart, .Yofr~~uwCost Es timntion Study-Study Results, Doty Associates, Inc., Final Technical Report, RADCTR-77-220. Vol. 1 (of two). June 1977: NTIS No. ADA042264. S. D. L. Doty, P. J. Nelson, and K. R. Stewart, Soft~twc, Cost Estimcttion
Strrc!\.--Glricl’elinesJi,r
Imptwled
Estimclti,zg , Doty Associates,
Soj-
Inc., Final Technical Report, RADC-TR-77-220, Vol. II (of two), August 1977: NTIS No. AD-A044609: see also errata sheet, J. H. Herd, Doty Associates, Inc., 416 Hungerford Dr., Rockville, MD 20850, 6 June 1978. 6. John Schneider, IV. ,4 Prdiminruy Culihrution of the RCA PRICED Softnuw Cost Estimcction MO&~, NTIS No. .4D-A046808, September 1977. 7. J. D. Aron, Estimcrtiq~ Resourcesfor Lurge Programnring Jystem~ , NATO Science Committee, Rome, Italy. October 1969. u‘(wc
Cost
8. C. P. Felix and C. E. Walston. A Method of Programming Measurement and Estimation, IBM s\.stc,m.\ .Iorrrnu/ 16 ( 1) ( 1977). 9. R. K. D. Black, R. P. Curnow. R. Katz, and M. D. Gray, BCS Sqfhure Protlrlctiwr Drlttr , Boeing Computer Services, Inc., Final Technical Report, KADCTR-77-116. March 1977: NTIS No. AD-A039852. 10. L. H. Putnam and R. W. Wolverton. “Quantitative Management: Software Cost Estimating. in lEEE Camp.
Sot.
Firxt
Int.
Computc,r
S(!/hwrc~
rrntl Appli-
(COMPSAC 77). IEEE Cat. No. EHO 129-7. Chicago, 8-11 November 1977, 326 pp. 11. B. W. Boehm, C. A. Bosch, A. S. Liddle, and R. W. Wolverton, The Impact of New Technologies on Software Configuration Management, TRW Report to USAF-ESD, Contract Fl%28-74-C-0154. IO June 1974. 12. M. H. Halstead, Elements c!f’Sc?fi,l~are Scirncc.. Elsevier North Holland, 1977. (utiouJ
Cor$
Cost Modeling:
B. W. Boehm
and R. W. Wolverton
This paper summarizes certain lessons we learned recently in developing a software cost estimation model for TRW. With respect to general needs in the cost estimation field, it was found particularly important and useful to address the following three needs: to develop a set of well-defined, agreed-on criteria for the “goodness” of a software cost model; to evaluate existing and future models with respect to these criteria; to emphasize “constructive” models that relate their cost estimates to actual software phenomenology and project dynamics. This paper presents a set of criteria found to be particularly Important in practical software cost estimation, together with examples of their importance.
Some Lessons Learned
Con.st,‘lrc.til.ene.ss. Can a user tell why the model gives
the estimates it does‘? Does it help one understand the software job to be done? Detail.
Does the model easily accommodate the estimation of a software system consisting of a number of subsystems and units‘? Does it give (accurate) phase and activity breakdowns?
Stubilif_v.
Do small differences in inputs produce small differences in output cost estimates?
Scope. Does the model cover the class of software projects whose costs you need to estimate’! Errsr c?f’ USC. Are the model
to understand
inputs
and options
easy
and specify?
P~o.sprc.ti1,~nrs.s. Does the model avoid the use of in-
CRITERIA FOR THE GOODNESS SOFTWARE COST MODEL
formation that will not be well known project is complete‘?’
OF A
Dc$/lition.
Has the model clearly defined which costs it is estimating, and which costs it is excluding‘?
Are the estimates close to the actual expended on the projects?
Objec~tilYt>~. Does the model
the
P~~~:sitnony. Does the model
Our initial set of criteria reflected our primary objective in developing the new TRW Software Cost Estimating Program (SCEP): to serve as an aid in developing cost estimates for competitive proposals on large government software projects. Other objectives for developing a cost model-e.g., to evaluate the impact of using new techniques-may require a somewhat different set of criteria. We found the following list of criteria important in our context:
Fidrlity.
until
costs
avoid allocating most of the software cost variance to poorly calibrated subjective factors (e.g., complexity)? That is, is it hard to jigger the model to obtain any result you want‘?
redundant factors, ciable contribution
avoid the use of highly or factors that make no appreto the results?
In the process of developing the TRW SCEP model and evaluating existing models [ I- IO]. we found each of these criteria important in terms of lessons learned involving software cost modeling. Results ofour evaluation and model development with respect to these criteria are described in the following section. EXAMPLES
OF THE UTILITY
OF THE CRITERIA
Definition. One of the first questions a proposal manager asks whenever we run a software cost model to support a cost proposal is, “Does this estimate include the cost of management? Requirements analysis? Training‘? Computer operations?” Surprisingly, the documentation for most existing cost models does not answer these questions satisfactorily. For the
‘Clearly, this criterion is specific to the objective of cost ,wc’tlkfio~~. For other objectives, such as technology impact assessment. a retrospective model could be appropriate.
The Journal of Systems and Sofware I 195-201 (1980) ti” Elsevie~ North Holland. Inc.. 1980
B. W. Boehm and R. W. Wolverton
196
SUBSYSTEM . * l
I SUBSYSTEM
SSX ACTIVITIES
SXl SSX
CODE AN0 UNIT TEST
l
SX411 UNIT TEST PLAN
PRE-ACCEPT.
TEST
MGMT.
SX4121
PLANS
SX4122
PROCEDURES
SX4123
INTEGRATION
SX4124
TEST
5X4125
REPORTS
l
SX413 ACCEPT. TEST
8X222 CONTRACT
SX2111
FCNL. ROTS.
:
1X223 END ITEM ACC. PLAN
SX4131
PLANS
SX2112
PERF. RQT?..
:
SX224 CUSTOMER
5X4132
PROCEDURES
SX2113
I/F RQTS.
:
SX225 OFFSITE
SX4133
ACC. TEST
SX2114
TEST RQTS.
:
SX226 SUBCONTRACT
SX4134
REPORTS
SX2115
RQTS.VAL.
SX212
S/W SSX DESIGN
SX2121
ARCH. DESIGN
$X2122
IIF DESIGN
SX2123
DB DESIGN
SX2124
DES. VEA.
SX2125
PDR
SX2126
CDR
MGMT.
I/F
OFF. MGMT. t&GMT.
$X227 DES. REVIEW BD.
SX414 TEST BEDS
SX228 TEST REVIEW
SX415 TEST TOOLS
BD.
: . . . l
SX213 PROJ. STDS. SX214 USER MANUALS MGMT.
SX2151
CCB
SX2151
CM0
. . ..**.*.**.~*.*~*~.~..**
SX2153
PROG. SUP. LIB.
: INCLUDED *IN STANDARD ~S~FTWARE ~0s~
SX217 STUDIES
Figure 1. Work hierarchy.
& ANAL
breakdown
.. .. .
l
l
l
SX216 OUAL ASSUR
* .
,............*........**.... :. SX42 POST-ACCEPT. TEST .. .. SX421 TECH. EVAL. SUPP. .
.
SX215 CONFIG.
l
SX416 TEST DATA
SX229 AUDITS
:
: i .. : i ‘ * i .. i . . l
i
l
. . . . : .
DOC’N
S/W RQTS.
i * SX2116 SRR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
: . . :
SX41
SX33
SX221 COSTISCHIPERF. SX211
: l * . . . . : .
ENGR. DATA MGMT. DATA
SX412 INTEG. & TEST
.I.*..~~.*...*~~~***~.**~*.***.
: l * . . . . . . .
SX31 SX32
ESTIMATES
.*.......................
structure
(WBS)
.. .. . ... .. .. :. .. l
. . :
+ .
activity
TRW SCEP model, we found that the best solution was to define a standard work breakdown structure (Figure 1) and use it to define those costs that were included in our estimates and those that were excluded. We found that some existing models seemed to work well for some classes of software and not very well for others. For example, Figure 2 presents the results of our analysis of the Boeing [ 91 and Putnam [ 101 estimation models when applied to their own (Boeing and U.S. Army Computer Systems Command) data. The Boeing model appears to estimate small projects reasonably well, but gives extreme overestimates on large projects; the Putnam 1978 model does just the reverse.2 It would be highly Fidelity.
Tutnam has recently developed a new model called SLIM which evidently eliminates this problem.
l
SX422 OPER. EVAL.
SUPP.
SX5 TRAINING SX6l
CONVERSION
SX62
INSTALLATION
t*..***.**....* :. NOT INCLUDED :. ...............
valuable for the field if similar information were available with respect to other models and other software project attributes. For comparison, Figure 3 indicates the performance of the TRW SCEP model on 20 completed TRW projects. The sequence of activities in developing, calibrating, and evaluating the model was to: 1. Survey the current state of the art in cost estimation. Table 1 shows one of the result of the survey, a summa~ of the factors used in the major current cost estimation models. 2. Develop a baseline model. The phase-sensitive nature of the model is based on a model developed earlier by the authors to evaluate the cost impact of new software technologies; it is functionally similar to the Boeing model in this regard. 3. Refine the model parameters via a two-round Delphi exercise involving 10 experienced TRW line and project managers. 4. Calibrate the model parameters using an initial sample of seven completed projects. 5. Evaluate the model using a sample of 20 completed
197
Table 1. Factors Used in Various Cost Models GROUP
SDC
FACTOR
TRW 72
65 SIZE
SOURCE
PUTNAM 78
X
NO.
ROUTINES
X
NO.
DATA
NO.
REPORTS
RCA
DOTY
IBM
X
X
INST. INST.
OBJECT
T
X
X
X X
ITEMS
X
COMPLEXITY
X
X
X
X
X
LANGUAGE
X
RE-USE
X -__
____ HARDWARt ATTRIB
TIME
H/W
CONSTRAINT
H/W
PROJECT
PERSONNEL
ATTRIB.
PERS. CONTINUITY
DEV.
TOOLS
X
X
X
___ X X X X _X X
X X
X
X
X
~-
c
X
X
X X
X
X
EXPER.
LANGUAGE
X
X
QUALITY
EXPERIENCE
APPLIC.
X
X
CONFIGURATION
CONCURRENT
H/W
X
CONSTRAINT
STORAGE
L / t
X X
X
X
ATTRIB.
X
X
NO. PERSONNEL TYPE
[
X
X
DOCUMENTATION
PROGRAM
TRW 78
BOEING
X X
X
X
EXPER. X
&TECHNIQUES
ENVIR.
CUSTOMER
ATTRIB.
ROTS.
DEFINITION
ROTS.
VOLATILITY
X
I/F
X
X X X
SCHEDULE SECURITY COMPUTER
X
ACCESS
X
REQUIRED
X
X
TRAVEL/REHOSTING QUALITY
7 RESULTS
A
ON BOEING
PUTNAM
1978 RESULTS
USA-CSC
PROJECTS
PROJECTS
ON
OD
6
0
A
A
Figure 2. Estimator Boeing and Putnam
fidelity vs project 1978 models.
size:
E
CARMOCS
aCOMIS
t E F f
A MPAS
*ACS
A SAAS 2
aMPMIS ,, COSCOM
I
5
10
20
50
loo ACTUAL ACTUAL
200
500
MAN-MONTHS
0
MAN-YEARS
a
1000
2000
B. W. Boehm and R. W. Wolverton
PROJECT
SIZE:
MEDIUM
TO VERY
PROJECT
TYPE:
C3, AVIONICS,
ACTUAL
Figure 3. Example performance.
of
the model produce any cost estimate desired, simply by modifying the subjective complexity factor. (In fact, PRICE S has a mode called ECIRP that will do this for you automatically.) Although modifying complexity may solve a user’s short-term pricing problem, it does not produce any objective and meaningful cost estimation function for the user. In developing the TRW SCEP model, we found that although we were unable to avoid including a complexity factor, we have made the complexity rating an attribute of each individual unit in the software, and have provided users with a set of scales for calibrating the complexity of different types of unit. Table 2 shows the complexity scale for computational/numerical analysis type units as an example: such a scale makes the complexity rating a much more objective, verifiable attribute.
LARGE
SENSOR,
ANALYTIC,
SUPPORT
MAN-MONTHS
TRW
software
cost
model
projects, including the initial seven. This evaluation was done by an independent third party. The high value of R2, the square of the correlation coefficient, should be interpreted with considerable care. For example, the sample consists of only medium to very large TRW government contract software efforts; we are not sure how it performs on other types of software effort. Further, the calibration is on past projects. We have included a factor to cover the impact of software technology improvements; however, until we get more experience in comparing SCEP estimates with the resulting project actuals, we must consider the model still in an experimental state.
Constructiveness. We use the TRW SCEP model as a means of cross-checking separate cost estimates developed by project personnel in different ways. Inevitably, this leads to differences between the various estimates, and a need to understand why one was higher or lower than the other. Similarly, project personnel want to know in project terms why different factor ratings give them different cost estimates. To help answer these questions, and to promote accurate factor ratings by users, we have provided a
Figure 4. Effect of complexity model (September 1977 version). 260,
factor
I
in RCA PRICE
S
I
I
Objectivity. Figure 4 shows one of the results of our analysis of the RCA PRICE S model in September 1977,” indicating the extreme sensitivity of the 1977 model to the subjective complexity (SDCPLX) factor. If a project may be described as “hard,” the model will produce a cost estimate that is six to seven times higher than an “easy” project; this is a huge source of variation for a parameter that is entirely subjective. Experience indicates that this means a user can make
“Subsequently, PRICE S has incorporated a set ofguidelines for using and adjusting the complexity factor which reduce the magnitude of this problem.
-0
1.0
2.0 PLATFORM.
PLTFM
Software
Cost Modeling
199 Table 2. Example of Complexity 1
VERY LOW
COMPUTATIONS EXPRESSIONS:
2
LOW
USE OF STANDARD MATH AND STATISTICAL ROUTINES AND BASIC MATRIX-VECTOR OPERATIONS
3
AVERAGE
USE OF BASIC NUMERICAL ANALYSIS SKILLS E.G., MULTIVARIATE INTERPOLATION ORDINARY DIFFERENTIAL EQUATIONS
4
HIGH
DIFFICULT BUT STRUCTURED NUMERICAL ANALYSIS E.G., NEAR-SINGULAR MATRIX EQUATIONS, PARTIAL DIFFERENTIAL EQUATIONS
5
VERY
*SPECIAL
OPTION
HIGH
Table 3. Example of Factor/Rating
I. REc3”IRED OVALITY
LITTLE
LITTLE
UNSTRUCTURED NUMERICAL E.G., FAST, HIGHLY ACCURATE OF NOISY, STOCHASTIC DATA
ment by a certain factor, which may vary by phase. Table 3 thus tells the user why costs have increased, in terms of the impact of a very high required quality on the project activities. In general, we have found that such tables do help the users of the cost model better understand the software job for which they are preparing.
Definitions: TRW Cost Model
REOUIREMENTS ANALYSIS
MANY
DIFFICULT, ANALYSIS ANALYSIS
TO EVALUATE SIMPLE A = B + C * (D - E)
AVAILABLE.
table for each of the factors in the TRW SCEP model indicating the impact on project activities of the various factors and their ratings. For example, the table for the “required quality” factor is shown as Table 3. When a system is rated as having a “very high” required quality, the SCEP model will increase the cost of performing the various phases of system develop-
FACTOR
Scale: TRW Cost Model
DETAIL TBD’S “ALID’N
PRELlMlNARY DESIGN
LITTLE MANY
DETAIL TBD’S
DESIGN
INFO.
MINIMAL
UOF’S
CODE & UNIT
MlNlMAL
TEST
INTEGRATION AND TEST
MINIMAL
U.T. PLAN
TEST PLANS
NO TEST PROCEDURES
MINIMAL
OA. CM
MINIMAL
DA. CM. STDS.
MINIMAL
CDR
MlNlMAL
OA. CM
MlNlMAL
OA, CM
MlNlMAL
POR
NO DRAFT
MINIMAL NOMINAL
I/O AND OFFTESTS
MINIMAL NOMINAL
STRESS,OFFTESTS
MlNlMAL
USER MAN
MINIMAL DOCU.
AS-BUILT
LlTTLE
BAS,C
“ERIF’N
INFO.
VERIF’N TED’S
BASIC DA.. CM. STOS BASIC
PDR
MlNlMAL
FULL
USER MAN.
USER MAN
FREDUENT
MODERATE
DETAIL
BAS,C “DF’S. CM. CDR MlNlMAL
PA,
USER MAN
TRW POLlCfES
BASIC
BASIC
OA. CM. U.M.
FULL
TRW POLICIES
DETAILED “ERIF’N DA. CM. STDS. CDR DOC’N
FULL
I/O AND TESTS
MANY
RDTS.
UNTESTED
BASlC TEST PLANS
U.T. PLAN
PARTlAL FCL’S, PATH TEST, STDS CHECK
PARTIAL NOMINAL
USER MAN.
DETAILED VERIF’N. OA, CM, STOS. PDR, OOC’N
I”&” INTERFACE ISUPPORT. RESPONSE)
BASIC
DESIGN
MlNlMAL FCL’S. PATH TEST, STDS CHECK
NO DRAFT
DETAILED “ALID’N. PLANS. SRR
DETAIL
OFF-
MINIMAL FREQUENT UNTESTED
ROTS
BASIC OA. CM, U M. PARTIAL NOMINAL
TRW POLICIES
TEST PROCED.
FULL
STRESS, TESTS
OFF
TRW POLICIES
DETAILED U.T. PLAN, FCL’S, OA. CM. DOC’N CODE WALKTHRUS
DETAlLED TEST PLANS. PROCEDURES. OA, CM, DOC’N
EXTENSIVE OFFNOMINAL TESTS
EXTENSlVE STRESS, OFF-NOMINAL TESTS
DETAILED “ERIF’N OA. CM. STDS, PDR. DOC’N
DETAILED VERIF’N. DA, CM. STOS. CDR. DOC’N
DETAILED U.T. PLAN, (FCL’S. OA, CM. DOC’NI CODE WALKTHRUS
VERY DETAILED TEST PLANS. PRDC’S, OA, CM. DOC’N
IV&”
IV&V
VERY EXTENSlVE NOMlNAL TESTS
VERY EXTENSlVE STRESS.OFF-NOMINAL TESTS
,NTERFACE
,NTERFACE
IV&V
INTERFACE
OFF-
IV&”
INTERFACE
B. W. Boehm and R. W. Wolverton
200 Detail. We have also found from experience that models requiring greater detail tend to produce more accurate estimates, mainly because:
the gathering of greater detail tends to increase people’s understanding of the job to be done; and ifthe added detail results in the overall estimate being the sum of some smaller individual estimates, the law of large numbers tends to work to decrease the variance in the estimate. Stability. Figure 5 shows one result ofour analysis of the Doty model [7,8] ; it has a severe discontinuity in the neighborhood of 10,000 source instructions. In that neighborhood, small differences in sizing can lead to large differences in estimated cost. Such a lack of stability is an important thing to know about a cost model. Scope. In our comparative cost model analysis to date, we have found that some existing models seems to work well for some classes of software and not very well for others. For example, we have found that the TRW SCEP model overestimates costs on projects that are less than 5 person-years in total effort, indicating that one of our linearity assumptions holds only
Figure 5. Doty model:
suggested
estimating
relationships.
over a certain range of project sizes. It is currently well calibrated over the range 60-2000 m&n-months. Ease of use. A cost model will be less useful in general to the extent that it relies on inputs that are difficult to understand and obtain, or to the extent that it makes it difficult for the user to perform common cost estimation functions such as the combination of the estimates of two subsystems into an estimate for the overall system. (We have encountered the latter difficulty on several existing models.)
One current problem with the use of Halstead model [ 121is that it relies on detailed properties (i.e., number of operaoperands) are generally unknown at the time estimate is needed.
Prospectiveness.
practical knowing tors and the cost
Parsimony. A model will be less useful to the extent that it requires a number of inputs where one would suffice. An example is the Walston-Felix model [S] which has separate entries for the use of different modern programming practices (MPPs) such as top-down development, structured code, and use of inspections. For the original research purpose of the Walston-Felix model, it was important to distinguish these entries, but for practical estimation of projects using modern management guidelines, a single “use of MPPs” factor would suffice.
CONCLUSIONS 500 -
We have found the 10 criteria discussed above extremely valuable in evaluating alternative software cost estimation methods and in developing the TRW SCEP model, which has been used routinely over the past 18 months in estimating and tracking the costs of over $100 million worth of software on over 50 projects and proposals. We have particularly found that its constructive nature has helped our staff (both hardware and software people) become more knowledgeable about and sensitive to software cost drivers.
P .” / / .’ .’ .’ .‘ .’
REFERENCES lo/ 5-
/’
/
/’
-.-NOMINAL
/’ ---NOMINAL
SOURCE INSTRUCTIONS
OR
1. E. A. Nelson, Management Handbook for the Estimation of Computer Progrumming Costs, SDC, ADA648750, 31 October 1966. 2. R. W. Wolverton, The Cost of Developing Large-Scale Software, IEEE Trmns. otz Comprrters (June), 615-636 (1974). 3. L. H. Putnam, A General Solution to the Macro Software Sizing and Estimating Problem, IEEE Trans. Software Engineering (July), 345-361 (1978).
Software Cost Modeling
201
4. J. K. Herd. J. N. Postak. W.E. Russell, and K. R. Stewart, .Yofr~~uwCost Es timntion Study-Study Results, Doty Associates, Inc., Final Technical Report, RADCTR-77-220. Vol. 1 (of two). June 1977: NTIS No. ADA042264. S. D. L. Doty, P. J. Nelson, and K. R. Stewart, Soft~twc, Cost Estimcttion
Strrc!\.--Glricl’elinesJi,r
Imptwled
Estimclti,zg , Doty Associates,
Soj-
Inc., Final Technical Report, RADC-TR-77-220, Vol. II (of two), August 1977: NTIS No. AD-A044609: see also errata sheet, J. H. Herd, Doty Associates, Inc., 416 Hungerford Dr., Rockville, MD 20850, 6 June 1978. 6. John Schneider, IV. ,4 Prdiminruy Culihrution of the RCA PRICED Softnuw Cost Estimcction MO&~, NTIS No. .4D-A046808, September 1977. 7. J. D. Aron, Estimcrtiq~ Resourcesfor Lurge Programnring Jystem~ , NATO Science Committee, Rome, Italy. October 1969. u‘(wc
Cost
8. C. P. Felix and C. E. Walston. A Method of Programming Measurement and Estimation, IBM s\.stc,m.\ .Iorrrnu/ 16 ( 1) ( 1977). 9. R. K. D. Black, R. P. Curnow. R. Katz, and M. D. Gray, BCS Sqfhure Protlrlctiwr Drlttr , Boeing Computer Services, Inc., Final Technical Report, KADCTR-77-116. March 1977: NTIS No. AD-A039852. 10. L. H. Putnam and R. W. Wolverton. “Quantitative Management: Software Cost Estimating. in lEEE Camp.
Sot.
Firxt
Int.
Computc,r
S(!/hwrc~
rrntl Appli-
(COMPSAC 77). IEEE Cat. No. EHO 129-7. Chicago, 8-11 November 1977, 326 pp. 11. B. W. Boehm, C. A. Bosch, A. S. Liddle, and R. W. Wolverton, The Impact of New Technologies on Software Configuration Management, TRW Report to USAF-ESD, Contract Fl%28-74-C-0154. IO June 1974. 12. M. H. Halstead, Elements c!f’Sc?fi,l~are Scirncc.. Elsevier North Holland, 1977. (utiouJ
Cor$