- Email: [email protected]
Benchmarking the price of computing
Benchmarkingthe price of computing Results of a survey Paul Heller EDUCOM - Interuniversity Communications Council Inc., P.O. Box 364, Princeton NJ 08540, U.S.A.
A computing benchmark survey was conducted in 1975 by the EDUCOM Planning Council on Computing in Education and Research. Thirteen different FORTRAN programs were run at major universities and colleges. An analysis is made of internal and external prices for individual jobs and representative university workloads. In the survey an attempt was m~de to control factors such as arithmetic precision, t'de organization, compiler type, and turn-around time. Significant differences in prices may have arisen, however, due to differences in cost accounting practices and cost recovery policies. Nevertheless, the data support the general view that substantial benefits can be achieved through selective use of cor~nutational resources at a variety of different computer centers. Keywords" benchmark programs, pricing of co,np..Ler services, workload pricing, price comparison, computer networks, network economics, job mix
I. Introduction During 1975, a benchmark survey o f prices charged for computin.7 services was c o n d u c t e d b y the Planning Council on Lomputing h-r Education and Research. (For a description o f the Planning Council, see E D U C¢9M Bulletin, Volume 9, Number 4 (winter 1974), p. 16.) Table 1 lists the 21 universities and colleges that are members o f the Planning Council. 26 computing facilities at 19 m e m b e r institutions provided the prices included in the results presented here. The variety o f mainframe hardware used b y the reporting installations is shown in Table 2. The purpose o f this survey was to gather comparative material on the prices o f computing services. It was felt that such information would be useful as the institutions which make up the Planning Council begin to b u y and sell computing services over a network. Accordingly, the results o f the survey provide some measure o f price differentials only, and do not necessarily reflect differences in technical performance o f the reporting computer installations.
2. Interpreting the results Webster defines benchmark as "a point o f reference from which measurements may be made". In the context o f this report the Foint of reference Js a set Table 1. Member institutions; Pianning Council on Computing in Education and Research.
Paul /Idler, a Senior Research Asso"
ciate with EDUCOM's Planning Council on Computing in Education and Research, is involved in ~ variety of network activities. His primary concern is with the planning and implementation of a national facilitating network for the higher education community. Prior to joining EDU! . CObl, Mr. Holler was at the Wharton :" School of ti~e University of Pennsylvania "''~-~,~,,~,~he taught courses in information systems and was Director of a management training program in information systems. He has a Master's degree in Business Administration from Wharton and a Baehelor.'s degree in Electrical Engineering from Purdue University. © North-Holland Pubfishing Company Computer Networks 1 (1976) 27-32
University of California California State University and C ,lieges Carnegie-Mellon University Case Western Reserve University The University of Chicago Dartmouth College Harvard University University of Illinois Lehigh University Massachusetts Institute of Techn~-,L~gy University of Minnesota University of North Carolina University of Notre Dame University of Pennsylvania Princeton University Stanford University State University of New York University of Texas University of Utah University of Wisconsin Yale University
28
P. HeUer/ Benchmarking the price of computing
Table 2, Hardware included in benchmark survey.
Table 3. The benchmark programs.
Manufacturer
Model(s)
Program name
Brief description
UNIVAC Control Data Corp. ~CDC)
i 108, 1 ! 10 3300, 6400, 6600 Cyber 70/74 360/50, 360/65,360/67, 360/75, 360/91,370/151L 370/165,370/168 G635 B6700 Sigma-7 PDP-I 0
TRIVIAL
Does nearly nothing in order to highlight job overhead and minimum charges. A loop containing the four arithmetic operators is executed one million times. Two 60 × 60 matrices are multipfied fifty times. Two 221 X 221 matrices a~remultipfied once. Card-to-disk, 2,000 data cards are read and I 0,000 card images are written sequentially on disk. Disk-i~.,~', the sequential file created by CTOD is accessed and summarized. 50,000 binary card images are written to disk. Writes and reads a 20-miifion-character randora-access file nonsequentially. Writes and reads a 20-million-character random-access file sequentially.
IBM Honeywell Burroughs Xerox Digital Equipment
CRUNCHER MATMULI MATMUL2 CTOD DSKRD PUREIO
of specific computational tasks written in FORTRAN and the measurements are the prices charged to academic users for having these tasks executed at 26 different computer installations. Price measurements c~m pros ide users and administrators with a basis for compa~ng the prices, fee structure~, and cost-recovery policies in effect at the installations included in the survey. In additior., an indication of the benefits that might be available through a national computer network may be inferred.However,interpreting the results requires an understanding of the ,ncertainty of price , , y has a pricing comparisons. Each computing .~'.~. .":':'" policy which reflects the institution's priorities and its attitude toward computing. At some institutions, certain classes of comput.er usage are treated much like library usage, with no price or direct charge involved in the service. At other institutions, there are implicit subsidies for computing which prices charged. At still other institutions qect a "full cost recovery" pricing policy~ In price comparisons made at this time must allow for the likelihood of significant change as the computer resource sharing network matures.
3. Benchmark tasks
Ideally, the tasks considered in arriving at benchmark prices should faithfully represent the functional tasks of instructional, research, and administrative users in higher education. A researcher, for taneous equations, but be indifferent to the choice of t h e algorithm, programming language, or hardware u s e d t o o b t a i n the solution. In the Planning Council's price benchmark survey, a uniform set of tasks was chosen t o represent the functional tasks desired b y a
ARMWHIP ARMGLIDE
hypothetical academic user. In order to insure compatibility among different computer installations, a uniform set o f FORTRAN programs was used. it is likely that several of the tasks could have been performed more efficiently at some facilities if other languages or packages were chosen, but comparability would have been lost. FORTRAN was chosen because it exists at all the installations in the survey and it would often be the language selected for many of the tasks. Tile name and a brief description of nine FORTRAN-defined t~,sks are shown in Table 3. The program TRIVIAL can be considered representative of the class of jobs that includes ~m~dl student jobs and the "compile and bomb" runs. CRUNCHER, MATMUL1, and MATMUL2 represent computation-intensive jobs with increasing needs for primary storage. The memory requirements for MATMUL2 exceeded what was available at several installations. CTOD, DSKRD, and PUREIO represent jobs ~ t h moderate 1/O activity, including card reading and sequential disk f'de access. ARMWHIP and ARMGLIDE, representing l/O-intensive jobs using random-access f'de organization, are perhaps the least representative, as FORTRAN would rarely be used for such intensive file processing. Because there is no generally recognized FORTRAN standard (such as ANSI F O R T ~ ) for binary or direct access files, the last three programs either used local conventions or could not be run at all. Any modification necessary
P. H&r
/ Bcnchtnarking
was subject to the condition that only widely known and available techniques be used: No programming “tricks” were allowed.
the price of computing
29
more decimal digits in all non-integer computations. Except for ARMWHIP and ARMGLIDE, it was specifi;d that turn-around time (and, implicit!y, job class) be such that, on a typical day, there would be at least a 9% likelihood that results would be in the user’s hands within one hour after submission. For ARMWHIP and ARMGLIDE, turn-around was specified as overnight.
! 4. Standard conditions In addition to having a common set of tasks, meaningful comparisons require uniform conditions for such factors as compiler type, arithmetic precision, and turn-around time. Three compiler types were defined: student, standard, and optimizing. Existihng compilers do not always fit neatly into one type, and installations did not always have a compile; of each type. In terms of generally available compilers for IBM equipment, a student compiler might be WATFIV, the standard compiler, FORTRAN G, and the optimizing compiler, FORTRAN H. Arithmetic p&sion was specified as fourteen or
5. The results
Table 4 shows several price statistics for program task/compile? combinations. The figures are internal prices that would be charged to an on-campus research project with a source of funds outside the university. At several installations, only one or two compiler types were available. The last three program tasks, which require binary or direct access files, were
Table 4. I;ORTRAN benchmarks, internal price statistics. TRIVIAL __._-.___. Compiler type Number of installations with successful run -__ ---__ Mean price ($) Median price (5) Highest price (S) 85th percentile price ($) 15th pcrcentilc price ($1 Lowest price ($) 85th pcrcentile/lSth Highest/lowest ----
percentile
Compiler type Number of installations with successful run
CRUNCf 1I:R _._-.. ~__~__
MATMUL!
Student
Stzmdard
Optimizing
Student
Standard
Standard
17
25
I8
13
26
24
Mean price ($) Median price ($) Highest price ($) 85 th percentile price ($) 15th percentile price ($) Lowest price (%) percentile
Note: N = 26 installations
Optimizing
~_ 0.21 0.14 0.91 0.37 0.00 0.00
0.55 0.48 I.51 0.84 0.15 0.00
0.73 0.63 1.90 1.09 0.29 0.17
7.57 6.23 18.42 12.16 3.37 0.00
2.99 2.19 10.76 4.49 I.15 0.67
20.17 16.48 54.19 39.26 6.211 3.18
-
5.52
3.12 11.18
3.61 -. _.___-____--
3.92 16.06
6.33 17.04 -~-
MATMUL2
CTOD
DSKRD ~-
Standard
Standard _-.-
Standard
24
23
9
---PURE10 --
25
20 ___
20 - ._._._._. 13.93 R,73 57.57 26.74 3.W 2.05 8.91 28.08 .-- -- _-------
ARMWHIP _~__ ~.
Standard Standard ___-_---------~.
___--
85th percentile/lSth Highest/lowest
_ __._.__~~. _
_ ~. ~__._
-I_____-
-
ARMGLIDI: _I_..-Standard __---.
19 __-.---
--.---
73.79 21.43 414.95 72.13 10.80 10.56
7.37 6.69 18.04 9.70 4.15 3.54
2.85 2.29 8.16 3.56 I .44 0.98
17.40 10.31 59.10 35.39 3.85 3.21
84.13 51.11 405.46 143.60 16.95 10.14
70.64 49.64 231.22 149.86 15.87 5.04
6.73 39.29
2.34 5.10
2.41 8.33
9.19 18.41
8.47 39.99
9.44 45.88 _-__
36
P. HeUer/ Benchmarking the price of computing
n o t u n i f o r m l y s u p p o r t e d . F o r t h e s e and similar rea~ n s , several installations were not able to run all
jobs. The number o f installations reporting is shown in ~ e I o f Table 4. Entries in the bottom row in the table indicate the enormous variation in price for each task - price differentials with ~tios as high as 45 : 1. Since the highest and lowest prices are likely t o contain anom~ies, another comparison was made eliminating the highest and lowest 15% o f the prices reported for each task. Ratios for this comparison are shown on the next-tolast line and still range as high as 9 : I. More than one installation does not charge at all for jobs such as TRIVIAL where the overhead necessary to account for resource usage might exceed the resources used by the job. However, it is surprising that one installation had a zero price policy for CRUNCHER, which had an over-ll average price o f $ 7.57!
6, B e w a r e
In evaluating any benchmark survey and comparing prices, many variables must be considered. The choice of tasks and the use o f FORTRAN obviously introduce bias. In addition, there are many applications that may not be adequately represented by one or a combination of the benchmarks. For example, the use of packaged software such as SPSS is not directly represented although much of SPSS is written in FORTRAN. Since administrative and other f'de processing applications would be more likely to use PL/I, COBOL, a report generator, or a data base mannot e of interactive computing is not included in this survey, although itis an i~lcreasingly important mode of com~ differences among installations are imporless ob~-,ious. Ideally, all costs incurred in rea~ed in tagement improve D,ices; however, important cost space and utilities, are not ac-
is ~.~cludubstantial b ,o
Federal, state, and institutional policies also fre-
quently impose unusual constraints. For example, federal policy of disallowing the cost of capital as a legitimate component in setting prices for federal contracts shrinks the cost base when equipment is purchased rather than leased or rented. Explicit subsidies from general university ~;ources, whether these subsidies are planned or the lesult of unforeseen deficits, also compound cost and pricing problems.
7. S y n t h e t i c w o r k l o a d
The dollar amounts presented in Table 4 allow a comparison of individual job prices across installations. However, to make a general comparison of the prices that would be charged for all research computing or all university computing, the composition of the broader workload must be known. Quantitative characterization o f a workload is fraught with difficulty. Such comparisons are usually made by taking a sample of actual jobs and running them at other installations. Because this procedure is both costly and time-consuming, it is used only when large purchases are being considered. The synthetic workload procedure described here can provide a basis for comparison of a large sample with only moderate effort although with some loss in accuracy. The procedure defines a workload or job mix as a linear combination of the thi: ~,,'~entasks cited in the benchmark survey. Two suc~ combinations shown in Table 5 (Mix A and Mix Bt :re designed to represent the workload at t~o spec~i:~c institutions.
Table 5. Synthetic workload job mix. Job name and compiler type
TRIVIAL - student TRIVIAL - standard TRIVIAL - optimizing CRUNCHER - student CRUNCHER - standard MATMUL 1 - standard MATMULI - optimizing MATMUL2 - standard G O D - standard DSKRD - standard PUREIO - standard A R ~ i P ~- standard ARMGL[DE - standard
Number af jobs Mix A
Mix B
50.00 14.58 9.45 0.96 6.38 0.61 1.15 0.00 0.90 3.49 0.58 0.20 0.20
21.43 4.17 0.00 0.80 20.55 1.21 0.57 0.00 0.30 4;37 0.58 0.02 0.02
P. HeUer / Benchmarking the price of computing
Table 6. Synthetic workload internal price statistics by installation. Price basis
Lowest price ($) 15th percentile price ($) Median price ($) Mean price f$) 85th percentile price ($) ltighest price ($) 85th percentile/l 5th percentile ilighest/lowest
Workload price Mix A
Mix B
60.10 75.10 127.66 144.82 211 A6 338.96
45.22 65.89 117.84 137.00 203.86 376.64
2.82 5.64
3.09 8.33
For example, in Mix A the workload would consist of 50.00 student TRIVIAL jobs, 14.58 standard TRIVIAL jobs, 9.45 optimizing TRIVIAL jobs, and so on. In comparison to Mix A, Mix B could be characterized as more research intensive, as evidenced by the larger coefficient in the computational-intensive job CRUNCHER. in addition to the previously discussed caveats, the reader should remember that the resultant workload price depends greatly on the definition o f the synthetic workload mix. in Table 6 the prices associated with each of the two mixes are summarized. A workload of Mix A would cost an on.campus user at the lowest-price facility $ 60.10. A user with the same workload at the highest-priced facility would pay $ 338.96. If a user could send the individual jobs of Mix A to the facility with the lowest price for each job type, the workload price would be $ 21.62. Of course, a user
31
sending jobs to several locations would not necessarily pay internal rates and would also incur communications costs.
8. External prices
Of the 26 installations surveyed, 20 supplied external as well as internal prices. The external price is defined as the price charged to an educational user not affiliated with the university supplying the service. In a national computer resource sharing network, remote users would ordinarily pay external prices. The most common external pricing strategy is a simple percentage surcharge on internal prices. Table 7 shows the surcharge percentage for the 20 installations reporting external prices. it turns out that the installation with the lowest price shown in Table 6 is lowest for both job mixes; and in addition, it has external rates identical to internal rates. Any educationai user would thus pay a price of $ 60.10 for a workload of Mix A done at that installation. If a user paid external rates and split up a workload (of Mix A) by job type in order to send jobs to the installations with the lowest individual job price, the price would be $ 23.83 and jobs would be run at ten different installations. Even local users at the installation with the lowest overall price might be interested in the lower individual job prices at other installations. In a mature network it is likely that external prices will receive more scrutiny from the supplying installation and from the user. External buyers will also need to consider other costs. The results shown here do not include any communications costs which, if included, would make shopping for services less attractive. In particular, if a file of twenty million characters, as necessary for ARMWHIP and ARMGLIDE, were shipped over the network, substantial communications costs would be incurred, Remote users would also face additional costs if they needed to support their own user services with, for example, documentation and consulting aids. Table 7. External user surcharge. Percent surcharge
No. of installations
0 I-I0 I 1-50 >50
9 6 3 2
32!
P. tleller / Benehmarking the price of computing
9. Conclu~ns While many uncertainties remain before judgments can be made about price comparisons, it seems certain that some of the potential benefits of shopping for realiTed. Besides giving institutions an opportunity to take advantage of price differentials, a network would provide access t o scarce or unique computing services not otherwise available. With the recent emergence of public value-added communications networks and distance-independent charges, the technical problems and costs of communications are less inhibiting than
ever before. Furthermore, as such user services as documentation, training, and debugging aids are automated and enhanced, the remote user will be at less of a disadvantage. Clearly, there are major problems to be overcome in developing the organ~tional structure and administrative procedures necessary for effective sharing of computer resources in higher education. However, when such steps have been accomplished, perhaps the greatest impact of a network will prove to be the emergence of entirely new services which could not have been developed at all without the existence of a viable national marketplace.
A computing benchmark survey was conducted in 1975 by the EDUCOM Planning Council on Computing in Education and Research. Thirteen different FORTRAN programs were run at major universities and colleges. An analysis is made of internal and external prices for individual jobs and representative university workloads. In the survey an attempt was m~de to control factors such as arithmetic precision, t'de organization, compiler type, and turn-around time. Significant differences in prices may have arisen, however, due to differences in cost accounting practices and cost recovery policies. Nevertheless, the data support the general view that substantial benefits can be achieved through selective use of cor~nutational resources at a variety of different computer centers. Keywords" benchmark programs, pricing of co,np..Ler services, workload pricing, price comparison, computer networks, network economics, job mix
I. Introduction During 1975, a benchmark survey o f prices charged for computin.7 services was c o n d u c t e d b y the Planning Council on Lomputing h-r Education and Research. (For a description o f the Planning Council, see E D U C¢9M Bulletin, Volume 9, Number 4 (winter 1974), p. 16.) Table 1 lists the 21 universities and colleges that are members o f the Planning Council. 26 computing facilities at 19 m e m b e r institutions provided the prices included in the results presented here. The variety o f mainframe hardware used b y the reporting installations is shown in Table 2. The purpose o f this survey was to gather comparative material on the prices o f computing services. It was felt that such information would be useful as the institutions which make up the Planning Council begin to b u y and sell computing services over a network. Accordingly, the results o f the survey provide some measure o f price differentials only, and do not necessarily reflect differences in technical performance o f the reporting computer installations.
2. Interpreting the results Webster defines benchmark as "a point o f reference from which measurements may be made". In the context o f this report the Foint of reference Js a set Table 1. Member institutions; Pianning Council on Computing in Education and Research.
Paul /Idler, a Senior Research Asso"
ciate with EDUCOM's Planning Council on Computing in Education and Research, is involved in ~ variety of network activities. His primary concern is with the planning and implementation of a national facilitating network for the higher education community. Prior to joining EDU! . CObl, Mr. Holler was at the Wharton :" School of ti~e University of Pennsylvania "''~-~,~,,~,~he taught courses in information systems and was Director of a management training program in information systems. He has a Master's degree in Business Administration from Wharton and a Baehelor.'s degree in Electrical Engineering from Purdue University. © North-Holland Pubfishing Company Computer Networks 1 (1976) 27-32
University of California California State University and C ,lieges Carnegie-Mellon University Case Western Reserve University The University of Chicago Dartmouth College Harvard University University of Illinois Lehigh University Massachusetts Institute of Techn~-,L~gy University of Minnesota University of North Carolina University of Notre Dame University of Pennsylvania Princeton University Stanford University State University of New York University of Texas University of Utah University of Wisconsin Yale University
28
P. HeUer/ Benchmarking the price of computing
Table 2, Hardware included in benchmark survey.
Table 3. The benchmark programs.
Manufacturer
Model(s)
Program name
Brief description
UNIVAC Control Data Corp. ~CDC)
i 108, 1 ! 10 3300, 6400, 6600 Cyber 70/74 360/50, 360/65,360/67, 360/75, 360/91,370/151L 370/165,370/168 G635 B6700 Sigma-7 PDP-I 0
TRIVIAL
Does nearly nothing in order to highlight job overhead and minimum charges. A loop containing the four arithmetic operators is executed one million times. Two 60 × 60 matrices are multipfied fifty times. Two 221 X 221 matrices a~remultipfied once. Card-to-disk, 2,000 data cards are read and I 0,000 card images are written sequentially on disk. Disk-i~.,~', the sequential file created by CTOD is accessed and summarized. 50,000 binary card images are written to disk. Writes and reads a 20-miifion-character randora-access file nonsequentially. Writes and reads a 20-million-character random-access file sequentially.
IBM Honeywell Burroughs Xerox Digital Equipment
CRUNCHER MATMULI MATMUL2 CTOD DSKRD PUREIO
of specific computational tasks written in FORTRAN and the measurements are the prices charged to academic users for having these tasks executed at 26 different computer installations. Price measurements c~m pros ide users and administrators with a basis for compa~ng the prices, fee structure~, and cost-recovery policies in effect at the installations included in the survey. In additior., an indication of the benefits that might be available through a national computer network may be inferred.However,interpreting the results requires an understanding of the ,ncertainty of price , , y has a pricing comparisons. Each computing .~'.~. .":':'" policy which reflects the institution's priorities and its attitude toward computing. At some institutions, certain classes of comput.er usage are treated much like library usage, with no price or direct charge involved in the service. At other institutions, there are implicit subsidies for computing which prices charged. At still other institutions qect a "full cost recovery" pricing policy~ In price comparisons made at this time must allow for the likelihood of significant change as the computer resource sharing network matures.
3. Benchmark tasks
Ideally, the tasks considered in arriving at benchmark prices should faithfully represent the functional tasks of instructional, research, and administrative users in higher education. A researcher, for taneous equations, but be indifferent to the choice of t h e algorithm, programming language, or hardware u s e d t o o b t a i n the solution. In the Planning Council's price benchmark survey, a uniform set of tasks was chosen t o represent the functional tasks desired b y a
ARMWHIP ARMGLIDE
hypothetical academic user. In order to insure compatibility among different computer installations, a uniform set o f FORTRAN programs was used. it is likely that several of the tasks could have been performed more efficiently at some facilities if other languages or packages were chosen, but comparability would have been lost. FORTRAN was chosen because it exists at all the installations in the survey and it would often be the language selected for many of the tasks. Tile name and a brief description of nine FORTRAN-defined t~,sks are shown in Table 3. The program TRIVIAL can be considered representative of the class of jobs that includes ~m~dl student jobs and the "compile and bomb" runs. CRUNCHER, MATMUL1, and MATMUL2 represent computation-intensive jobs with increasing needs for primary storage. The memory requirements for MATMUL2 exceeded what was available at several installations. CTOD, DSKRD, and PUREIO represent jobs ~ t h moderate 1/O activity, including card reading and sequential disk f'de access. ARMWHIP and ARMGLIDE, representing l/O-intensive jobs using random-access f'de organization, are perhaps the least representative, as FORTRAN would rarely be used for such intensive file processing. Because there is no generally recognized FORTRAN standard (such as ANSI F O R T ~ ) for binary or direct access files, the last three programs either used local conventions or could not be run at all. Any modification necessary
P. H&r
/ Bcnchtnarking
was subject to the condition that only widely known and available techniques be used: No programming “tricks” were allowed.
the price of computing
29
more decimal digits in all non-integer computations. Except for ARMWHIP and ARMGLIDE, it was specifi;d that turn-around time (and, implicit!y, job class) be such that, on a typical day, there would be at least a 9% likelihood that results would be in the user’s hands within one hour after submission. For ARMWHIP and ARMGLIDE, turn-around was specified as overnight.
! 4. Standard conditions In addition to having a common set of tasks, meaningful comparisons require uniform conditions for such factors as compiler type, arithmetic precision, and turn-around time. Three compiler types were defined: student, standard, and optimizing. Existihng compilers do not always fit neatly into one type, and installations did not always have a compile; of each type. In terms of generally available compilers for IBM equipment, a student compiler might be WATFIV, the standard compiler, FORTRAN G, and the optimizing compiler, FORTRAN H. Arithmetic p&sion was specified as fourteen or
5. The results
Table 4 shows several price statistics for program task/compile? combinations. The figures are internal prices that would be charged to an on-campus research project with a source of funds outside the university. At several installations, only one or two compiler types were available. The last three program tasks, which require binary or direct access files, were
Table 4. I;ORTRAN benchmarks, internal price statistics. TRIVIAL __._-.___. Compiler type Number of installations with successful run -__ ---__ Mean price ($) Median price (5) Highest price (S) 85th percentile price ($) 15th pcrcentilc price ($1 Lowest price ($) 85th pcrcentile/lSth Highest/lowest ----
percentile
Compiler type Number of installations with successful run
CRUNCf 1I:R _._-.. ~__~__
MATMUL!
Student
Stzmdard
Optimizing
Student
Standard
Standard
17
25
I8
13
26
24
Mean price ($) Median price ($) Highest price ($) 85 th percentile price ($) 15th percentile price ($) Lowest price (%) percentile
Note: N = 26 installations
Optimizing
~_ 0.21 0.14 0.91 0.37 0.00 0.00
0.55 0.48 I.51 0.84 0.15 0.00
0.73 0.63 1.90 1.09 0.29 0.17
7.57 6.23 18.42 12.16 3.37 0.00
2.99 2.19 10.76 4.49 I.15 0.67
20.17 16.48 54.19 39.26 6.211 3.18
-
5.52
3.12 11.18
3.61 -. _.___-____--
3.92 16.06
6.33 17.04 -~-
MATMUL2
CTOD
DSKRD ~-
Standard
Standard _-.-
Standard
24
23
9
---PURE10 --
25
20 ___
20 - ._._._._. 13.93 R,73 57.57 26.74 3.W 2.05 8.91 28.08 .-- -- _-------
ARMWHIP _~__ ~.
Standard Standard ___-_---------~.
___--
85th percentile/lSth Highest/lowest
_ __._.__~~. _
_ ~. ~__._
-I_____-
-
ARMGLIDI: _I_..-Standard __---.
19 __-.---
--.---
73.79 21.43 414.95 72.13 10.80 10.56
7.37 6.69 18.04 9.70 4.15 3.54
2.85 2.29 8.16 3.56 I .44 0.98
17.40 10.31 59.10 35.39 3.85 3.21
84.13 51.11 405.46 143.60 16.95 10.14
70.64 49.64 231.22 149.86 15.87 5.04
6.73 39.29
2.34 5.10
2.41 8.33
9.19 18.41
8.47 39.99
9.44 45.88 _-__
36
P. HeUer/ Benchmarking the price of computing
n o t u n i f o r m l y s u p p o r t e d . F o r t h e s e and similar rea~ n s , several installations were not able to run all
jobs. The number o f installations reporting is shown in ~ e I o f Table 4. Entries in the bottom row in the table indicate the enormous variation in price for each task - price differentials with ~tios as high as 45 : 1. Since the highest and lowest prices are likely t o contain anom~ies, another comparison was made eliminating the highest and lowest 15% o f the prices reported for each task. Ratios for this comparison are shown on the next-tolast line and still range as high as 9 : I. More than one installation does not charge at all for jobs such as TRIVIAL where the overhead necessary to account for resource usage might exceed the resources used by the job. However, it is surprising that one installation had a zero price policy for CRUNCHER, which had an over-ll average price o f $ 7.57!
6, B e w a r e
In evaluating any benchmark survey and comparing prices, many variables must be considered. The choice of tasks and the use o f FORTRAN obviously introduce bias. In addition, there are many applications that may not be adequately represented by one or a combination of the benchmarks. For example, the use of packaged software such as SPSS is not directly represented although much of SPSS is written in FORTRAN. Since administrative and other f'de processing applications would be more likely to use PL/I, COBOL, a report generator, or a data base mannot e of interactive computing is not included in this survey, although itis an i~lcreasingly important mode of com~ differences among installations are imporless ob~-,ious. Ideally, all costs incurred in rea~ed in tagement improve D,ices; however, important cost space and utilities, are not ac-
is ~.~cludubstantial b ,o
Federal, state, and institutional policies also fre-
quently impose unusual constraints. For example, federal policy of disallowing the cost of capital as a legitimate component in setting prices for federal contracts shrinks the cost base when equipment is purchased rather than leased or rented. Explicit subsidies from general university ~;ources, whether these subsidies are planned or the lesult of unforeseen deficits, also compound cost and pricing problems.
7. S y n t h e t i c w o r k l o a d
The dollar amounts presented in Table 4 allow a comparison of individual job prices across installations. However, to make a general comparison of the prices that would be charged for all research computing or all university computing, the composition of the broader workload must be known. Quantitative characterization o f a workload is fraught with difficulty. Such comparisons are usually made by taking a sample of actual jobs and running them at other installations. Because this procedure is both costly and time-consuming, it is used only when large purchases are being considered. The synthetic workload procedure described here can provide a basis for comparison of a large sample with only moderate effort although with some loss in accuracy. The procedure defines a workload or job mix as a linear combination of the thi: ~,,'~entasks cited in the benchmark survey. Two suc~ combinations shown in Table 5 (Mix A and Mix Bt :re designed to represent the workload at t~o spec~i:~c institutions.
Table 5. Synthetic workload job mix. Job name and compiler type
TRIVIAL - student TRIVIAL - standard TRIVIAL - optimizing CRUNCHER - student CRUNCHER - standard MATMUL 1 - standard MATMULI - optimizing MATMUL2 - standard G O D - standard DSKRD - standard PUREIO - standard A R ~ i P ~- standard ARMGL[DE - standard
Number af jobs Mix A
Mix B
50.00 14.58 9.45 0.96 6.38 0.61 1.15 0.00 0.90 3.49 0.58 0.20 0.20
21.43 4.17 0.00 0.80 20.55 1.21 0.57 0.00 0.30 4;37 0.58 0.02 0.02
P. HeUer / Benchmarking the price of computing
Table 6. Synthetic workload internal price statistics by installation. Price basis
Lowest price ($) 15th percentile price ($) Median price ($) Mean price f$) 85th percentile price ($) ltighest price ($) 85th percentile/l 5th percentile ilighest/lowest
Workload price Mix A
Mix B
60.10 75.10 127.66 144.82 211 A6 338.96
45.22 65.89 117.84 137.00 203.86 376.64
2.82 5.64
3.09 8.33
For example, in Mix A the workload would consist of 50.00 student TRIVIAL jobs, 14.58 standard TRIVIAL jobs, 9.45 optimizing TRIVIAL jobs, and so on. In comparison to Mix A, Mix B could be characterized as more research intensive, as evidenced by the larger coefficient in the computational-intensive job CRUNCHER. in addition to the previously discussed caveats, the reader should remember that the resultant workload price depends greatly on the definition o f the synthetic workload mix. in Table 6 the prices associated with each of the two mixes are summarized. A workload of Mix A would cost an on.campus user at the lowest-price facility $ 60.10. A user with the same workload at the highest-priced facility would pay $ 338.96. If a user could send the individual jobs of Mix A to the facility with the lowest price for each job type, the workload price would be $ 21.62. Of course, a user
31
sending jobs to several locations would not necessarily pay internal rates and would also incur communications costs.
8. External prices
Of the 26 installations surveyed, 20 supplied external as well as internal prices. The external price is defined as the price charged to an educational user not affiliated with the university supplying the service. In a national computer resource sharing network, remote users would ordinarily pay external prices. The most common external pricing strategy is a simple percentage surcharge on internal prices. Table 7 shows the surcharge percentage for the 20 installations reporting external prices. it turns out that the installation with the lowest price shown in Table 6 is lowest for both job mixes; and in addition, it has external rates identical to internal rates. Any educationai user would thus pay a price of $ 60.10 for a workload of Mix A done at that installation. If a user paid external rates and split up a workload (of Mix A) by job type in order to send jobs to the installations with the lowest individual job price, the price would be $ 23.83 and jobs would be run at ten different installations. Even local users at the installation with the lowest overall price might be interested in the lower individual job prices at other installations. In a mature network it is likely that external prices will receive more scrutiny from the supplying installation and from the user. External buyers will also need to consider other costs. The results shown here do not include any communications costs which, if included, would make shopping for services less attractive. In particular, if a file of twenty million characters, as necessary for ARMWHIP and ARMGLIDE, were shipped over the network, substantial communications costs would be incurred, Remote users would also face additional costs if they needed to support their own user services with, for example, documentation and consulting aids. Table 7. External user surcharge. Percent surcharge
No. of installations
0 I-I0 I 1-50 >50
9 6 3 2
32!
P. tleller / Benehmarking the price of computing
9. Conclu~ns While many uncertainties remain before judgments can be made about price comparisons, it seems certain that some of the potential benefits of shopping for realiTed. Besides giving institutions an opportunity to take advantage of price differentials, a network would provide access t o scarce or unique computing services not otherwise available. With the recent emergence of public value-added communications networks and distance-independent charges, the technical problems and costs of communications are less inhibiting than
ever before. Furthermore, as such user services as documentation, training, and debugging aids are automated and enhanced, the remote user will be at less of a disadvantage. Clearly, there are major problems to be overcome in developing the organ~tional structure and administrative procedures necessary for effective sharing of computer resources in higher education. However, when such steps have been accomplished, perhaps the greatest impact of a network will prove to be the emergence of entirely new services which could not have been developed at all without the existence of a viable national marketplace.