Classroom exercises in applying the delphi method for decision-making

So&-&on. Plan. Sri. Vol. 5, pp. 363-375(1971).PergamonPress.Printedin Great Britain

CLASSROOM EXERCISES TN APPLYING THE DELPHI METHOD FOR DECISION-MAKING L. R. DOYON Equipment Development

Laboratories,

Raytheon Company, Sudbury, Massachusetts

T. V. Pfizer Pharmaceuticals,

01776

SHEEHAN

235 East 42 Street, New York, N.Y. 10017 and 1-I. I. ZAGOR

Polytechnic

Institute of Brooklyn. 333 Jay Street, Brooklyn, N.Y. 11201

(Received 6 January 1971) This paper describes the results of a series of graduate-engineering classroom exercises in applying the Delphi procedures for formulating group judgments. The exercises were conducted in two phases. The first phase of the exercises was essentially a demonstration of the basic principles of the Delphi method using almanac, factual-type questions for which the answers are known. The second phase was a group exercise in formulating opinions about the educational goals of the United States for the 1970-80 decade, using questions for which the true answers are not known and, therefore, value-judgments must be made. In the demonstration phase of the exercises, successive iterations with information feedback resulted in higher group-accuracies, when estimating the true numerical values of the answers to given questions. Furthermore, the individual numerical estimates exhibited a lognormal-distribution behavior. These findings are consistent with those obtained by other experimenters on applications of Delphi techniques. In the latter phase of the exercises, a consensus was readily obtained in regard to identifying the educational goals deemed most desirable and beneficial to the general welfare of the nation. However, considerable difficulty was experienced when value-judgment attempts were made by the group to categorize these goals according to four criteria: desirability, benefit, feasibility and cost. Here, also, our findings were consistent with those of other experimenters. INTRODUCTION

A SERIES of Delphi exercises were conducted at six weekly classroom sessions at the Polytechnic Institute of Brooklyn Graduate Center. The size of the class varied from 14 to 17 male, graduate-engineering students. The purpose of the exercises was to familiarize the students with the effectiveness of the Delphi method in formulating group judgments on subjects about which exact knowledge is known, and to consider the application of this method in a classroom exercise concerning questions about which exact knowledge is presently unknown. PHASE

I. DEMONSTRATION OF METHOD FACTUAL JUDGMENTS

USING

Anonymous response iteration and controlled feedback of information Method. Following the methods of Brown et al. [l], questionnaires (Fig. 1) were given to each student. Responses by the students were anonymous, but the group made personal 363

L. R. DOYON, T. V. SHEEHANand H. 1. ZAGOR

364 Question

number

Question

Answer

What was the number of telephones

(in thousands)

in

How many million barrels of beer were produced in the U.S. in 1966 1

How many millions of dollars were expended for public elementary and secondary education in the U.S. in 1967 ?

What was the total US. billions of dollars?

5.

IABCDE i

Budget

income

in 1968

in

How many thousand new private housing units were started in the U.S. in 19697

IIABCDE In 1967 what was the average number of pounds of milk consumption per person in the U.S.? What was the total motor fuel consumption of gallons in 1967 in N.Y. State?

8.

IABCDE i

in millions

What was the total number of deaths from falls in the U.S. in 1967 ?

IIABCDE

9.

How many elementary and secondary school teachers were there in the U.S. in 1967 ?

10.

How many radio and TV broadcast stations were operating in the U.S. in 1967 1

FOG. I. Sample questionnaire of first iteration showing boxes along left margin for student to circle appropriate reason for changing (I) or not changing ([I) original answers (see Table 2).

records of their answers to be used later as references. The correct answers were not disclosed until the exercise had been completed. In the second round (first iteration) only the 25th, 50th (median) and 75th percentiles of the group’s responses for each question were disclosed to all students as controlled feedback information (see first three columns of Table 1). The students were asked to reconsider their original answers in comparison with this feedback information. They were then asked to circle a letter in either row I or row 11 in the boxes on the questionnaire indicating the reason for changing or not changing their original answers (see list of reasons in Table 2). Table 1 only includes data from students who participated in both rounds because two students were absent at the time of the first round, and one student was absent from the second round. Table 3 includes data collected at both rounds. In actuality, Table 3 is valid since a student answering the second-round questionnaire without having answered the original questionnaire has all the feedback information necessary to make a good judgment-hence, the effect can be considered to be the same as if he had previousIy answered the first-round questionnaire.

365

Classroom Exercises in Applying the Delphi Method for Decision-making TABLE ~.DELPHIMETHOD-FEEDBACKANDITERAJION CIN~L~JDESONLYIDENTIFIABLESECOND ROUND DATA)

Results of original questionnaire 25th percentile 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

150 3 100 10 80 50 200 1000 4OWOO loo0

50th (median)

Results of first iteration

75th percentile

5088 50 982 208 580 208

1,ooo,ooo

1000 so00

z 25,000

800,008 so00

25th percentile 500 17 500 100 100 52 500 1000 750,000

240 10,000 500 1000

1Pn~

75th percentile

50th (median)

3000 1000 30,000 260 750 500 3000 25,000

low

50

58881 190* 500

4w loo0 5800 1,080,808+

1,ooo,ooo

10,000

10,000

Summary of Results with Feedback and Iteration: Number of questions Convergence *More accurate 3 Same 5 t Less accurate 2

Correct answer 21,758 110 31,511 165 1490 286 5057 19,800 1,854,700 6253

Divergence

Both ends 5 One end 2 (low) Neither 1

0 2 (high)

Totals: (Nine persons filled out this part of the questionnaire) Number of changes of answers Number of no changes Not indicated whether changed or not Total

5 90

Breakdown Reason for changing answer (I) (a) I had misread the question (b) I made a mistake in computation (c) I remembered some additional facts (d) My estimate was too far from the group median (e) The other members of the group are likely to know more about the question than I do Total number of changes Number not indicated:

Reason for not changing answer (II) 6 7 12

(a) I believe my original estimate (b) The other members of the group are not likely to know more about the question than I do (c) No good reason to change (d) My estimate was close to the group median

30 3 !

10 5 40

(e) It would be more effort than it is worth to 1 rethink the answer Total number not changed 45

5.

Observations. Tables 1 and 3 reveal two phenomena

which occurred at the second

round: (a) the accuracy of the answers improved ; (b) the spread of the answers decreased. A third phenomenon (see Fig. 2) is that the numerical values of the answers exhibit a lognormal distribution. These findings concur with those obtained in experiments conducted by Dalkey [2] (see Fig. 3). For some unknown reason, however, Dalkey [3, 41 failed to recognize that the histogram for his second round (see Fig. 4) is also a lognormal distribution, but with a different standard deviation than shown for the fitted curve. Dalkey [5] P

366

L. R. DOYON,T.

V. SHEEHAN and H. I.ZACOR

states in his report that his second-round This is not so-it is lognormal.

curve “is distinctly

TABLE 3. DELPHI METHOD-FEEDBACK

3 100

3. 4. 5. 6. 7. 8, 9.

:I: 50 ?.oo loo0 400,000 1000

10.

Results of first iteration 25th percentile

1,ooo,ooo

150 50

5% too loo 52 500 1000 800,000 2500

1,ooo,ooo 10,000

Summary of Results with Feedback and Iteration: Convergence Number of questions _.___~_ -- .-___II *More accurate 4 Both ends 7 Same 5 One end 2 Neither t Less accurate 1

75th percentile

50th (median)

500

240 10,000 500 1000 700 6000 25,000

982 200 500 200 1000 5ooo 800,000 5000

from lognormat”.

ANDITERATION (INCLUDES ALL DATA)

Results of original questionnaire --.-_-___~~ 25th 75th 50th percentile percentile (median) I. 2.

different

3ooot II* 5000* 200 500 200 1000 gooo* I ,ooo,ooo* 5000

10,000

240 8000 300 640 500 5000 IO,000

1,@wo@J 6000

Correct answer 21,758 110 31,511 165 1490 286 5057 19,800 1,854,700 6253

Divergence .-.. -..

--__--.

0 0 I

DELPHI METHOD RESULTSOFQUESTIONNAIRE

0.5 INITIAL

ROUND #I

ANSWERS

0.4 > b z ti

BEST-FIT

0.3

^x z

CURVES ROUND #2

ANSWERS

0.2

0 0

0.5

1.0

1.5 NORMALIZED

FIG. 2. Distribution

2.0

2.5

3.0

3.5

4.0

ESTIMATE

of answers from true values (data summarized

in Tables I and 3).

Tecktzical discussion. Hald [6] has shown that a lognormal probability density-function curve is a negatively-skewed concave curve when the variance is high (viz. 0 M OS), but resembles a normal (Gaussian) curve with negligible skewness when the variance is low (viz. (I w 0.1) (see Fig. 5). Engineers who have made studies in maintainability recognize this lognormal-curve peculiar behavior [7-g]. It has been shown that the human effort

Classroom Exercises in Applying the Delphi Method for Decision-making

367

or process of placing objects in logical categories, which is the type of process demanded of participants in the Delphi method, usually leads to a lognormal distribution [lo]. Results. During the second round of our classroom exercises the medians shifted towards the true answers and the spreads decreased, as expected. This resulted in a shift from a highly-skewed curve to one that is less skewed, still lognormal, but starting to resemble a normal curve (see Fig. 2). This phenomenon agrees remarkably well with that observed by Dalkey (compare Fig. 2 with Figs. 3-a& 4).

03

1

j-l

OBSERVED

-

LOGNORMAL 1

,-(LoGH2

x42?

0

0.5

1.0 NORMALIZED

FIG.

0.B

3.

Distribution

2:o

n

NORMALIZED 4.

Distribution

2:5

ESTIMATE

of results obtained in experiment by Dalkey (first round).

1

FIG.

2

OBSERVED

ESTIMATE

of results of first iteration by Dalkey (second round).

3:O

L. R. DOYON,T. V. SHEEHANand H. 1. ZAGOR

368

0.160.160.140. I2 -

0

IO

I 30

20

I 40

I 50

X FIG. 5. Three logarithmic-normal

distribution

curves.

A detailed breakdown of the data of Table 3 yielded some interesting findings, not shown herein. In a class of 17 students, one student changed eight of his original ten answers during the second round. A detailed study showed that this “swinger’s” performance (deviations of his original answers from the true answers) was, however, no worse than the class average. There were two “stand-patters” (persons who refused to change their original answer for at least eight of the ten questions) in the group. One of these refused to change nine of his ten original answers and gave as a reason that he believed his original estimate was correct for each of the nine answers. Ironically, a detailed study of this person’s performance showed that, of his ten answers, two were better and eight were worse than the class average. The other stand-patter indicated various reasons for not changing his answers. This person’s performance, though, was not significantly worse than the class average. The important point to stress is that these detailed findings illustrate dramatically the effectiveness of the anonymous-response, iteration and controlled-feedback-information feature of the Delphi technique in minimizing the biasing effects of the “dominant individual”, semantic “noise”, and group pressure for conformity [I I]. In spite of the stand-patters and the swingers in the group, the anonymous-response feature led to much more accurate answers than those obtained in the face-to-face group discussion and debate exercise. This latter exercise is reported in the paragraph that follows. Face-to-face

structured group responses

To complete the Delphi exercise on subjects for which exact knowledge is known, the class was separated into three groups. Again, following the method of Brown et al. concensus was sought from each group through discussion and debate on five questions. Again our findings agreed with those of Brown et al. Table 4 summarizes these results. As the reader will note, the accuracy of the answers leaves much to be desired, with, perhaps, the sole exception being the second question. An amusing highlight of our classroom face-to-face group exercise was that a member

Classroom Exercises in Applying the Delphi Method for Decision-making

369

of one group was so adamant in his disagreement with the concensus of his group that he gave his own separate estimates on two questions. Ironically, both of his separate answers were further from the correct answers than those of his group. Here we observed the “dominant individual” described by Dalkey. TABLE 4. DELPHI METHOD: RESULTS OF FACE-TO-FACESTRUCTURED GROUP RESPONSESTO FIVE QUESTIONS

Group I I. What is the area of the U.S.S.R. in millions of square miles?

Group II

20

Group III

correct answer

5

8.65

:;+ 2. How many popular votes did Nixon get in Texas in the 1968 presidential election? 3. What was the U.S. gold production for the year 1967 ? 4. What was the highest price for a seat on the New York Stock Exchange in 1929?

1,ooo,ooo

1,207,417

2,ooo,ooo

900,ooo

$8&n (lo)*

$75m

t75m

$55.4m

~400,ooo

$100,000

$625,000

25

28

19

$250,000

5. In ranking of states by arca in descending order, what is the rank of Missouri?

+ These two values are dissident votes by one member of the group. It is interesting to note that both values are further from the correct answer than the consensus of the group. t In a group of four persons there was an even split of two persons replying “8” and the other two replying “5”. PHASE

II.

APPLICATION

OF THE

PRINCIPLE

TO VALUE-JUDGMENTS

Background

In the Rand report, Dalkey briefly mentions the application of the Delphi procedure to value-judgments [12]. The validity of the application of Delphi in an area where the true answer may not exist seems intuitively sound, but, nevertheless, obscure. In spite of this and encouraged and impressed by the work of Helmer [13] in applying the Delphi technique to educational planning, we proceeded to formulate an approach to selecting national educational goals for the 1970-80 decade on which the Delphi technique could be implemented. Selecting a set of goals

Not wishing to duplicate the work of Helmer, a set of twenty tentative goals (see Fig. 6) was derived from general knowledge, newspaper and magazine articles, and interface with educators. The set contained contemporary, conventional and controversial ideas about education. The twenty goals were then presented to the class and anonymous responses elicited. A numeric value relating to a qualitative judgment was requested (see Table 5). If a majority of the class suggested a goal be implemented, then that goal was selected for the second step. TABLE 5. “VALUE”

Value 1 2 3 4 5

WEIGHT FOR QUESTIONNAIREIN FIG. 6

Meaning Strongly recommend implementation Recommend implementation Recommend consideration as a secondary goal Do not recommend implementation Strongly against implementation

Legends:

Selected goals

Value number

Sum

Response number

1

i

3

4

5

6

7

8

14 3 1 5334251 5 3 2 2 1 4 3 1 32422522543242433435

r-5

2

10

11

12

234322334334

9

I 1

23

45

37

26

1 3 15 61

30

NYNS

S, secondary goals;

YNYYNS

3

47

3

2

3

3

3

3

14

13

7

36

3

4

33 2

40

NYS

0

2444483193

51

5111111111

Y

0

2

32 1 3

39

3

4

3

N

S

0

8 4

23

1

2

1

18

YYSN

1

4 4

34

2

1

2

17

Fro.6

522103 2 0

2

46

2

4

3

1.5 16

25211123112 1 1 1 22525115

N, not selected

4

30

5 1 3 2 324533344441122 1 1 3

9 1 1 6034112 23540653424371 223641 5 620701421141 401006003071

2

50

3 3 1

3423535323531235 2334223213 3 2 2 4 3 3435142432133

113214211 4 1 1 5 4 5 -3 1 2

41

Y, primary goals;

11 12 13 14

a;5143 105 9

Z&21441 4 1 3 4 2 4 512422532334134 2 2 2 3 2 2 x 43422432335243423233 42433531 445442332235

2 3

1

Goal number

TABLE 6. DETAILED TABULATION OF RESULTS SHOWNIN

0

4 1

33

3

3

1

19

4

3

1

48

2

3

4

20

-

37 47 65 35

65 45 57

56 54 61 66

57 57 64

Sum 2

I’52733 2 5 0 7 2 7 4 5 2 0 6 1 4 2 2 3 E 3 4 4 8 10 6 94214 2 3 -14’ 1

1

6 3

8 5 6 4 8 6 8 8 7 2

3

6 0

4 5 5 5 5 6 2 5 1 1

4

3 2

1

1 3 0 2 1 3 5 0 0

5

Value number

371

Classroom Exercises in Applying the Delphi Method for Decision-making I. Schools should be racially integrated 2. 3. 4. 5. 6. 7. 8. 9. 10. II. 12. 13. 14. 15. 16. 17. 18. 19.

20.

even when the surrounding communities are not racially integrated. There should be a rating system for teachers regardless of tenure. Education courses should not be required for teachers’ licenses. Schools should provide compulsory sex education. Schools should provide a drug awareness program. “Frill” courses such as Drivers Education should be withdrawn from school curricula. Schooling should begin at age four to take advantage of the child’s natural curiosity. Foreign language study should be introduced in the primary grades. Elementary classes should be ungraded so that each student could proceed at his own pace. Programmed instruction techniques should be introduced wherever practicable. The government should provide financial aid to sectarian schools. The so called “educational-industrial” complex should be held accountable for demonstrating the effectiveness of government funded programs. Vocational education should be extended to the junior college level. Junior colleges should award specialized degrees recognized by industry. Colleges should grant credit to students engaged in volunteer community projects. Colleges should coordinate with industry on a one year training program for students. There should be a separate degree, equivalent to the PhD, which does not require esoteric research. Universities shnuld not base advancement of their instructors on the “publish-orperish” principle. There should be a national program of sabbaticals allowing professionals and businessmen to spend every fifth summer at a university studying new techniques and ideas in their fields. College level courses should be graded complete or incomplete rather than the present system. Legend : Y-primary goals ; S-secondary goals ; N-not selected.

FIG. 6 Results of questionnaire

N Y(I) N S Y(3) N Y(5) Y(4) N S N Y(7) S Y(6) N S Y(8) Y(2) S N

for educational goals for the 1970-80 decade (see Tables 5 and 6).

Each of the answers was equally evaluated even though some responses suggested that few goals be implemented. Table 6 illustrates the results. In brief, of the original twenty tentative goals, the exercise yielded eight primary goals and five secondary goals. It should be noted that goals generally considered to be controversial were not selected as primary goals. This may have been largely due to the social and economic structure of the group which was white, middle-class, technical and professional. Ranking the primary goals For the second step the group was told how eight primary goals were selected. The group was in four categories: desirability, feasibility, benefit was discussed and the following instructions were

the goals were categorized and how the asked to place these eight primary goals and cost. The meaning of these words given:

(a) desirability-weigh subjectively according to one’s own personal best interest; (b) feasibility-weigh according to estimated ease of implementation; (c) benefit-weigh objectively according to the good and best interest of the nation as a whole; (d) cost-weigh in order of increasing estimated cost. The results of the second step were not as clear-cut as the results of the first step. Figure 7 shows the rankings as a result of the second step. It should be mentioned that ranking is not a good quantitative technique and is recommended only when no other

DOYON, T. V. SHEEHAN and

L. R.

372

H. I.

ZAWR

analytical technique is applicable [14]. Categorizing forces a ranking fails to quantify the relative difference between ranks. R ANKS

1. There should be a rating regardless of tenure.

system

for

teachers

should not base advancement 2. Universities their instructors on the “publish-or-perish” principle.

(1 THF fDU GH8l

of alternatives,

but

CIRC

DESIRABILITY

DESIRABILITY

of g

65

63”

3. Schools program.

should

provide

drug

awareness 0

4. Foreign language study in the primary grades. 5. Schooling advantage

a

should

be introduced

0

should begin at age four to of the child’s natural curiosity.

6. Junior colleges should award specialized recognized by industry.

47

63”

take

0

degrees

complex 7. The so called “educational-industrial” should be held accountable for demonstrating the effectiveness of government funded programs. 8. There should be a separate degree. equivalent to the Ph.D., which does not require esoteric research. Notes -Top number within each box is summation of “value” weights (see Table 5). -Second number (circled) is the assigned rank. *Tie-rank made on basis of cost results. tTie-rank made on basis of feasibility results.

77 @

85 0 L

L

FIG. 7. Summation of weights and the final rankings of chosen goals. Testing for correlation The results of the rankings obtained in the second step were tested for correlation with the rankings of the first step. Table 7 lists the results of the correlation analysis. Only two rankings show any significant correlation. From these results one may surmise that feasibility was the dominating factor in the ranking. Secondly, in the students’ minds there seems to have been some similarity between the subjective and objective evaluation requested in the second step. However, it must be stated that even in these two cases the correlation could not be considered to be highly significant. Deriving a cost-benefit equation One can appreciate the difficulty encountered in estimating the knowledge of the group in this area was limited. Also, because benefit and desirability, benefit was deemed the more altruistic vide the most reasonable measure for ranking in a cost analysis. Deriving an indicator for forecasting the opinion of a group

and ranking costs, since of the similarity between criterion and would prowith social backgrounds

Classroom Exercises in Applying the Delphi Method for Decision-making

373

similar to those of the group which took part in these exercises also appeared highly desirable in devising a cost-analysis procedure. In our cost-benefit exercise, estimates of benefit were based on the total population affected by the implementation of each goal. Both the expected number of people affected and the expected cost were extracted from data published in 1969 World Almanac and Book of Facts. Some extrapolation was necessary since detailed breakdowns were not available. Table 8 shows these expected benefit and cost values. Table 9 presents the empiricallyderived cost-benefit equation. Note that only two rankings match the class results of the second step. This may be due to the inability of the group to rate the expected cost accurately. TABLE 7. CORRELATION ANALYSISOF RANKINGS

Rankings

Step No. 2 results with Step No. 1 results

Step No. 2 results only

Correlation coefficient

Desirability

0.513

Feasibility

0.599*

1

Benefit

0.449

i

cost

0.282

Desirability with feasibility

0.095

Desirability with benefit

0.693*

Desirabihty with cost

0.008

Feasibility with benefit

0.385

Feasibility with cost

0.290

Benefit with cost

-0.433

* Denotes significant correlation. CONCLUSION

The Delphi exercises described in this paper were conducted primarily for tutorial demonstration purposes. No attempt was made to conduct these exercises as controlled experiments for proving or disproving conclusions reached by other authors. The favorable outcome of these exercises on factual information was most gratifying and has strengthened our credence of findings attained by other authors, principally Brown et al. Encouraged with this favorable outcome, we subsequently proceeded to formulate an approach for deriving a cost-benefit equation. This equation would be a meaningful index and would serve as a first step in selecting national educational goals for the 1970-80 decade to which value-judgments subsequently may be applied. The work of applying the full Delphi technique, with at least two iterations for each step in deriving this cost-benefit equation, remains to be done.

L. R. DOYON, T. V. SHEEHANand H. I. ZAGOR

374

TABLE8. COST-BENEFITESTIMATES Benefit Number of people benefited (millions)

Primary goal

I

52.0 2.0 7.0 0.5 52.0 155.0 30.0 0.1 4.0 0. I5 I .5 52.0 155.0 0.01

2 3 4 5 6 7 8 * ($m)=($

Students Teachers Students Professors Students Citizens Students Teachers Students Teachers Students Students Citizens Students

cost ($m)* 30 6 300 1000 2500 25 40 5

millions).

TABLE9. COST-BENEFIT EQUATIONS Cost index

Benefit index 1. High (more than 60’;/, of the nation benefited) 2. Medium (between 20 and 60% benefited)

1. Very low (less than JJOm) 2. Low ($lOm-99m)

3. Low (less than 20% benefited)

4. High (more than S999m)

Goal

I 2 3 4 5 67 8

(Benefit index)

v

2 3

Y :i

I 3 3

,,‘ :’

31 3

-7 \.I

(Cost index)

3. Medium ($lOOm-999m)

Product

2

I 3 4 4

4 3 3 I2 12

Calculated rank

Step No. 2 rank

5 2* 3* 7t

2 I 5 7:

8t

8: : 4

2

26

6

1

3

i*

* Tie. t Tie. $ Step No. 2 rank in agreement with equation rank.

REFERENCES I. 9. BROWN, S. COCHRANand N. DALKEY,The Delphi

2. 3. 4. 5.

Method-II, Structure of Experiments, Memorandum RM-5957-PR, pp. 5-9, 13-42, The Rand Corporation, Santa Monica, California (1969). N. DALKEY,The Delphi Method: An Experimental Study of Group Opinion, Memorandum RM-5888-PR, p. 20, The Rand Corporation, Santa Monica, California (1969). N. DALKEY,The Delphi Method: An Experimental Study of Group Opinion, Memorandum RM-5888-PR, p. 26, The Rand Corporation, Santa Monica, California (1969). N. DALKEY,The Delphi Method: An Experimental Study of Group Opinion, Memorandum RM-5888-PR, p. 27, The Rand Corporation, Santa Monica, California (1969). N. DALKEY,The Delphi Method: An Experimental Study of Group Opinion, Memorandum RM-5888-PR, pp. 25, 28, The Rand Corporation, Santa Monica, California (1969).

Classroom Exercises in Applying the Delphi Method for Decision-making 6. A. HALD, Statistical Theory with EnginLcs:+wpApplications, p. 161. John Wiley, New York (1952). 7. H. I. ZAGER and R. L. BOUAIRD,Lognormal distribution and maintainability in support systems research, Nav. Res. Lo&. Q. 8, 343-356 (1961). 8. L. R. DOYON and K. E. APPLEY, On the question of A4 incentive contracting, 1967, Proceedings by the Society of Automotive Engineers. Ann. Reliability Maintainability 6, 610-622 (1967). 9. J. AITCHISONand J. A. BROWN,The Lognormal Distribrrtion, p. 10. Cambridge University Press, New York (1957). 10. A, S. GOLDMANand T. B. SLATTEKY,Maintainability: .1 Major Element of System Effectiveness, p. 50. John Wiley, New York (1964). 11. N. DALKEY,The Delphi Method: An Experimental Study ot Group Opinion, Memorandum RM-5888-PR, p. 14, The Rand Corporation, Santa Monica California (1969). 12. N. DALKEY,The Delphi Method: An Experimental Study of G:o~.p Opinion, Memorandum RM-5888-PR, pp. 73-75, The Rand Corporation, Santa Monica, California (1969). 13. 0. HELMER,The Delphi Technique and Educational Innovation Inventing Education for the Future, edited by W. Z. HIRSCH, pp. 74-83. Chandler (1967). 14. D. W. MILLER and M. K. STARR,The Structure of Human Decisron_:, p. 89. PrenticeHall, Englewood Cliffs (1967).

375