- Email: [email protected]
The use of Bayesian techniques in Delphi forecasts
The Use of Bayesian Techniques in Delphi Forecasts
Procedures are developed which allow an estimation to be made of rhe uncertainty associated with Delphi forecasts and which allow the judgment and intuition of the decisionmaker wing the forecast (and/or statistical data on the accuracy of past Delphi experimen&) to be incorporated into and impact upon the forecct rest&s. The methodology utilixd relates the conhdence probability of the forecast of an event to the dispersion of individual estimatck of the elapsed time for the event to occur. The confidence probabiliti~ of individual es~imatcts are then modified by the outcome of the group estimate and the decisionmaker’s subjective judgment through the use of Bayesian techniques. The final forecat is obtained as the sum of the individuaP estimates weighted by their modified confidence probabilitks.
u.Introductjon Delphi Q~recastin~ procedu~ are designed to elicit opinions from a group with the aim of formulating a kmup judgment for subject matter where precise information is lacking. Since the early development of the Delphi fbrecxuting technique 11J and its initial application to techndogical forecasting [2]! there h*zsbeen concern over the accuracy of the forecasts and the potcntiai increase in uncertninty in estimates as the fbrecast events become more remote in time. The rc&nt literature is-61 reflects these ccmcerns. The uncertainty which is associated with most syitems analyses can be generally classified u statistical uncertainty, such as variations in costs or other input parameters, or yes-no uncertainty, such as whether or not a new technology or research breakthrough wifl occur by a specified time. It is the latter type which is characteristic of Delphi forecasts. Statisticai uncertainty generally has the effixt of reducing expected benefits, but rarely will it have the catastrophic effects which can sometimes result from yes-no uncertainties. It is customary in syatem.uanalyses involving uncerrainties the yes-no type, to make: evaluations ofaltemativ~ which are prcdicatcd an various uncertainty assumptions. Thcfc altcmatjwy are then presented to a derisionmaker whtise final decision is made on the basis of a subjective judgment concerning the rclat.ive likelihood of the alternatives. The final evaluation of the uncertainty is b,ased on thr decisianmaker’s intuition and Judgmer& It would appear that the resutts of Delphi forecasts could be made somewhat more pglpable, if they were presented in such a context, because when Delphi fareczxstsfind practical application in systems analyses, evaluatir>n of the credibility of their predictions rests fina& in the hands of a decisionmaker’s subjective opinion. iz desirabie adjunct to the usual Delphi forecast would therefore appear to bc an
of
b which subjective judgment and intuition tramof the pfarm their malts. Such an analytical description can be provided through the application of Bay&an statistical techniques. It is the objective of the work d&W herein to dewelop these proocdures,
adytid
d&p&n
In. Ir3esmipltion OfP~dUrezr below will be develop& in the Context of one round of a mht procedures d Delphi forecast in which individual estimates from a group are obtained and a group jiudgment derived. Questions concerned with the design of the Delphi experiment such as group size, number of rounds, the use of self-ratings f4), Gtl not be explicitly disctlsacd; it wilii be assumed that all such factors have brrn considered and the DeIphi experiment has been structured to minimize inlherent errors. It till be hypothesized that individual estimator have, in the back c,f their minds, an uruier’lying probability distribution related to the quantity in question and that this distribution ~31 be lognormal in character. Although sufficient experimental work has not be-n cornpitted to definitely establish the shape sf the underlying distribution curve for in&l e&nates, it is stated by Dalkcy 131 that “the observed distribution of answm on individual questions are compatiMe with the raAJunrptionthat they are lognormal,” and data given by Martinez [7] related to the diitribution of,g~up estimarcs tend also to add support to thii assumption. Argumc;rts based on the central timi; theorem will be presented subsequently which further justify this hypothesis. It wil1 be assumds that two estimates of the date of an event have been obtained from e~tch participant in the Delphi experiment: 41j the number of years fmm the current year estimated to define a 6.h left&l probability &at ~thefoxcast event will occur:, and (Z!)the number of years fmm the current date estimated to d&nr: a 0.1 right-tail probshiry Of 0CXWWncefor the f&ecast event. In other words, in the estimator’s Opinion the odds would bc eight to two that the number of years which will elapse before the event OCCURS ~oulcl fd between these eariiest and latest estimates (see Fig. 1). Then estimat+s GUL then be used to determine the median elapsed time of the assumed lognormal - - -_.-- - -_ -...-- -__ _.. -__.-__ _. -* .._._.___ ~^~.~--._~-.__--~-L”--i
I
Fig.
1
distribution. The mean, p, and the standard deviation, u, of the normal underlying the lognormal distribution can be computed from In X0.1 +-in Aij.0 pL” 2 u=
In &.B-- In X0.1 2.56 ”
distribution
(1) (2)
where X0.1 and &.o nfcr to the 0.1 and 0.9 tail estimates of the elapsed time required for the occurrence of the forecast event, Thcsc expressions are derived in Peswmier [8:j and a diwukon of the properties of lognormal distributions is presented in Aitchinson and Rrown [9]. T’he median estimate ofthe elapsed time for the occurrence of the: fortcast event can be caIcul.atcd from x?I ==~-w IA (3) where & ===ertimateof the median elapsed time. The lognwmal density function can be expressed as (see Pessemier [8] and Croxton and Cowdcn [IO]) --(In x--/q (4) 2,2 P W -& =P I [ If the forecat distributions arc normaliied by dividing by the median, p will be zero zmd the density function becomes p (Q,=E where A’,,=
[ -$j$q
(5)
Number of ye;wi bdore occum?ce of forecast event Median elaplstxl time
The above expression is plotted in Fig. 2 for variaus valucp of a. The probability that the individual estimate will lie between plus and minus I@‘& of the median will be adopted as a rneasum of the confidence wciated with the individual estimates. This probability can be determined by constructing on lognormal graph paper
a rcugm&tive fmluency plot of the individual estimate from a lcnowledge of ~aand u as determined fi-om Eqs. (1) and (2) and the calcuiati vaiuc of the median from Eq. (3). Two points determine the plot; viz., the median ir located at the 0.5 cumulative fkqacncy point and.a vaiue for the 0.84 cumulative frequency point (see Pessemitr [8j) is determined Corn (6: &Lar==~P (P-4-4 Once the plat is constructed, the probability that the estimate will lie bMvcen plus or minus 10% of the median can be determined. It should be noted that the measure of coafidence employed has the property that its magni?ude is inversely related to O, dccwasing as u goes up. This characteristic is in agrecmznt wtth the experiments of Dalkcy [3] which showed that the mean group error misted with a Delphi forecast is relateci to the extent of the agreement the members of the grip display in their answers (as mc;ilrured by the standard deviation of the estimate). A relative measure of the confidence of the individual estimate can be obtained by normaiizing the individual canfklence proba.biliti= such that the confidence probabilities of all the group’s individual dmatea sum to unity. Assuming the dktribution oftbe graup estimates will be iognormal (see Dalkey [3) and Martino [7]) ) the pt: and PC shape parameter af the group’s cstirnat.es can be determined by ordering the individual median e&mates (as determined Corn the 0.1 and 0.3 tai! cstirnata;) a~;d plotzing them on eumulatiuc: 1~~04 probability paper as discussed by Pescmier [S]. The shape parmeters can brs determined from a ccumulativc lognormal frequency plot by noting the points at whish the straight cumulative distribution curve croJsE3 the 0.5 and the 0.84 tine-s on the probability scale andi using the reMor ‘p
where F0.a and F o.sl respectively,
denote
the cumulative
frequency
pollntr where
the
dktribution curve crtxscs the 0.84 and 0.5 lines on the probability scale, The median, X,, is quaI to theFg.6 point, and w can be determined frcjm Eq. (3). ff the data are normalized by the group median in the cumulative plots, ~0 becomes zero. From a knowledge of the cumulative frequency distribution of the voup estimate, the probability that the group estimate will lie between plus and minus 100, of the median can be determined .zs dkcussed previously. Since the hypothesis that individual and group estimations follow a lognormal distribution hm been central to the development of the preceding mtthodolog-y, some further discussion justieing this hypothesis will be given. In the Delphi experiments of Dalkey 131 the distribution of initial round group answers was shown to follow a lognormal distribution with ag = I + as shown in Fig. 3. When the data fmm 13) for the second round answers arc compared with a Io.gnormal frequency curve computed with a lower ur; value of CLMS as shown in Fig. 4, they also appear to follow a lognormal distribution remxtably well. ‘I’he iterations between the first and second rounds reduced the dispersion of the distribution as indicated by the reduced UC: value. ‘ik agreement of the cuperimental distributkn ofthe groupestimates with the lognormal distribution function indicates that an individual forecast estimate is determined by the product rather than the sum ofa great many random factors. ‘I’he lwarithm of the forecast would then be the sum of independent random factors (cf [J]). As the number of these random factors increases, the distribution of the logarithm of the forecast would approach a normal &tribution according to the centA limit theorem, and the forecast would be logrnxmally distributed, If there were a large numkr of’ estimates in a
group forcEaJt estinz;ntce would imply tit the individual forecsrsta are lognormally distributed and that correlation exkua between the individual forecast estimates {otherwise the group ~t4rna.t~ would be normally distributed). Application of the proccdu~~ developed in the precxxiinlg discussion to Delphi forecasts1will yield v&es for the individual median estimtes, a relative confidence probabilit3.f for ea& individual estimate, a median group atimatc, and a confidence probabiait-v for &c group estimate. With thex data in hand, Bayes theorem of c ptl~tiori probabtity can then be applied to modify the rcrults by the in tuition and judgment of t.hc final decisionmaker. {As employed herein the term “final decisionmaker” is used interchangeably to represent such possible final decision&n as the conductor of the Delphi experiment, a group of experts s&x ted from the Deiphi experiment, or the perslon tmpioying the results; i.e., any individual 10’group to be held prEsponsib3efor the resdts *f the forecasts and who should thereby have the prerogative of exercising personal judgment.) The proczdure is illwtrated a5 follows. Athat &red individual forecast estimates and the group estimatesof the median elaplrad time b&ore the CMXU~~CF a spa&d event have been established and the values shown in Table I kve bexn obtained.
of
Table 1
n'iwtmtion
ofhKedUrc!
With these data Wore him, tbs decisionmaker must now mtisfy himself concerning the extent to which the ra&s can be believed. Suppose the dccisionmakcr has some doubts about the vaSidity of the gpoup results and he is unwilling to accept them completely. Ht then USB his judgment, inttition, past experience, and/or historkal stati&.aJ dam on the auxzracy of past IM;phi experiments to establish a truth table as shown in Table 2 whi& cl&in- the co&dence that he places in the group estimate. Bayczt’ thcx~rem (see ‘84zrq~ [II] and Ma&man fP2)) as applied to this problem says that the pf&&ilit~ that tic clapsad rime for the forecast event to occur is really 25 years, given the group
The chancesthat the group crtimntewill indicate the median cIapd time arc
WikHn the true median d ap3edtime(in years) Irrdl)fil:
10 4;
10
20
25
0.60 0,4Q 0.10
0.30 0.70 0.30
0.10 0.10, O.ti@
estimate has indicated that it is 20 years, is equal to the probability that the groupestimate wodd say the elapsed time is 20 yc;ars when it is really 23 years; multiplied by the confidence probability of the individual estimate of 25 years; divided by the probability that the group estimate is going to say that the elapsed time is 20 years. whatever it is. This can be written in statktical notation as
Using the same procedure, revised confidence probabihties of the individuaf the other median dates can be computed as given in Table 3.
c%timates of
Table 3 Illustration of f%we&rr Hci isfxi confi~lcnre prot-rability esi:imirte fswm Ekyesian aruilysis -A.a. 16
0.37 0.47
The final expected estimate of the date of occurrence of the event can be computed a~ shown in Table 4 for the three assumed individual e&mate; (in a real case, the number of kiividual estimates would be larger).
N. Cuncludiag Remarks The primary advantages of ahe procedures developed in the preceding discussion are : they allowsan estimate of the uncertainty of a Delphi fkecast to be indicated, and (2) they allow the decisionmaker’s judgment and intuition as well as statistical data on the accuracy of past Delphi experiments to be incorp,..+red into and impact upon the forecasts being develqed. The measure of uncertainty adopeed herein has been the probability that the forecast date of occurrence will f&l within plus or minus IO?& of the median estimate of the forecast of the elapsed time befcre a specified future event happens. The argument for the use (1)
of this parameter as a measure of uncertainty rests on the results of the pioneering work related to Delphi crxperiments in f3] in which it was indicated that: ( I) observed distributiions of armers of individual Delphi questions were compatible with the a~umption that they are lognormal and (2) a relationship exists between the standard deviation of .answers on a given question and the mean group error. To the extent that these results
of
A. Wade thkm~,Jr.
Procedures are developed which allow an estimation to be made of rhe uncertainty associated with Delphi forecasts and which allow the judgment and intuition of the decisionmaker wing the forecast (and/or statistical data on the accuracy of past Delphi experimen&) to be incorporated into and impact upon the forecct rest&s. The methodology utilixd relates the conhdence probability of the forecast of an event to the dispersion of individual estimatck of the elapsed time for the event to occur. The confidence probabiliti~ of individual es~imatcts are then modified by the outcome of the group estimate and the decisionmaker’s subjective judgment through the use of Bayesian techniques. The final forecat is obtained as the sum of the individuaP estimates weighted by their modified confidence probabilitks.
u.Introductjon Delphi Q~recastin~ procedu~ are designed to elicit opinions from a group with the aim of formulating a kmup judgment for subject matter where precise information is lacking. Since the early development of the Delphi fbrecxuting technique 11J and its initial application to techndogical forecasting [2]! there h*zsbeen concern over the accuracy of the forecasts and the potcntiai increase in uncertninty in estimates as the fbrecast events become more remote in time. The rc&nt literature is-61 reflects these ccmcerns. The uncertainty which is associated with most syitems analyses can be generally classified u statistical uncertainty, such as variations in costs or other input parameters, or yes-no uncertainty, such as whether or not a new technology or research breakthrough wifl occur by a specified time. It is the latter type which is characteristic of Delphi forecasts. Statisticai uncertainty generally has the effixt of reducing expected benefits, but rarely will it have the catastrophic effects which can sometimes result from yes-no uncertainties. It is customary in syatem.uanalyses involving uncerrainties the yes-no type, to make: evaluations ofaltemativ~ which are prcdicatcd an various uncertainty assumptions. Thcfc altcmatjwy are then presented to a derisionmaker whtise final decision is made on the basis of a subjective judgment concerning the rclat.ive likelihood of the alternatives. The final evaluation of the uncertainty is b,ased on thr decisianmaker’s intuition and Judgmer& It would appear that the resutts of Delphi forecasts could be made somewhat more pglpable, if they were presented in such a context, because when Delphi fareczxstsfind practical application in systems analyses, evaluatir>n of the credibility of their predictions rests fina& in the hands of a decisionmaker’s subjective opinion. iz desirabie adjunct to the usual Delphi forecast would therefore appear to bc an
of
b which subjective judgment and intuition tramof the pfarm their malts. Such an analytical description can be provided through the application of Bay&an statistical techniques. It is the objective of the work d&W herein to dewelop these proocdures,
adytid
d&p&n
In. Ir3esmipltion OfP~dUrezr below will be develop& in the Context of one round of a mht procedures d Delphi forecast in which individual estimates from a group are obtained and a group jiudgment derived. Questions concerned with the design of the Delphi experiment such as group size, number of rounds, the use of self-ratings f4), Gtl not be explicitly disctlsacd; it wilii be assumed that all such factors have brrn considered and the DeIphi experiment has been structured to minimize inlherent errors. It till be hypothesized that individual estimator have, in the back c,f their minds, an uruier’lying probability distribution related to the quantity in question and that this distribution ~31 be lognormal in character. Although sufficient experimental work has not be-n cornpitted to definitely establish the shape sf the underlying distribution curve for in&l e&nates, it is stated by Dalkcy 131 that “the observed distribution of answm on individual questions are compatiMe with the raAJunrptionthat they are lognormal,” and data given by Martinez [7] related to the diitribution of,g~up estimarcs tend also to add support to thii assumption. Argumc;rts based on the central timi; theorem will be presented subsequently which further justify this hypothesis. It wil1 be assumds that two estimates of the date of an event have been obtained from e~tch participant in the Delphi experiment: 41j the number of years fmm the current year estimated to define a 6.h left&l probability &at ~thefoxcast event will occur:, and (Z!)the number of years fmm the current date estimated to d&nr: a 0.1 right-tail probshiry Of 0CXWWncefor the f&ecast event. In other words, in the estimator’s Opinion the odds would bc eight to two that the number of years which will elapse before the event OCCURS ~oulcl fd between these eariiest and latest estimates (see Fig. 1). Then estimat+s GUL then be used to determine the median elapsed time of the assumed lognormal - - -_.-- - -_ -...-- -__ _.. -__.-__ _. -* .._._.___ ~^~.~--._~-.__--~-L”--i
I
Fig.
1
distribution. The mean, p, and the standard deviation, u, of the normal underlying the lognormal distribution can be computed from In X0.1 +-in Aij.0 pL” 2 u=
In &.B-- In X0.1 2.56 ”
distribution
(1) (2)
where X0.1 and &.o nfcr to the 0.1 and 0.9 tail estimates of the elapsed time required for the occurrence of the forecast event, Thcsc expressions are derived in Peswmier [8:j and a diwukon of the properties of lognormal distributions is presented in Aitchinson and Rrown [9]. T’he median estimate ofthe elapsed time for the occurrence of the: fortcast event can be caIcul.atcd from x?I ==~-w IA (3) where & ===ertimateof the median elapsed time. The lognwmal density function can be expressed as (see Pessemier [8] and Croxton and Cowdcn [IO]) --(In x--/q (4) 2,2 P W -& =P I [ If the forecat distributions arc normaliied by dividing by the median, p will be zero zmd the density function becomes p (Q,=E where A’,,=
[ -$j$q
(5)
Number of ye;wi bdore occum?ce of forecast event Median elaplstxl time
The above expression is plotted in Fig. 2 for variaus valucp of a. The probability that the individual estimate will lie between plus and minus I@‘& of the median will be adopted as a rneasum of the confidence wciated with the individual estimates. This probability can be determined by constructing on lognormal graph paper
a rcugm&tive fmluency plot of the individual estimate from a lcnowledge of ~aand u as determined fi-om Eqs. (1) and (2) and the calcuiati vaiuc of the median from Eq. (3). Two points determine the plot; viz., the median ir located at the 0.5 cumulative fkqacncy point and.a vaiue for the 0.84 cumulative frequency point (see Pessemitr [8j) is determined Corn (6: &Lar==~P (P-4-4 Once the plat is constructed, the probability that the estimate will lie bMvcen plus or minus 10% of the median can be determined. It should be noted that the measure of coafidence employed has the property that its magni?ude is inversely related to O, dccwasing as u goes up. This characteristic is in agrecmznt wtth the experiments of Dalkcy [3] which showed that the mean group error misted with a Delphi forecast is relateci to the extent of the agreement the members of the grip display in their answers (as mc;ilrured by the standard deviation of the estimate). A relative measure of the confidence of the individual estimate can be obtained by normaiizing the individual canfklence proba.biliti= such that the confidence probabilities of all the group’s individual dmatea sum to unity. Assuming the dktribution oftbe graup estimates will be iognormal (see Dalkey [3) and Martino [7]) ) the pt: and PC shape parameter af the group’s cstirnat.es can be determined by ordering the individual median e&mates (as determined Corn the 0.1 and 0.3 tai! cstirnata;) a~;d plotzing them on eumulatiuc: 1~~04 probability paper as discussed by Pescmier [S]. The shape parmeters can brs determined from a ccumulativc lognormal frequency plot by noting the points at whish the straight cumulative distribution curve croJsE3 the 0.5 and the 0.84 tine-s on the probability scale andi using the reMor ‘p
where F0.a and F o.sl respectively,
denote
the cumulative
frequency
pollntr where
the
dktribution curve crtxscs the 0.84 and 0.5 lines on the probability scale, The median, X,, is quaI to theFg.6 point, and w can be determined frcjm Eq. (3). ff the data are normalized by the group median in the cumulative plots, ~0 becomes zero. From a knowledge of the cumulative frequency distribution of the voup estimate, the probability that the group estimate will lie between plus and minus 100, of the median can be determined .zs dkcussed previously. Since the hypothesis that individual and group estimations follow a lognormal distribution hm been central to the development of the preceding mtthodolog-y, some further discussion justieing this hypothesis will be given. In the Delphi experiments of Dalkey 131 the distribution of initial round group answers was shown to follow a lognormal distribution with ag = I + as shown in Fig. 3. When the data fmm 13) for the second round answers arc compared with a Io.gnormal frequency curve computed with a lower ur; value of CLMS as shown in Fig. 4, they also appear to follow a lognormal distribution remxtably well. ‘I’he iterations between the first and second rounds reduced the dispersion of the distribution as indicated by the reduced UC: value. ‘ik agreement of the cuperimental distributkn ofthe groupestimates with the lognormal distribution function indicates that an individual forecast estimate is determined by the product rather than the sum ofa great many random factors. ‘I’he lwarithm of the forecast would then be the sum of independent random factors (cf [J]). As the number of these random factors increases, the distribution of the logarithm of the forecast would approach a normal &tribution according to the centA limit theorem, and the forecast would be logrnxmally distributed, If there were a large numkr of’ estimates in a
group forcEaJt estinz;ntce would imply tit the individual forecsrsta are lognormally distributed and that correlation exkua between the individual forecast estimates {otherwise the group ~t4rna.t~ would be normally distributed). Application of the proccdu~~ developed in the precxxiinlg discussion to Delphi forecasts1will yield v&es for the individual median estimtes, a relative confidence probabilit3.f for ea& individual estimate, a median group atimatc, and a confidence probabiait-v for &c group estimate. With thex data in hand, Bayes theorem of c ptl~tiori probabtity can then be applied to modify the rcrults by the in tuition and judgment of t.hc final decisionmaker. {As employed herein the term “final decisionmaker” is used interchangeably to represent such possible final decision&n as the conductor of the Delphi experiment, a group of experts s&x ted from the Deiphi experiment, or the perslon tmpioying the results; i.e., any individual 10’group to be held prEsponsib3efor the resdts *f the forecasts and who should thereby have the prerogative of exercising personal judgment.) The proczdure is illwtrated a5 follows. Athat &red individual forecast estimates and the group estimatesof the median elaplrad time b&ore the CMXU~~CF a spa&d event have been established and the values shown in Table I kve bexn obtained.
of
Table 1
n'iwtmtion
ofhKedUrc!
With these data Wore him, tbs decisionmaker must now mtisfy himself concerning the extent to which the ra&s can be believed. Suppose the dccisionmakcr has some doubts about the vaSidity of the gpoup results and he is unwilling to accept them completely. Ht then USB his judgment, inttition, past experience, and/or historkal stati&.aJ dam on the auxzracy of past IM;phi experiments to establish a truth table as shown in Table 2 whi& cl&in- the co&dence that he places in the group estimate. Bayczt’ thcx~rem (see ‘84zrq~ [II] and Ma&man fP2)) as applied to this problem says that the pf&&ilit~ that tic clapsad rime for the forecast event to occur is really 25 years, given the group
The chancesthat the group crtimntewill indicate the median cIapd time arc
WikHn the true median d ap3edtime(in years) Irrdl)fil:
10 4;
10
20
25
0.60 0,4Q 0.10
0.30 0.70 0.30
0.10 0.10, O.ti@
estimate has indicated that it is 20 years, is equal to the probability that the groupestimate wodd say the elapsed time is 20 yc;ars when it is really 23 years; multiplied by the confidence probability of the individual estimate of 25 years; divided by the probability that the group estimate is going to say that the elapsed time is 20 years. whatever it is. This can be written in statktical notation as
Using the same procedure, revised confidence probabihties of the individuaf the other median dates can be computed as given in Table 3.
c%timates of
Table 3 Illustration of f%we&rr Hci isfxi confi~lcnre prot-rability esi:imirte fswm Ekyesian aruilysis -A.a. 16
0.37 0.47
The final expected estimate of the date of occurrence of the event can be computed a~ shown in Table 4 for the three assumed individual e&mate; (in a real case, the number of kiividual estimates would be larger).
N. Cuncludiag Remarks The primary advantages of ahe procedures developed in the preceding discussion are : they allowsan estimate of the uncertainty of a Delphi fkecast to be indicated, and (2) they allow the decisionmaker’s judgment and intuition as well as statistical data on the accuracy of past Delphi experiments to be incorp,..+red into and impact upon the forecasts being develqed. The measure of uncertainty adopeed herein has been the probability that the forecast date of occurrence will f&l within plus or minus IO?& of the median estimate of the forecast of the elapsed time befcre a specified future event happens. The argument for the use (1)
of this parameter as a measure of uncertainty rests on the results of the pioneering work related to Delphi crxperiments in f3] in which it was indicated that: ( I) observed distributiions of armers of individual Delphi questions were compatible with the a~umption that they are lognormal and (2) a relationship exists between the standard deviation of .answers on a given question and the mean group error. To the extent that these results
of
A. Wade thkm~,Jr.