- Email: [email protected]
Assimilation of aggregated inputs into Delphi forecasts: A regression analysis
TECHNOLOGICAL FORECASTING
AND SOCIAL CHANGE 5,243-247
(1973)
243
Assimilation of Aggregated Inputs into Delphi Forecasts: A Regression Analysis J. R. SALANCIK
ABSTRACT To examine the manner in which the individual assessments of a panel of delphi experts are combined into a delphi forecast, the supporting reasons they gave for their forecasts of 40 computer applications were coded into categories of assessed “technical feasibility,” “cost of initiating,” and “benefits or needs provided.” Even though different sets of experts provided these statements in support of their individual forecasts, with some experts emphasizing one aspect and others another, it was found that the median forecast of the entire panel was significantly related to the average proportion of reasons in each category which favored bringing about the development. That is, the delphi forecasts of computer applications suggest that the computer application is forecasted to occur sooner to the extent it was judged to be technically feasible, beneficial to users or society, and not costly to develop. The results indicate further that delphi forecasting among a group of experts has logical validity, and that individual contributions are integrated into a group outcome.
Introduction Delphi forecasting is one of the most popular forms of deriving reasoned expectations about the nature and consequences of emerging developments. An unexamined but critical assumption of the Delphi method is that individual opinions of Delphi participants are assimilated into an overall judgment that integrates the current state of knowledge into a viable forecast. Although it is well established that individual judgment can be represented as a simple additive combination of informational input (Anderson 1968; Hoffman, 1960; Slavic and Lichtenstein 1971), there has been scant work done on how the informational inputs of a group of individuals are integrated into a group judgment (Davis, 1969). A Delphi forecast can be represented as a group judgment situation in which a number of individuals from various disciplinary backgrounds with input relevant to the forecasting topic judge the probable occurrence of events and provide supporting rationale for their judgments. A recent forecast about the probable time of widespread use of a computer-based medical diagnosis bank will serve as an illustration (Salancik et al., 1972). One respondent, an expert in computer services, said he thought it would not be likely to occur until after 1985, arguing that a sufficient number of data transmission facilities would not be available until at least then. Another respondent, a hospital administrator experimenting with automated systems, forecasted the event would not occur until about 1980, supporting his expectation with the opinion that it would take many years to convince and train doctors or paraprofessionals to utilize such equipment adequately. However, a medical researcher on the Delphi panel projected the event as not likely until after 1990, because it would take many years to develop the data bank to sufficient quality for use in such a sensitive area. There are three things to note about this example: J. R. SALANCIK is with the University of Illinois, Urbana-Champaign. The analysis and material for this study were obtained while the author was an Associate at the Institute for the Future, Menlo Park, California. 0 American Elsevier Publishing Company, Inc., 1973
244
JERRY R. SALANIK
(1) All three individuals agree in their skepticism that the event will occur very soon; (2) each derives his judgment from a different perspective relevant to his own sphere of knowledge or expertise; (3) there are differences in opinion as to the exact location of the event along a scale of time. How do such differences and similarities in perspective and judgment transform into a stable forecast? The typical Delphi forecast feeds the initial judgments and rationales of each participant back to the entire group of participants. The presumption is that the individuals will examine the contributions of the others in light of their own and then process the information into a subsequent forecast. The question, however, is “How well are the multiple and divergent inputs of the group integrated into a forecast?” The purpose of the present study was to examine this question. A Model of the Delphi Process In order to examine how aggregated inputs are assimilated into a forecast it is necessary to first state what is believed to be the conventional forecasting Delphi process. Three assumptions have guided this examination: (1) The main contribution of experts is in the form of impressions about the probable effect of a given future event or of knowledge about the conditions underlying the event’s occurrence (its causes). (2) The experts’ impressions represent assessments of the feasibility, the benefits, and the potential costs of the event. (3) The forecasters use a logical model in generating a forecast; that is, they combine the informational inputs of the participants in an additive fashion, such that an event is more likely to occur to the extent that it is assessed to be technically feasible, beneficial, and not costly to initiate or bring about. Test of the Proposition: Method and Results To examine whether Delphi participants assimilate inputs in the manner described above, a regression analysis was made of a sample of forecasts for future uses of computer-based systems (Salancik et al., 1972). The study was initiated in May, 1970, and began with the forecasting of forty applications of computers. The panel was drawn from diverse backgrounds, including computer manufacturing, computer services, computer technologists (developers of hardware), lawyers (specializing in computer related problems), sociologists, insurance executives, systems analysts, and computer users in specialized areas, such as data processing, process control, and medicine. Thirty individuals in all were part of the panel. A sample of fourteen applications was randomly selected to test the regression. One potential application suggested that computers be used to automatically regulate air traffic control in and around airports, setting flight patterns, entry and departure priorities and points, etc. Other applications were in the area of monitoring and diagnosis, process control, and data processing or data banks. For each application, the participants were asked (through a mail questionnaire) to estimate the date when they expected the computer application to have a 0.5 probability of occurrence as stated. Each application was described explicitly. For example, the Medical Diagnosis Bank was described as being used by at least 50% of U.S. doctors at least once a year. In addition to obtaining the
AGGREGATED
245
IMPUTS INTO DELPHI FORECASTS
projected dates of occurrence, the respondents’ reasons for their projections were obtained. Such reasons are assumed to be the respondents’ assessments of the potential future event. Accordingly, these reasons were coded into three categories: (1) Benefit statements: statements which suggested that the application, once adopted, would satisfy some current need or produce some desired value; (2) Cost statements: statements which suggested that the application would be economically unjustified or have costly effects on interested parties; and (3) FeasibiZity statements: statements which suggested that the application was technically and economically feasible or could be made so from available technology. From the fourteen computer applications used in the initial study, over 400 statements were coded into these three categories. The coding was done by two independent coders who used a sheet of formal guidelines to aid in the coding (Space does not permit presenting these guidelines; interested parties will tind them in Salancik et aZ., 1972). In a later study, all 40 computer applications were coded similarly, and it was found that 30% of the statements were classified as benefit statements, 37% as cost statements, and 33% as feasibility statements. The two coders disagreed in only about 4 percent of the 1,030 classifications; these disagreements were in all cases as to whether a particular statement represented a statement of cost or a statement of feasibility, which are very similar conceptually. Statements in each category, of course, can take on two directional values. A benefit statement may suggest that a given computer application would produce desired benefits; alternatively, a statement may suggest that a given computer application would not produce desired benefits but instead produce undesired harm. These directional implications were retained by the coders; no disagreements were found between coders as to the directional implication of a given statement. In addition to the coder reliability, an analysis was made to attest the reliability of the panel in producing the statements. Each computer application could be described by six numbers: the number of judgments indicating the application would be beneficial, the number indicating it would not be beneficial, the number indicating it would be technically feasible, the number indicating it would not, etc. One measure of the reliability of the judgments as representing real assessments is to compute the probability that out of, say, 14 judgments of benefit that 12 of them would be, by chance alone, assessments that the application was, say, beneficial, while 2 out of the 14 suggested it would not be beneficial. This probability is computable from the binomial expansion, and, in that case, the probability is less than 0.01. Similar probabilities were computed for each of the applications for each of the three dimensions of assessment (benefit, cost, and feasibility). Out of 52 such probabilities, only 7 indicated that the direction of the assessment could have obtained by chance alone, with a cut-off point of 0.05. Thus, in short, the agreement among a group of Delphi participants in their assessments about the benefit, cost, and feasibility of potential computer developments is substantial. Since both the reliability of the coders and of the Delphi participants suggest the data is reliable, it is possible to use them in a multiple regression analysis of the projected data of occurrence as a function of the judged feasibility, benefit, and cost of the potential development. A model representing this function is, Median Expected Date =A,
t b, (B) t b&T)
- b, (C).
The regression was performed by using the final second-round median date of occurrence expected for each of the 14 applications as the dependent variable, and the number
246
JERRY
R. SALANIK
of benefit statements indicating it would be beneficial minus the number indicating it would not be beneficial as one independent variable, and the number of positive and negative feasibility and cost statements as the other independent variables. The “b’s” in the model above indicate the relative weight contributed by each dimension of assessment to the group’s overall judgment of the event’s occurrence. The multiple regression was successful and suggested that the model proposed to represent the Delphi process is essentially adequate. Eighty-five percent of the variations in expected dates could be accounted for by the three assessments; the F-value of the regression mean squared over the residual was 30.81 with 3 and 10 degrees of freedom (p
Both the regression
analysis
of the aggregated inputs
of the Delphi panel and the
AGGREGATED IMPUTS INTO DELPHI FORECASTS
241
correlation of the panel’s forecasts to independent assessments suggest very strongly that these Delphi group forecasts represent a simple additive combination of information about the feasibility, cost, and benefit of potential developments. Since the Delphi panel’s members supply the verbal assessments in each of these three dimensions, and since these are feedback to all participants prior to making their final forecasts, these results confirm the assumption that the Delphi forecasts do assimilate the knowledge or impressions about the forecasted event. As Amara and Salancik (1972) have argued, one indicator of the adequacy of a forecast is the extent to which it integrates current knowledge and perceptions adequately and accurately. The current study suggests that a Delphi panel does make such integration; moreover, by the fact that the panel agrees so strongly in their assessments of benefit and feasibility, there is some indication that their knowledge of the potential events is accurate, in as much as social consensus is a basis for establishing validation in the absence of physical validation. References R. Amara and J. R. SaIancik, Forecasting:
from conjectural art toward science. Tech. Forecasting
Social Change, 3, No. 4 (1972). N. H. Anderson,
A simple model for information integration in R. P. Abelson, E. Aronson, W. J. McGuire, T. M. Newcomb, M. J. Rosenberg, and P. H. Tamrenbaum (Eds.), Theories of Cognitive Consistency: A Sourcebook. Rand McNally, Chicago (1968). J. H. Davis Group Performance. Addison-Wesley, New York (1969). P. J. Hoffman, The paramorphic representation of clinical judgment. Psych. B&L, 47, 116-131 (1960). J. R. Salancik, T. J. Gordon, and N. Adams, On the Nature of Economic Losses Arising from Computer-Based Systems in the Next Fifteen Years. Institute for the Future, Menlo Park, California R-23 (1972). Received February
15, 19 73
AND SOCIAL CHANGE 5,243-247
(1973)
243
Assimilation of Aggregated Inputs into Delphi Forecasts: A Regression Analysis J. R. SALANCIK
ABSTRACT To examine the manner in which the individual assessments of a panel of delphi experts are combined into a delphi forecast, the supporting reasons they gave for their forecasts of 40 computer applications were coded into categories of assessed “technical feasibility,” “cost of initiating,” and “benefits or needs provided.” Even though different sets of experts provided these statements in support of their individual forecasts, with some experts emphasizing one aspect and others another, it was found that the median forecast of the entire panel was significantly related to the average proportion of reasons in each category which favored bringing about the development. That is, the delphi forecasts of computer applications suggest that the computer application is forecasted to occur sooner to the extent it was judged to be technically feasible, beneficial to users or society, and not costly to develop. The results indicate further that delphi forecasting among a group of experts has logical validity, and that individual contributions are integrated into a group outcome.
Introduction Delphi forecasting is one of the most popular forms of deriving reasoned expectations about the nature and consequences of emerging developments. An unexamined but critical assumption of the Delphi method is that individual opinions of Delphi participants are assimilated into an overall judgment that integrates the current state of knowledge into a viable forecast. Although it is well established that individual judgment can be represented as a simple additive combination of informational input (Anderson 1968; Hoffman, 1960; Slavic and Lichtenstein 1971), there has been scant work done on how the informational inputs of a group of individuals are integrated into a group judgment (Davis, 1969). A Delphi forecast can be represented as a group judgment situation in which a number of individuals from various disciplinary backgrounds with input relevant to the forecasting topic judge the probable occurrence of events and provide supporting rationale for their judgments. A recent forecast about the probable time of widespread use of a computer-based medical diagnosis bank will serve as an illustration (Salancik et al., 1972). One respondent, an expert in computer services, said he thought it would not be likely to occur until after 1985, arguing that a sufficient number of data transmission facilities would not be available until at least then. Another respondent, a hospital administrator experimenting with automated systems, forecasted the event would not occur until about 1980, supporting his expectation with the opinion that it would take many years to convince and train doctors or paraprofessionals to utilize such equipment adequately. However, a medical researcher on the Delphi panel projected the event as not likely until after 1990, because it would take many years to develop the data bank to sufficient quality for use in such a sensitive area. There are three things to note about this example: J. R. SALANCIK is with the University of Illinois, Urbana-Champaign. The analysis and material for this study were obtained while the author was an Associate at the Institute for the Future, Menlo Park, California. 0 American Elsevier Publishing Company, Inc., 1973
244
JERRY R. SALANIK
(1) All three individuals agree in their skepticism that the event will occur very soon; (2) each derives his judgment from a different perspective relevant to his own sphere of knowledge or expertise; (3) there are differences in opinion as to the exact location of the event along a scale of time. How do such differences and similarities in perspective and judgment transform into a stable forecast? The typical Delphi forecast feeds the initial judgments and rationales of each participant back to the entire group of participants. The presumption is that the individuals will examine the contributions of the others in light of their own and then process the information into a subsequent forecast. The question, however, is “How well are the multiple and divergent inputs of the group integrated into a forecast?” The purpose of the present study was to examine this question. A Model of the Delphi Process In order to examine how aggregated inputs are assimilated into a forecast it is necessary to first state what is believed to be the conventional forecasting Delphi process. Three assumptions have guided this examination: (1) The main contribution of experts is in the form of impressions about the probable effect of a given future event or of knowledge about the conditions underlying the event’s occurrence (its causes). (2) The experts’ impressions represent assessments of the feasibility, the benefits, and the potential costs of the event. (3) The forecasters use a logical model in generating a forecast; that is, they combine the informational inputs of the participants in an additive fashion, such that an event is more likely to occur to the extent that it is assessed to be technically feasible, beneficial, and not costly to initiate or bring about. Test of the Proposition: Method and Results To examine whether Delphi participants assimilate inputs in the manner described above, a regression analysis was made of a sample of forecasts for future uses of computer-based systems (Salancik et al., 1972). The study was initiated in May, 1970, and began with the forecasting of forty applications of computers. The panel was drawn from diverse backgrounds, including computer manufacturing, computer services, computer technologists (developers of hardware), lawyers (specializing in computer related problems), sociologists, insurance executives, systems analysts, and computer users in specialized areas, such as data processing, process control, and medicine. Thirty individuals in all were part of the panel. A sample of fourteen applications was randomly selected to test the regression. One potential application suggested that computers be used to automatically regulate air traffic control in and around airports, setting flight patterns, entry and departure priorities and points, etc. Other applications were in the area of monitoring and diagnosis, process control, and data processing or data banks. For each application, the participants were asked (through a mail questionnaire) to estimate the date when they expected the computer application to have a 0.5 probability of occurrence as stated. Each application was described explicitly. For example, the Medical Diagnosis Bank was described as being used by at least 50% of U.S. doctors at least once a year. In addition to obtaining the
AGGREGATED
245
IMPUTS INTO DELPHI FORECASTS
projected dates of occurrence, the respondents’ reasons for their projections were obtained. Such reasons are assumed to be the respondents’ assessments of the potential future event. Accordingly, these reasons were coded into three categories: (1) Benefit statements: statements which suggested that the application, once adopted, would satisfy some current need or produce some desired value; (2) Cost statements: statements which suggested that the application would be economically unjustified or have costly effects on interested parties; and (3) FeasibiZity statements: statements which suggested that the application was technically and economically feasible or could be made so from available technology. From the fourteen computer applications used in the initial study, over 400 statements were coded into these three categories. The coding was done by two independent coders who used a sheet of formal guidelines to aid in the coding (Space does not permit presenting these guidelines; interested parties will tind them in Salancik et aZ., 1972). In a later study, all 40 computer applications were coded similarly, and it was found that 30% of the statements were classified as benefit statements, 37% as cost statements, and 33% as feasibility statements. The two coders disagreed in only about 4 percent of the 1,030 classifications; these disagreements were in all cases as to whether a particular statement represented a statement of cost or a statement of feasibility, which are very similar conceptually. Statements in each category, of course, can take on two directional values. A benefit statement may suggest that a given computer application would produce desired benefits; alternatively, a statement may suggest that a given computer application would not produce desired benefits but instead produce undesired harm. These directional implications were retained by the coders; no disagreements were found between coders as to the directional implication of a given statement. In addition to the coder reliability, an analysis was made to attest the reliability of the panel in producing the statements. Each computer application could be described by six numbers: the number of judgments indicating the application would be beneficial, the number indicating it would not be beneficial, the number indicating it would be technically feasible, the number indicating it would not, etc. One measure of the reliability of the judgments as representing real assessments is to compute the probability that out of, say, 14 judgments of benefit that 12 of them would be, by chance alone, assessments that the application was, say, beneficial, while 2 out of the 14 suggested it would not be beneficial. This probability is computable from the binomial expansion, and, in that case, the probability is less than 0.01. Similar probabilities were computed for each of the applications for each of the three dimensions of assessment (benefit, cost, and feasibility). Out of 52 such probabilities, only 7 indicated that the direction of the assessment could have obtained by chance alone, with a cut-off point of 0.05. Thus, in short, the agreement among a group of Delphi participants in their assessments about the benefit, cost, and feasibility of potential computer developments is substantial. Since both the reliability of the coders and of the Delphi participants suggest the data is reliable, it is possible to use them in a multiple regression analysis of the projected data of occurrence as a function of the judged feasibility, benefit, and cost of the potential development. A model representing this function is, Median Expected Date =A,
t b, (B) t b&T)
- b, (C).
The regression was performed by using the final second-round median date of occurrence expected for each of the 14 applications as the dependent variable, and the number
246
JERRY
R. SALANIK
of benefit statements indicating it would be beneficial minus the number indicating it would not be beneficial as one independent variable, and the number of positive and negative feasibility and cost statements as the other independent variables. The “b’s” in the model above indicate the relative weight contributed by each dimension of assessment to the group’s overall judgment of the event’s occurrence. The multiple regression was successful and suggested that the model proposed to represent the Delphi process is essentially adequate. Eighty-five percent of the variations in expected dates could be accounted for by the three assessments; the F-value of the regression mean squared over the residual was 30.81 with 3 and 10 degrees of freedom (p
Both the regression
analysis
of the aggregated inputs
of the Delphi panel and the
AGGREGATED IMPUTS INTO DELPHI FORECASTS
241
correlation of the panel’s forecasts to independent assessments suggest very strongly that these Delphi group forecasts represent a simple additive combination of information about the feasibility, cost, and benefit of potential developments. Since the Delphi panel’s members supply the verbal assessments in each of these three dimensions, and since these are feedback to all participants prior to making their final forecasts, these results confirm the assumption that the Delphi forecasts do assimilate the knowledge or impressions about the forecasted event. As Amara and Salancik (1972) have argued, one indicator of the adequacy of a forecast is the extent to which it integrates current knowledge and perceptions adequately and accurately. The current study suggests that a Delphi panel does make such integration; moreover, by the fact that the panel agrees so strongly in their assessments of benefit and feasibility, there is some indication that their knowledge of the potential events is accurate, in as much as social consensus is a basis for establishing validation in the absence of physical validation. References R. Amara and J. R. SaIancik, Forecasting:
from conjectural art toward science. Tech. Forecasting
Social Change, 3, No. 4 (1972). N. H. Anderson,
A simple model for information integration in R. P. Abelson, E. Aronson, W. J. McGuire, T. M. Newcomb, M. J. Rosenberg, and P. H. Tamrenbaum (Eds.), Theories of Cognitive Consistency: A Sourcebook. Rand McNally, Chicago (1968). J. H. Davis Group Performance. Addison-Wesley, New York (1969). P. J. Hoffman, The paramorphic representation of clinical judgment. Psych. B&L, 47, 116-131 (1960). J. R. Salancik, T. J. Gordon, and N. Adams, On the Nature of Economic Losses Arising from Computer-Based Systems in the Next Fifteen Years. Institute for the Future, Menlo Park, California R-23 (1972). Received February
15, 19 73