- Email: [email protected]
Consumerism comes to Delphi: Comments on Delphi assessment, expert opinion, forecasting, and group process by H. Sackman
TECHNOLOGICAL
FORECASTING
Consumerism
AND SOCIAL CHANGE I, 215-219 (1975)
215
Comes to Delphi:
Comments on Delphi Assessment,
Expert Opinion,
Forecasting, and Group Process by H. Sackman*
D. SAM SCHEELE
All of you readers who are hoping to do a simple, cheap, highly insightful, bombshell of a future forecast on a topic of great currency, using a do-it-yourself Delphi methodology, and become famous, may be about to be found out. The bureaucracy of science is now asserting its jurisdiction over fame-and-fortune seekers plying the futurism trade. Special investigator Sackman has surveyed Delphi sales, and he’s found the product mislabeled, poorly packaged, varying in quality, of uncertain benefit, and potentially addictive to the unsuspected procurer of research and management information. On first reading Delphi Assessment, you are stunned by the revelations of what a Delphi study is not. He points out many important caveats, but, in the end, you feel that you just finished reading an assessment of the painting art of Franz Kline and Robert Motherwell that revealed how many times they got their colors “outside the lines.” If you’d like to read how Alexander Graham Bell’s laboratory would have fared in an OHSA (Occupational Health and Safety Act) inspection, you’ll love this Delphi assessment. This is not to say that the Delphi technique and other forecasting methodologies are not in need of improvement, even reformulation. The philosophical and epistemological underpinnings of forecasting methodologies are often shallow. Finding ways to shape policies and actions by stimulating conceptualization and collecting information about the future are also needed. Some efforts along these lines may be found in Linstone and Turoffs book The Delphi Method: Techniques and Applications (Addison-Wesley Publ. Co., 1975). However, I don’t expect any theoretical or methodological breakthroughs will come from enforcing procedural standards and specifying guidelines for interaction that are drawn from inquiry techniques of the past. Further, I find that the author’s exhortations in the name of scientific humanism, in the face of a need to work through real intellectual difficulties, are as hollow as the political pronouncements of wellmeaning liberals were in dealing with the societal problems. Reforming science so that it doesn’t hurt anybody is not enough. The more basic challenge is to create the underpinnings for a nondeterministic, developmentally oriented, accessible set of intellectual activities that will produce information out of which more desirable futures can be invented and negotiated. Much attention, both very serious and popularistically trivial, is being directed to pre-, D. SAM SCHEELE is with Social Engineering Technology, Inc., Angeles, Calif. 90024. *Rand Report R-1283-PR, Santa Monica, California, April, 1974. 0 American
1722
Westwood
Elsevier Publishing
Blvd.,
Company,
Los
1975
216
D. SAM SCHEELE
para-, and tram-scientific ways of knowing, and sources of knowledge. Investigators in the sociology of knowledge are looking into exactly what it is we do when we say we are doing “science’‘-or planning, or policy making, or whatever. Where do insights come from? How are conclusions drawn? From what sources do results obtain their force and believability? Investigations of the phenomenon of parapsychology and the interest of increasing numbers of people in a wide variety of belief systems are challenging not only our ability to explain, but the relevance of explanations to action and predictions to power. Ethnomethodologists are reexamining the practices of prescience in an attempt to uncover other paradigms of knowledge generation. Criticism based on an “aesthetic” view of the results of competing epistemological frameworks and schemes of inquiry is more in evidence. The criteria for structuring investigations are less frequently involved with the accuracy of the information to be produced and more frequently concerned with the investigations’ appropriateness. Appropriateness is judged by the intended and potential uses of the investigation and its results along with their sociocultural believability and acceptance. This brings us whole circle to the metaphor of art criticism injected earlier. In this assessment Sackman wants to tell all consumers of futures research and other buyers of inquiries into the nature of the future that the Delphi technique is to be suspected, its practitioners have questionable motives, and that the whole enterprise is likely not even fixable. This all may be true, but does this danger require external warnings? From a developmental viewpoint, consumerism tends to infantilize the buyers of products and transfer the responsibility for judgments about quality and worth to external authorities and regulatory bodies. While this is often necessary, and expedient in dealing with some of the trivial things of life, it smacks of antideviationalism when applied to the marketplace of ideas. The other issue raised by Sackman that was significant for me was the variability of what the term Delphi is used to describe. He points out that it’s a catchy name, bears connotation of oracular insight, and serves to put a patina of regularity and uniformity on what is in fact a highly diverse collection of sometimes-used procedures and general practices. For me, the Delphi technique is a search. It’s a strategy for methodological development where one labels the process first in order to provide a cover for inventing it as one goes. Such a strategy may not seem very noble, but it is often useful. It is based on a corollary to Donald Schoen’s law that you can never be funded for what needs to be done so you get funded for what’s being funded and then move into what you believe needs to be done. This might be termed a conspiratorial approach to the production of knowledge; I realize it doesn’t fit well with the prevailing righteousness of relevancyseeking scientist-scholars, but I believe a conspiratorial approach may be productive of healthy pluralism in modes of thought and action. This we badly need. As the largest contributor to the closed literature on the subject of Delphi, I did appreciate the rather extensive bibliography that Sackman has prepared. Particular note should be taken of his “semi-annotation.” I think it’s unfair for people to prepare bibliographies without indicating something about the materials they’ve included. Bibliographies are not tests of one’s library stamina or ability to divine content from the limited set of words that seem to be combined in titles on any given subject. Sackman’s writing style is somewhat laborious, very hortatory, and has more than a faint taint of selfrighteousness. However, his assessment should point out to Delphi practitioners that they have not been clear enough in their writings about what they’re about. Sackman, in making a comprehensive assessment of what they’ve been doing, missed the point of their efforts. This indicates to me that much more attention needs to be put into delineating
CONSUMERISMCOMESTODELPHI
217
and explicating the philosophic and methodological premises of each Delphi inquiry. Such statements would act as caveats to protect the unsophisticated buyer of Delphi results. Heretofore, most buyers have probably had a fairly clear idea of what they were getting. But, in the years ahead there are likely to be many less astute and more easily taken-in buyers. And, as Sackman points out, the Delphi technique has many compelling qualities for lazy, sales-oriented, popularity-seeking investigators, so that there is a danger of a stampede of Delphi hustlers forming. Now for a quick tour of the report. It begins with a preface, which tells you that a critical evaluation of Delphi is long overdue. Further, it’s nice that this can be conducted at RAND where the techniques emerged-at least a half-non-sequitur. The criteria for assessing conventional Delphi are stated. These are (1) the established professional standards for opinion questionnaires, plus associated scientific standards for experimentation with human subjects, and (2) the assumptions, principles, and methodologies inherent in the technique. Next comes the summary. This says much the same thing as the preface except it raises further weighty issues-whether or not Delphi is accurate, does it produce real or artificial consensus, and are snap judgments of any value? It’s a relief to find out here that Delphis may be of possible value as an informal exercise for heuristic purposes and that they are not harmful to your health if swallowed in this manner. The final recommendation is that the conventional Delphis be dropped from institutional, corporate, and governmental use until these practitioners clean up their act and become scientifically kosher, which is likely to be impossible. Next come the acknowledgements. Here I note a curious lack of familiar Delphirelated names, though many are in the Los Angeles area. Then you come to the prologue. Here everything that was said before is said again, plus another complaint-there isn’t a fixed, universally agreed-upon working definition of Delphi and therefore it’s tough to lay on standards. To get over this, the author defines a “conventional Delphi.” I guess this definition is something between a “full-time equivalent student” and the standard Lhasa Apso, but such are the needs of assessment. The following section describes the Delphi method and the procedural characteristics, which the author finds objectionable. The writing proceeds from rhetorical question to rhetorical question, building up such a convincing case that you would have to think that those who have undertaken Delphi studies are either ignorant or crazy. Since neither is obviously the case, Sackman might have wondered why it is that people undertook these Delphi studies. This would have been a better investigation. There must be something more important than the rules of questionnaire design. As for the issue of experimenting on human subjects, it strikes me as slightly bizarre that it’s even raised. I’m sure there are more “moth wings” in the tuna fish cans tested by Consumer Reports than there are individuals who have been trifled with in Delphi studies. Further, I think there is a useful distinction between subjects and participants. Recent articles in the psychological literature indicate a growing interest in returning to study behavior in so-called natural settings rather than continue setting up experimental situations. Next we come to the kernel of the assessment: “Delphi Versus Social Science Standards.” Here the responsibility and jurisdiction of the scientific bureaucracy is asserted. Sackman even gives a few accolades for dedication. But, alas, the critique is mundane, boring, and prosaic. This section sequentially reveals the eternal truths contained in interpretation standards, empirical validity, standards for the use of experts,
218
D. SAM SCHEELE
theoretical standards, questionnaire replicability, and experimental sampling. He concludes that the conventional Delphi violates virtually every major tenet of established professional standards for questionnaire design, administration, application, and validation. Delphi purveyors and users are in an obvious conspiracy against professional standards. In the next chapter on Delphi evaluation, we find that the Delphi technique is misused. We learn it has more justification as a management and marketing tool than it has for producing scientific knowledge. At this point Sackman should have wondered why the terrible “bucket” that he had been evaluating as a container served so well as a “sieve” and was being used that way. From this he might have discovered some insights about the direction of Delphi development and its relevance to extending our knowledge about how the future is negotiated and invented. Here, too, we find an interesting critique of what constitutes an expert in a field and how those experts are employed in the Delphi process. The Delphi applications he cites show that in some types of successive polling, experts produce results that are no different from “housewives” or graduate students. The reason, I think, is that those particular applications misused the experts. Because the polling techniques are rigorous and narrowly focused, the inquiry instrument misses the experts’ most important contributions-not “what? but the “whys” and “because ofs.” Sackman fails to point out the importance of the practical and procedural advantage of Delphi studies, that is, being able to get opinions from a broader group of people than could be assembled in a single place without great difficulty. More difficult would be to hold a series of discussions on related topics. Even if it were possible to get everyone there at the same time for these discussions, you can be reasonably sure that not everyone will be focused on the topic or have free attention for the topic at that time. Then introduce the dynamic of personal styles of interaction and you lower the collective productivity significantly. One way to view Delphi exercises is to see them as managed interactions attempting to be productive of particular kinds of results. Sackman’s critique of the length and ambiguous exhaustiveness of Delphi questionnaires is well taken. I have no explanation for how so many investigators have gotten such large numbers of respondents to go through them. Having people spend a minute per item on such Delphi questionnaires is asking a lot and getting very little. I have found that by employing unfinished scenarios for completion by the respondent I get interesting results on important dimensions without a lot of superfluous items. Using mini-scenarios anchors the responses in concrete situations which produces a richer set of data and stimulates more insightful interpretations. The subsection on the epistemology of Delphi is confused about the kinds of confusion Delphi creates. There is no issue of accuracy, validity, or correctness, but there is much ambiguity in the statement of Delphi items in many inquiries I’ve seen. Practitioners in the future should be careful to see that this is minimized. In some cases, however, ambiguous items can be used knowingly to identify the different “habits of mind” or world views employed by the respondents. These underlying psychocultural patterns are often important in depicting how the future might be negotiated and what will be seen as issues by various constituencies should a new technology or policy be introduced. In conclusion, Sackman finds that Delphis are often poorly done. So do I, but for different reasons. Maybe there should be a different label for this class of inquiry. I think it is important to look not at what Delphis mean, but at how meaning is constructed in
CONSUMERISM
COMES TO DELPHI
219
the Delphi process. The Delphi technique can make the process of formulating policy and making of assumptions accessible. Rigor might better be applied not to improving the Delphi procedures for their own sake but to the study of how results are produced by the technique. That is, the Delphi technique can serve as a seminatural behavioral setting or interactional situation. Given a role in the policy making, planning, management, budgeting, and other ongoing processes of the society and its organizations will provide for greater and wider participation by those affected. The study of Delphi could produce additional insights about how meaning is constructed, how coalitions are formed, how possibilities are generated . . . In turn, this information could be used then to structure Delphis as nontemporal, spatial settings for’interaction about the future. From a management perspective, it has to be better than more meetings. Received
12 November
1974
FORECASTING
Consumerism
AND SOCIAL CHANGE I, 215-219 (1975)
215
Comes to Delphi:
Comments on Delphi Assessment,
Expert Opinion,
Forecasting, and Group Process by H. Sackman*
D. SAM SCHEELE
All of you readers who are hoping to do a simple, cheap, highly insightful, bombshell of a future forecast on a topic of great currency, using a do-it-yourself Delphi methodology, and become famous, may be about to be found out. The bureaucracy of science is now asserting its jurisdiction over fame-and-fortune seekers plying the futurism trade. Special investigator Sackman has surveyed Delphi sales, and he’s found the product mislabeled, poorly packaged, varying in quality, of uncertain benefit, and potentially addictive to the unsuspected procurer of research and management information. On first reading Delphi Assessment, you are stunned by the revelations of what a Delphi study is not. He points out many important caveats, but, in the end, you feel that you just finished reading an assessment of the painting art of Franz Kline and Robert Motherwell that revealed how many times they got their colors “outside the lines.” If you’d like to read how Alexander Graham Bell’s laboratory would have fared in an OHSA (Occupational Health and Safety Act) inspection, you’ll love this Delphi assessment. This is not to say that the Delphi technique and other forecasting methodologies are not in need of improvement, even reformulation. The philosophical and epistemological underpinnings of forecasting methodologies are often shallow. Finding ways to shape policies and actions by stimulating conceptualization and collecting information about the future are also needed. Some efforts along these lines may be found in Linstone and Turoffs book The Delphi Method: Techniques and Applications (Addison-Wesley Publ. Co., 1975). However, I don’t expect any theoretical or methodological breakthroughs will come from enforcing procedural standards and specifying guidelines for interaction that are drawn from inquiry techniques of the past. Further, I find that the author’s exhortations in the name of scientific humanism, in the face of a need to work through real intellectual difficulties, are as hollow as the political pronouncements of wellmeaning liberals were in dealing with the societal problems. Reforming science so that it doesn’t hurt anybody is not enough. The more basic challenge is to create the underpinnings for a nondeterministic, developmentally oriented, accessible set of intellectual activities that will produce information out of which more desirable futures can be invented and negotiated. Much attention, both very serious and popularistically trivial, is being directed to pre-, D. SAM SCHEELE is with Social Engineering Technology, Inc., Angeles, Calif. 90024. *Rand Report R-1283-PR, Santa Monica, California, April, 1974. 0 American
1722
Westwood
Elsevier Publishing
Blvd.,
Company,
Los
1975
216
D. SAM SCHEELE
para-, and tram-scientific ways of knowing, and sources of knowledge. Investigators in the sociology of knowledge are looking into exactly what it is we do when we say we are doing “science’‘-or planning, or policy making, or whatever. Where do insights come from? How are conclusions drawn? From what sources do results obtain their force and believability? Investigations of the phenomenon of parapsychology and the interest of increasing numbers of people in a wide variety of belief systems are challenging not only our ability to explain, but the relevance of explanations to action and predictions to power. Ethnomethodologists are reexamining the practices of prescience in an attempt to uncover other paradigms of knowledge generation. Criticism based on an “aesthetic” view of the results of competing epistemological frameworks and schemes of inquiry is more in evidence. The criteria for structuring investigations are less frequently involved with the accuracy of the information to be produced and more frequently concerned with the investigations’ appropriateness. Appropriateness is judged by the intended and potential uses of the investigation and its results along with their sociocultural believability and acceptance. This brings us whole circle to the metaphor of art criticism injected earlier. In this assessment Sackman wants to tell all consumers of futures research and other buyers of inquiries into the nature of the future that the Delphi technique is to be suspected, its practitioners have questionable motives, and that the whole enterprise is likely not even fixable. This all may be true, but does this danger require external warnings? From a developmental viewpoint, consumerism tends to infantilize the buyers of products and transfer the responsibility for judgments about quality and worth to external authorities and regulatory bodies. While this is often necessary, and expedient in dealing with some of the trivial things of life, it smacks of antideviationalism when applied to the marketplace of ideas. The other issue raised by Sackman that was significant for me was the variability of what the term Delphi is used to describe. He points out that it’s a catchy name, bears connotation of oracular insight, and serves to put a patina of regularity and uniformity on what is in fact a highly diverse collection of sometimes-used procedures and general practices. For me, the Delphi technique is a search. It’s a strategy for methodological development where one labels the process first in order to provide a cover for inventing it as one goes. Such a strategy may not seem very noble, but it is often useful. It is based on a corollary to Donald Schoen’s law that you can never be funded for what needs to be done so you get funded for what’s being funded and then move into what you believe needs to be done. This might be termed a conspiratorial approach to the production of knowledge; I realize it doesn’t fit well with the prevailing righteousness of relevancyseeking scientist-scholars, but I believe a conspiratorial approach may be productive of healthy pluralism in modes of thought and action. This we badly need. As the largest contributor to the closed literature on the subject of Delphi, I did appreciate the rather extensive bibliography that Sackman has prepared. Particular note should be taken of his “semi-annotation.” I think it’s unfair for people to prepare bibliographies without indicating something about the materials they’ve included. Bibliographies are not tests of one’s library stamina or ability to divine content from the limited set of words that seem to be combined in titles on any given subject. Sackman’s writing style is somewhat laborious, very hortatory, and has more than a faint taint of selfrighteousness. However, his assessment should point out to Delphi practitioners that they have not been clear enough in their writings about what they’re about. Sackman, in making a comprehensive assessment of what they’ve been doing, missed the point of their efforts. This indicates to me that much more attention needs to be put into delineating
CONSUMERISMCOMESTODELPHI
217
and explicating the philosophic and methodological premises of each Delphi inquiry. Such statements would act as caveats to protect the unsophisticated buyer of Delphi results. Heretofore, most buyers have probably had a fairly clear idea of what they were getting. But, in the years ahead there are likely to be many less astute and more easily taken-in buyers. And, as Sackman points out, the Delphi technique has many compelling qualities for lazy, sales-oriented, popularity-seeking investigators, so that there is a danger of a stampede of Delphi hustlers forming. Now for a quick tour of the report. It begins with a preface, which tells you that a critical evaluation of Delphi is long overdue. Further, it’s nice that this can be conducted at RAND where the techniques emerged-at least a half-non-sequitur. The criteria for assessing conventional Delphi are stated. These are (1) the established professional standards for opinion questionnaires, plus associated scientific standards for experimentation with human subjects, and (2) the assumptions, principles, and methodologies inherent in the technique. Next comes the summary. This says much the same thing as the preface except it raises further weighty issues-whether or not Delphi is accurate, does it produce real or artificial consensus, and are snap judgments of any value? It’s a relief to find out here that Delphis may be of possible value as an informal exercise for heuristic purposes and that they are not harmful to your health if swallowed in this manner. The final recommendation is that the conventional Delphis be dropped from institutional, corporate, and governmental use until these practitioners clean up their act and become scientifically kosher, which is likely to be impossible. Next come the acknowledgements. Here I note a curious lack of familiar Delphirelated names, though many are in the Los Angeles area. Then you come to the prologue. Here everything that was said before is said again, plus another complaint-there isn’t a fixed, universally agreed-upon working definition of Delphi and therefore it’s tough to lay on standards. To get over this, the author defines a “conventional Delphi.” I guess this definition is something between a “full-time equivalent student” and the standard Lhasa Apso, but such are the needs of assessment. The following section describes the Delphi method and the procedural characteristics, which the author finds objectionable. The writing proceeds from rhetorical question to rhetorical question, building up such a convincing case that you would have to think that those who have undertaken Delphi studies are either ignorant or crazy. Since neither is obviously the case, Sackman might have wondered why it is that people undertook these Delphi studies. This would have been a better investigation. There must be something more important than the rules of questionnaire design. As for the issue of experimenting on human subjects, it strikes me as slightly bizarre that it’s even raised. I’m sure there are more “moth wings” in the tuna fish cans tested by Consumer Reports than there are individuals who have been trifled with in Delphi studies. Further, I think there is a useful distinction between subjects and participants. Recent articles in the psychological literature indicate a growing interest in returning to study behavior in so-called natural settings rather than continue setting up experimental situations. Next we come to the kernel of the assessment: “Delphi Versus Social Science Standards.” Here the responsibility and jurisdiction of the scientific bureaucracy is asserted. Sackman even gives a few accolades for dedication. But, alas, the critique is mundane, boring, and prosaic. This section sequentially reveals the eternal truths contained in interpretation standards, empirical validity, standards for the use of experts,
218
D. SAM SCHEELE
theoretical standards, questionnaire replicability, and experimental sampling. He concludes that the conventional Delphi violates virtually every major tenet of established professional standards for questionnaire design, administration, application, and validation. Delphi purveyors and users are in an obvious conspiracy against professional standards. In the next chapter on Delphi evaluation, we find that the Delphi technique is misused. We learn it has more justification as a management and marketing tool than it has for producing scientific knowledge. At this point Sackman should have wondered why the terrible “bucket” that he had been evaluating as a container served so well as a “sieve” and was being used that way. From this he might have discovered some insights about the direction of Delphi development and its relevance to extending our knowledge about how the future is negotiated and invented. Here, too, we find an interesting critique of what constitutes an expert in a field and how those experts are employed in the Delphi process. The Delphi applications he cites show that in some types of successive polling, experts produce results that are no different from “housewives” or graduate students. The reason, I think, is that those particular applications misused the experts. Because the polling techniques are rigorous and narrowly focused, the inquiry instrument misses the experts’ most important contributions-not “what? but the “whys” and “because ofs.” Sackman fails to point out the importance of the practical and procedural advantage of Delphi studies, that is, being able to get opinions from a broader group of people than could be assembled in a single place without great difficulty. More difficult would be to hold a series of discussions on related topics. Even if it were possible to get everyone there at the same time for these discussions, you can be reasonably sure that not everyone will be focused on the topic or have free attention for the topic at that time. Then introduce the dynamic of personal styles of interaction and you lower the collective productivity significantly. One way to view Delphi exercises is to see them as managed interactions attempting to be productive of particular kinds of results. Sackman’s critique of the length and ambiguous exhaustiveness of Delphi questionnaires is well taken. I have no explanation for how so many investigators have gotten such large numbers of respondents to go through them. Having people spend a minute per item on such Delphi questionnaires is asking a lot and getting very little. I have found that by employing unfinished scenarios for completion by the respondent I get interesting results on important dimensions without a lot of superfluous items. Using mini-scenarios anchors the responses in concrete situations which produces a richer set of data and stimulates more insightful interpretations. The subsection on the epistemology of Delphi is confused about the kinds of confusion Delphi creates. There is no issue of accuracy, validity, or correctness, but there is much ambiguity in the statement of Delphi items in many inquiries I’ve seen. Practitioners in the future should be careful to see that this is minimized. In some cases, however, ambiguous items can be used knowingly to identify the different “habits of mind” or world views employed by the respondents. These underlying psychocultural patterns are often important in depicting how the future might be negotiated and what will be seen as issues by various constituencies should a new technology or policy be introduced. In conclusion, Sackman finds that Delphis are often poorly done. So do I, but for different reasons. Maybe there should be a different label for this class of inquiry. I think it is important to look not at what Delphis mean, but at how meaning is constructed in
CONSUMERISM
COMES TO DELPHI
219
the Delphi process. The Delphi technique can make the process of formulating policy and making of assumptions accessible. Rigor might better be applied not to improving the Delphi procedures for their own sake but to the study of how results are produced by the technique. That is, the Delphi technique can serve as a seminatural behavioral setting or interactional situation. Given a role in the policy making, planning, management, budgeting, and other ongoing processes of the society and its organizations will provide for greater and wider participation by those affected. The study of Delphi could produce additional insights about how meaning is constructed, how coalitions are formed, how possibilities are generated . . . In turn, this information could be used then to structure Delphis as nontemporal, spatial settings for’interaction about the future. From a management perspective, it has to be better than more meetings. Received
12 November
1974