- Email: [email protected]
Policy studies: Integration and evaluation
This is an important book for program planners and evaluators. It includes a presentation of basic concepts and criteria for conducting policy analyses, and it contains descriptions and comparisons of methods employed for evaluating policy choices. Broadly defining policy studies as “. . . describing policies, expfaining their existence, and evaluating them.” (p. 5), Nagel’s major purpose is to present and compare optimization procedures for ev~uating policies in reference to cconomic (effectiveness, efficiency, equity) and politicaf scieace (public participation, pred~~a~~lit~, proccduraf due process} criteria. The book is subdivided into seven sections that progressively incorporate concepts and methods from chapter to chapter: Basic Concepts and Criteria, Policy Goals, Goal-Achieving Means, Methods of 0ptimit;ing and Sensitivity Analysis, Evaluating Alternative Methods of Evaluation, Evaluating the Field of Policy Studies, and Other Policy-Reiated Fields. Basically, Nag& advocates for a multiple criteria decision-making approach (MCDM) to policy analysis, using p~/whole ~rcentaging with respect to goals and policies. The method is described clearly and is compared with mathematical optirn~~at~o~ procedures and other methods such as quasi-ex~~me~tatio~. With the assistance of Benjamin Radcliff, a chapter is provided that compares software packages for policy/goal percentaging analysis and other MCDM procedures. Nagel is a clear thinker. He provides a number of insightful discussions. Employing a calculus for combining validity, importance, adoption, and utilization as criteria for judging policy studies, he shows how relative weights can be combined in pay-off matrices. He uses equations to describe and deduce relations among equity, effectiveness, and efficiency, and he illustrates the thought processes involved in developing an index of unemployment that includes categories beyond the usual indices. Moreover, he presents an excellent anaIysis of the administration of public functions. Although the calculus for combining weights is clear, the assignment of specific weights often appears to be arbitrary rather than being based on empirical data or consensually validating observations. For example, in constructing an index of wrong doing, Nagel says ‘(. . . Therefore each crime committed should receive
about 2-3 times as much weight as each civil wrong, other things held constant.” (p. 67). In his discussion of what is better than the optimum, it seems to be a matter of judgment as to what values are held by society; for example, he says, “‘In effect, high goals are desirable because they stimulate greater productivity and creativity and thereby benefit society, even if individuals with high goals are not happy-” (p. 77). Optimization methods for choosing among aiternatives can be employed for planning and evaluation on the supposition that the relevant decision makers and choices are clearly delineated. ~b~ously~ when there are multiple decision makers who are at cross-purposes and/or have different sets of multiple criteria, such methods are less useful. This may be due to the notion that the procedures are less responsive and flexible to criteria1 shifts as a result of negotiation and conflict resolution strategies. In spite of his assertion that policy evahration is becoming increasingly i~ltcrdisciplin~ (p_ 9), Nagel narrowfy divides public policies by social science and science disciplines, for example, political science, suciolo~-psych~lQ~y, economics, etc., rather than by social problems and special populat~ons~ which are more likely to emphasize interdisciplinary foci, Policies on child abuse and neglect, for example, might involve lawyers, pediatricians, nurses, teachers, social workers, psychologists, management information specialists, etc. For the most part, Nagel is internally consistent in providing his ideas; however, the concept of utilization appears to be ambiguously applied. At the beginning of the book, “utilization” is defined as reference to the policy study by relevant policy makers as dis~i~~ished from “adoption,” which refers to an adoptian of the policy fp. 15); in the latter part of the book, “utilization” is less precisely conceived as being either useful or influential in decision-making. In summation, this book is useful for program planners in that it provides concepts and criteria for estimating the consequences of policies. Moreover, it contains important guidelines for combining various criteria related to policy decisions. Nagel is right in advocating that evaluators should exchange more ideas with policy analysts. Evaiuative re-
102
Book Reviews
searchers will find this book especially intriguing, It is an exemplar in the use of ciearly designated and differentiated criteria for analyzing given sets of policy alternatives&Nagel’s chapter on effectiveness, efficiency, and equity is extremely illuminating; it is must-reading for
program plarmers and evaluators. Furthermore, this book should stimulate evaluative researchers to conduct methodological studies comparing and contrasting the relative merits and limitations of optimizing techniques with various evaluative research strategies.
Debates m I3wtluation by Marvin C. Alkin, with contributions by Michael Quinn Patton and Carol H, Weiss. Newbury Park, CA: Sage Publications, 1990, 302 pp. RevieMrers: Thomas A. Schwandt and Peter Magolda Sometime in the mid I98oS in Mahbu, ~~ifor~ia, Marvin Alkin convened a group of distinguished evaluation professionals (Ross Conner, Ernest House, Michael Kean, Jean King, Susan Klein, Alex Law, Milbrey Mctaughlin, Michael Patton, and Carol Weiss) for 3 days of informal talks. The meeting’s purpose was to ‘“synthesize positions on the current status of evaluation utilization: research, implications for practice, and potential new directions” [p+ 91. De&z&s on Evaluation is primarily an edited transcript of these unstructured discussions. Professor Alkin edited, rearranged, and sorted the ~ransc~b~d dialogue into four major parts. W’nhin each part, portions of the dialogue are identified further by I3 “‘chapters” (topics) and 50 subtopics. An Appendix lists all the topics and subtopics indexed to the fine numbers in the original transcript. Part I, ‘LEvaluation Utilization,” begins with five pages of introductory comments from each participant on the notion of “utilization” followed by comments on utilization factors and issues, and research, dissemination, and training. Part II, “Evaluation Theory,‘” introduces the reader to the participants’ views on the purpose and function of evafuation, and on the notions of evaluation audiences, the F& of the evafuator, and assorted methodologic~ issues. Part III, *~~val~ation Theory ~~stin~i~ns~ Further Debate,” departs from a presentation of the discussion that took place at the Malibu conference, It consists of four papers previously published in Evaluation Practice that comprise an exchange of views between Michael Patton and Carol Weiss on the purpose and use of evaluation Part IV, “Politics and Ethics,” returns to the transcript presentation format. Highlighted are political, ethica1, and misuse issues that evaluation professionals encounter. Each part concludes with a “Review and Commentary” by Professor Alkin. The comments of two ~~discussan~s” who read the book manuscript- Fred Ellett, Jr. and Michael Hendricks-also appear in the introductions to each part and in footnotes. In a few instances, footnotes also are used to present participants” clarifications and/or elaborations on their original comments. On July 9, 1991, we met in a Bloomington, Indiana diner for 2 hours of informal talks over breakfast about Debates on ~v~~~at~~n. The purpose of the meeting was to synthesize our reactions and to draft a review of the book. Herewith is the edited and rearranged transcript of that meeting:
T. Schwa~dt~ Well, Peter, what are your general reactions to the book? F. Magold~~ Hummh. Overall, my reaction is mixed. The book lived up to the author’s assertion of a “most unusual book”’ [p* 91. Assembling a group of evaluation theoreticians and practitioners to debate issues on evaluation and then transcribing and publishing the conversations is a good idea, but it didn’t work well in this instance. There are several reasons for my tepid assessment _ _ _ T. ~chw~dt: “Tepid,” I like that8 it sums up my reactions as well. I liked the idea of exe perimenting with text formats, and the dialogue is at times thought-provoking, Yet this experiment isn’t particularly satisfying, in part, because of the disjointed and wide-ranging nature of the dialogue. I don’t think that the term “debate’” accurately characterizes most of the participant exchanges. It’s more like a discussion. And while seemingly cordial and frank, it’s not really a ~‘pe~etrati~~‘~ discussion as Alkin suggests. The evaluation waterfront is covered - 13 different issues are mentioned . _ . few are thoroughly discussed and less than that actually debated. There are exceptions, such as the exchange over social justice and serving client’s needs. [Flips through book to find pages.] Remember this exchange? 2%. J--Won:
E. House:
A@*Fat&m:
* . . .$rom m_v ~e~~~~~~~~~, I #ace a higkr ~~s~~~s~b~~~tyfm serving ~~~e~ts than on the broader aims. And that frame of reference does have the potential far getting into some differeerztarientations and doing things d~~fereentiy~and that would be interesting to put ta the test. We& how far would you pursue this orie~t~t~~~? Surely, you can ‘t consider yotcr opt& purpose to be meeting your &nt “s ~~t~~est~ Tell me why I can%
about 2-3 times as much weight as each civil wrong, other things held constant.” (p. 67). In his discussion of what is better than the optimum, it seems to be a matter of judgment as to what values are held by society; for example, he says, “‘In effect, high goals are desirable because they stimulate greater productivity and creativity and thereby benefit society, even if individuals with high goals are not happy-” (p. 77). Optimization methods for choosing among aiternatives can be employed for planning and evaluation on the supposition that the relevant decision makers and choices are clearly delineated. ~b~ously~ when there are multiple decision makers who are at cross-purposes and/or have different sets of multiple criteria, such methods are less useful. This may be due to the notion that the procedures are less responsive and flexible to criteria1 shifts as a result of negotiation and conflict resolution strategies. In spite of his assertion that policy evahration is becoming increasingly i~ltcrdisciplin~ (p_ 9), Nagel narrowfy divides public policies by social science and science disciplines, for example, political science, suciolo~-psych~lQ~y, economics, etc., rather than by social problems and special populat~ons~ which are more likely to emphasize interdisciplinary foci, Policies on child abuse and neglect, for example, might involve lawyers, pediatricians, nurses, teachers, social workers, psychologists, management information specialists, etc. For the most part, Nagel is internally consistent in providing his ideas; however, the concept of utilization appears to be ambiguously applied. At the beginning of the book, “utilization” is defined as reference to the policy study by relevant policy makers as dis~i~~ished from “adoption,” which refers to an adoptian of the policy fp. 15); in the latter part of the book, “utilization” is less precisely conceived as being either useful or influential in decision-making. In summation, this book is useful for program planners in that it provides concepts and criteria for estimating the consequences of policies. Moreover, it contains important guidelines for combining various criteria related to policy decisions. Nagel is right in advocating that evaluators should exchange more ideas with policy analysts. Evaiuative re-
102
Book Reviews
searchers will find this book especially intriguing, It is an exemplar in the use of ciearly designated and differentiated criteria for analyzing given sets of policy alternatives&Nagel’s chapter on effectiveness, efficiency, and equity is extremely illuminating; it is must-reading for
program plarmers and evaluators. Furthermore, this book should stimulate evaluative researchers to conduct methodological studies comparing and contrasting the relative merits and limitations of optimizing techniques with various evaluative research strategies.
Debates m I3wtluation by Marvin C. Alkin, with contributions by Michael Quinn Patton and Carol H, Weiss. Newbury Park, CA: Sage Publications, 1990, 302 pp. RevieMrers: Thomas A. Schwandt and Peter Magolda Sometime in the mid I98oS in Mahbu, ~~ifor~ia, Marvin Alkin convened a group of distinguished evaluation professionals (Ross Conner, Ernest House, Michael Kean, Jean King, Susan Klein, Alex Law, Milbrey Mctaughlin, Michael Patton, and Carol Weiss) for 3 days of informal talks. The meeting’s purpose was to ‘“synthesize positions on the current status of evaluation utilization: research, implications for practice, and potential new directions” [p+ 91. De&z&s on Evaluation is primarily an edited transcript of these unstructured discussions. Professor Alkin edited, rearranged, and sorted the ~ransc~b~d dialogue into four major parts. W’nhin each part, portions of the dialogue are identified further by I3 “‘chapters” (topics) and 50 subtopics. An Appendix lists all the topics and subtopics indexed to the fine numbers in the original transcript. Part I, ‘LEvaluation Utilization,” begins with five pages of introductory comments from each participant on the notion of “utilization” followed by comments on utilization factors and issues, and research, dissemination, and training. Part II, “Evaluation Theory,‘” introduces the reader to the participants’ views on the purpose and function of evafuation, and on the notions of evaluation audiences, the F& of the evafuator, and assorted methodologic~ issues. Part III, *~~val~ation Theory ~~stin~i~ns~ Further Debate,” departs from a presentation of the discussion that took place at the Malibu conference, It consists of four papers previously published in Evaluation Practice that comprise an exchange of views between Michael Patton and Carol Weiss on the purpose and use of evaluation Part IV, “Politics and Ethics,” returns to the transcript presentation format. Highlighted are political, ethica1, and misuse issues that evaluation professionals encounter. Each part concludes with a “Review and Commentary” by Professor Alkin. The comments of two ~~discussan~s” who read the book manuscript- Fred Ellett, Jr. and Michael Hendricks-also appear in the introductions to each part and in footnotes. In a few instances, footnotes also are used to present participants” clarifications and/or elaborations on their original comments. On July 9, 1991, we met in a Bloomington, Indiana diner for 2 hours of informal talks over breakfast about Debates on ~v~~~at~~n. The purpose of the meeting was to synthesize our reactions and to draft a review of the book. Herewith is the edited and rearranged transcript of that meeting:
T. Schwa~dt~ Well, Peter, what are your general reactions to the book? F. Magold~~ Hummh. Overall, my reaction is mixed. The book lived up to the author’s assertion of a “most unusual book”’ [p* 91. Assembling a group of evaluation theoreticians and practitioners to debate issues on evaluation and then transcribing and publishing the conversations is a good idea, but it didn’t work well in this instance. There are several reasons for my tepid assessment _ _ _ T. ~chw~dt: “Tepid,” I like that8 it sums up my reactions as well. I liked the idea of exe perimenting with text formats, and the dialogue is at times thought-provoking, Yet this experiment isn’t particularly satisfying, in part, because of the disjointed and wide-ranging nature of the dialogue. I don’t think that the term “debate’” accurately characterizes most of the participant exchanges. It’s more like a discussion. And while seemingly cordial and frank, it’s not really a ~‘pe~etrati~~‘~ discussion as Alkin suggests. The evaluation waterfront is covered - 13 different issues are mentioned . _ . few are thoroughly discussed and less than that actually debated. There are exceptions, such as the exchange over social justice and serving client’s needs. [Flips through book to find pages.] Remember this exchange? 2%. J--Won:
E. House:
A@*Fat&m:
* . . .$rom m_v ~e~~~~~~~~~, I #ace a higkr ~~s~~~s~b~~~tyfm serving ~~~e~ts than on the broader aims. And that frame of reference does have the potential far getting into some differeerztarientations and doing things d~~fereentiy~and that would be interesting to put ta the test. We& how far would you pursue this orie~t~t~~~? Surely, you can ‘t consider yotcr opt& purpose to be meeting your &nt “s ~~t~~est~ Tell me why I can%