- Email: [email protected]
Debates on evaluation
102
Book Reviews
searchers will find this book especially intriguing, It is an exemplar in the use of ciearly designated and differentiated criteria for analyzing given sets of policy alternatives&Nagel’s chapter on effectiveness, efficiency, and equity is extremely illuminating; it is must-reading for
program plarmers and evaluators. Furthermore, this book should stimulate evaluative researchers to conduct methodological studies comparing and contrasting the relative merits and limitations of optimizing techniques with various evaluative research strategies.
Debates m I3wtluation by Marvin C. Alkin, with contributions by Michael Quinn Patton and Carol H, Weiss. Newbury Park, CA: Sage Publications, 1990, 302 pp. RevieMrers: Thomas A. Schwandt and Peter Magolda Sometime in the mid I98oS in Mahbu, ~~ifor~ia, Marvin Alkin convened a group of distinguished evaluation professionals (Ross Conner, Ernest House, Michael Kean, Jean King, Susan Klein, Alex Law, Milbrey Mctaughlin, Michael Patton, and Carol Weiss) for 3 days of informal talks. The meeting’s purpose was to ‘“synthesize positions on the current status of evaluation utilization: research, implications for practice, and potential new directions” [p+ 91. De&z&s on Evaluation is primarily an edited transcript of these unstructured discussions. Professor Alkin edited, rearranged, and sorted the ~ransc~b~d dialogue into four major parts. W’nhin each part, portions of the dialogue are identified further by I3 “‘chapters” (topics) and 50 subtopics. An Appendix lists all the topics and subtopics indexed to the fine numbers in the original transcript. Part I, ‘LEvaluation Utilization,” begins with five pages of introductory comments from each participant on the notion of “utilization” followed by comments on utilization factors and issues, and research, dissemination, and training. Part II, “Evaluation Theory,‘” introduces the reader to the participants’ views on the purpose and function of evafuation, and on the notions of evaluation audiences, the F& of the evafuator, and assorted methodologic~ issues. Part III, *~~val~ation Theory ~~stin~i~ns~ Further Debate,” departs from a presentation of the discussion that took place at the Malibu conference, It consists of four papers previously published in Evaluation Practice that comprise an exchange of views between Michael Patton and Carol Weiss on the purpose and use of evaluation Part IV, “Politics and Ethics,” returns to the transcript presentation format. Highlighted are political, ethica1, and misuse issues that evaluation professionals encounter. Each part concludes with a “Review and Commentary” by Professor Alkin. The comments of two ~~discussan~s” who read the book manuscript- Fred Ellett, Jr. and Michael Hendricks-also appear in the introductions to each part and in footnotes. In a few instances, footnotes also are used to present participants” clarifications and/or elaborations on their original comments. On July 9, 1991, we met in a Bloomington, Indiana diner for 2 hours of informal talks over breakfast about Debates on ~v~~~at~~n. The purpose of the meeting was to synthesize our reactions and to draft a review of the book. Herewith is the edited and rearranged transcript of that meeting:
T. Schwa~dt~ Well, Peter, what are your general reactions to the book? F. Magold~~ Hummh. Overall, my reaction is mixed. The book lived up to the author’s assertion of a “most unusual book”’ [p* 91. Assembling a group of evaluation theoreticians and practitioners to debate issues on evaluation and then transcribing and publishing the conversations is a good idea, but it didn’t work well in this instance. There are several reasons for my tepid assessment _ _ _ T. ~chw~dt: “Tepid,” I like that8 it sums up my reactions as well. I liked the idea of exe perimenting with text formats, and the dialogue is at times thought-provoking, Yet this experiment isn’t particularly satisfying, in part, because of the disjointed and wide-ranging nature of the dialogue. I don’t think that the term “debate’” accurately characterizes most of the participant exchanges. It’s more like a discussion. And while seemingly cordial and frank, it’s not really a ~‘pe~etrati~~‘~ discussion as Alkin suggests. The evaluation waterfront is covered - 13 different issues are mentioned . _ . few are thoroughly discussed and less than that actually debated. There are exceptions, such as the exchange over social justice and serving client’s needs. [Flips through book to find pages.] Remember this exchange? 2%. J--Won:
E. House:
A@*Fat&m:
* . . .$rom m_v ~e~~~~~~~~~, I #ace a higkr ~~s~~~s~b~~~tyfm serving ~~~e~ts than on the broader aims. And that frame of reference does have the potential far getting into some differeerztarientations and doing things d~~fereentiy~and that would be interesting to put ta the test. We& how far would you pursue this orie~t~t~~~? Surely, you can ‘t consider yotcr opt& purpose to be meeting your &nt “s ~~t~~est~ Tell me why I can%
Book Reviews
E. House: M. Patton:
E. House: M. Patton: E. House:
Pa Magolda:
Why, it’s an immoral position. 1 could argue it’s immoral to take anything else into account when that’s the person who is supposedly . . . You can’t. You can’t. It would be a long argument which you ‘[Ilose, Go for it. There are too many co~ntere~amp~es. For examplq, who has the money to purchase evaluations The most proper people in society. You serve only the most proper people in society~ You wouldn’t condone that, . . . (pp. AX-lO2J
Now that’s more of a debate, or at least a non-tepid discussion [laughing]. The fact that participants didn’t often run with ideas like that is what bothered me. Most of the exchange centered more on identifying issues and positions and not fully discussing or debating them, For example, the participants’ introductory comments on “utilization” [pp. 1%231 showcased divergent views on the nature of evaluation, many of which were never again mentioned. T. Schwandt: I would add to your list here of unfinished discussions, Conner’s idea for a metaevaluative Institute of Evaluation and Policy Analysis in the executive branch of government [p. 471, the role of practitioner-evaluators fp. 481, evaluators as promoters of their products fp. 37], and House’s idea of working with practitioner knowl~ge fp. SS-SS]. These are interesting ideas, and I wanted to hear the participants talk more about them. P. Magolda: I think that the unstructured meeting agenda contributed to these problems. Alkin did “sort” the transcript into topics and subtopics to achieve some continuity within and across ideas, but sorting can’t compensate for the underdevelopment of ideas. IIouse prefaced one of his comments with the statement “Without getting into a detailed discussion.” I appreciate his sensitivity to monopolizing the discussion. Yet his comment illuminates the book’s greatest shortcoming, T. Schwandt: Yet, Alkin anticipates and acknowledges this limitation and the participants’ “‘stream of consciousness” presentation style in the first commentary chapter. The four summary chapters compensate in part for the meeting format shortcomings,
103
True. This is the book’s greatest asset. Alkin does a masterful job of synthesizing disjointed conversations, recounting discussants’ comments, articulating the various theoretical positions, including his own, highlighting points of agreement and disagreement, and identifying issues in need of further consideration. It would have been better to use these chapters as the core of the book and supplement them with excerpts from the transcripts. I’ll admit, though, that capturing genuine dialogue about ideas and practices in the format of this book is a refreshing alternative to the traditional publications of academic conference proceedings. T. Schwandt: I agree on all counts. Did you feel you really got to know the points of view of all participants at the meeting? What did Alkin say? [Flipping through book, looking for correct page] Here it is: “this book presents the unusual opportunity for readers to come to know an important group of evaluation professionals and to begin to understand them and their viewpoints more fully” [p. lo]. P. Magolda: I would argue Alkin’s goal was only partially achieved. Several voices seemed to dominate the meeting’s agenda. By the end of the book, I was able to distinguish among Alkin’s [based on his summary chapters], House’s, Patton’s, and Weiss’s evafuation ideologies, but not the other participants. Not having read much of the work of the other participants, it was difficult to get a feel for their philosophies or points of view. I needed more than the two pages allocated to participants’ backgrounds at the beginning of the book and the snippets of conversation to overcome the sense of disembodied voices. What did you think about the inclusion of the previously published papers from Weiss and Patton? T. Schwandt: I’ve used them in class. They provide a nice point-counterpoint introduction to some important ideas about the nature and purpose of evaluation and the notion of evaluation use. They certainly fit the general theme of the book, but it seems that they could be left out without much damage to the book. P. Magoida: You know, not o&y is a fair portion of the book devoted to reprinting these published articles, but some conversaP. Magolda:
Book Reviews
tions are repeated in chapters of the book. For example, the McLaughlin and Patton discussion about the role of technology in evaluation is in the “Framing Questions” section of the “Role of the Evaluator” chapter and in the “Data for Multilevel Use” section of the “Other Methodological Issues” chapter. I was having deja vu all over again! T. Schwandt: Atkin apparently thought that because these conversations applied to several topics discussed in the book, it was necessary to repeat them. But he could just as easily have cross-referenced them in a footnote. Speaking of footnotes . . . What did you think about Hendricks and Ellett’s responses to the conversations in their footnotes? P. Magolda: The inclusion of discussants’ reactions is an excellent idea. Unfortunately, using footnotes as the medium didn’t allow respondents ample space for their critiques. For example, Hendricks, reacting to a participant’s reason for evaluation, stated “I find that we usually do evaluations for lots of different reasons, not just one or two” [p. 481. The reader never learns Hendrick’s reasons. Not only was the amount of response space insufficient, but the location of the comments was distracting. I found myself volleying between the dialogue and the notes. This format undermines the author’s intent of presenting a free-flowing diaIogue. T. Schwandt: I’d characterize the Elfett and Hendrick notes as running random kibitzing. I generally found the kibitzing more annoying than illuminating . . . Much of what we’ve been talking about, it seems, deals with the format and presentation of the book. For me, anyway, the form often inhibited my appreciation of the content. What do you think?
P. Magolda:
Maybe I appreciated the content more than you. The misuse discussion, including Alkin’s misuse categorization model [p. 2931, broadened my understanding of the ways in which evaluation can be misused. I liked the emphasis placed on the evaluator-client relationships and on evaluation planning. I also liked the idea that the text itself modeled the fluid nature of evaluation thinking and practice. T. Schwandt: interesting observations. I’m not sure for whom the book is tarP. Magolda: geted. As a graduate student, I would prefer reading many of the original publications referenced in the discussions, rather than this more fragmented dialogue. In defense of the book, it does provide glimpses into the research agendas and current thinking of an impressive collection of evaluation theoreticians and practitioners. It highlights the diversity of evaluation methodologies and methods. T. Schwandt: I think that graduate students interested in a loosely organized survey of issues in evaluation could benefit from this book provided it was supplemented by an examination and discussion of references of the kind you refer to. I think readers attracted to the title “debates” on evaluation will be somewhat disappointed, and they may not be pleased to find that a portion of the book is devoted to the Weiss and Patton papers. To be sure, important, interesting, and provocative issues that contribute to our understanding of evaluation practice are to be found in the dialogue . . . I need some more coffee; whose idea was it to meet this early, surely not mine. P. Magolda: I suppose you’re going to stick me with the bill . . .
Back to Work: Testing ~eErnpIo~rn~~t Service for DispIaeed Workers by Howard S. Boom. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 1990, 180 pp. Reviewer: Richard H. Price This book is a detailed review of what the author describes as “a rigorously designed and carefully implemented randomized field experiment to study the implementation, impacts, and costs of job-search assistance and retraining services for displaced workers” (p, v). The social importance of such evaluations is substantial. The number of displaced workers, that is, those
who have lost stable, well-paying jobs because of economic changes or changes in technology is estimated to be about 1 million a year, or about 10% of the unemployed. National efforts in the 1980s to respond to this problem were undertaken primarily under the Job Training Partnership Act (JTPA) enacted in 1982 and implemented in 1983. The author of this monograph
Book Reviews
searchers will find this book especially intriguing, It is an exemplar in the use of ciearly designated and differentiated criteria for analyzing given sets of policy alternatives&Nagel’s chapter on effectiveness, efficiency, and equity is extremely illuminating; it is must-reading for
program plarmers and evaluators. Furthermore, this book should stimulate evaluative researchers to conduct methodological studies comparing and contrasting the relative merits and limitations of optimizing techniques with various evaluative research strategies.
Debates m I3wtluation by Marvin C. Alkin, with contributions by Michael Quinn Patton and Carol H, Weiss. Newbury Park, CA: Sage Publications, 1990, 302 pp. RevieMrers: Thomas A. Schwandt and Peter Magolda Sometime in the mid I98oS in Mahbu, ~~ifor~ia, Marvin Alkin convened a group of distinguished evaluation professionals (Ross Conner, Ernest House, Michael Kean, Jean King, Susan Klein, Alex Law, Milbrey Mctaughlin, Michael Patton, and Carol Weiss) for 3 days of informal talks. The meeting’s purpose was to ‘“synthesize positions on the current status of evaluation utilization: research, implications for practice, and potential new directions” [p+ 91. De&z&s on Evaluation is primarily an edited transcript of these unstructured discussions. Professor Alkin edited, rearranged, and sorted the ~ransc~b~d dialogue into four major parts. W’nhin each part, portions of the dialogue are identified further by I3 “‘chapters” (topics) and 50 subtopics. An Appendix lists all the topics and subtopics indexed to the fine numbers in the original transcript. Part I, ‘LEvaluation Utilization,” begins with five pages of introductory comments from each participant on the notion of “utilization” followed by comments on utilization factors and issues, and research, dissemination, and training. Part II, “Evaluation Theory,‘” introduces the reader to the participants’ views on the purpose and function of evafuation, and on the notions of evaluation audiences, the F& of the evafuator, and assorted methodologic~ issues. Part III, *~~val~ation Theory ~~stin~i~ns~ Further Debate,” departs from a presentation of the discussion that took place at the Malibu conference, It consists of four papers previously published in Evaluation Practice that comprise an exchange of views between Michael Patton and Carol Weiss on the purpose and use of evaluation Part IV, “Politics and Ethics,” returns to the transcript presentation format. Highlighted are political, ethica1, and misuse issues that evaluation professionals encounter. Each part concludes with a “Review and Commentary” by Professor Alkin. The comments of two ~~discussan~s” who read the book manuscript- Fred Ellett, Jr. and Michael Hendricks-also appear in the introductions to each part and in footnotes. In a few instances, footnotes also are used to present participants” clarifications and/or elaborations on their original comments. On July 9, 1991, we met in a Bloomington, Indiana diner for 2 hours of informal talks over breakfast about Debates on ~v~~~at~~n. The purpose of the meeting was to synthesize our reactions and to draft a review of the book. Herewith is the edited and rearranged transcript of that meeting:
T. Schwa~dt~ Well, Peter, what are your general reactions to the book? F. Magold~~ Hummh. Overall, my reaction is mixed. The book lived up to the author’s assertion of a “most unusual book”’ [p* 91. Assembling a group of evaluation theoreticians and practitioners to debate issues on evaluation and then transcribing and publishing the conversations is a good idea, but it didn’t work well in this instance. There are several reasons for my tepid assessment _ _ _ T. ~chw~dt: “Tepid,” I like that8 it sums up my reactions as well. I liked the idea of exe perimenting with text formats, and the dialogue is at times thought-provoking, Yet this experiment isn’t particularly satisfying, in part, because of the disjointed and wide-ranging nature of the dialogue. I don’t think that the term “debate’” accurately characterizes most of the participant exchanges. It’s more like a discussion. And while seemingly cordial and frank, it’s not really a ~‘pe~etrati~~‘~ discussion as Alkin suggests. The evaluation waterfront is covered - 13 different issues are mentioned . _ . few are thoroughly discussed and less than that actually debated. There are exceptions, such as the exchange over social justice and serving client’s needs. [Flips through book to find pages.] Remember this exchange? 2%. J--Won:
E. House:
A@*Fat&m:
* . . .$rom m_v ~e~~~~~~~~~, I #ace a higkr ~~s~~~s~b~~~tyfm serving ~~~e~ts than on the broader aims. And that frame of reference does have the potential far getting into some differeerztarientations and doing things d~~fereentiy~and that would be interesting to put ta the test. We& how far would you pursue this orie~t~t~~~? Surely, you can ‘t consider yotcr opt& purpose to be meeting your &nt “s ~~t~~est~ Tell me why I can%
Book Reviews
E. House: M. Patton:
E. House: M. Patton: E. House:
Pa Magolda:
Why, it’s an immoral position. 1 could argue it’s immoral to take anything else into account when that’s the person who is supposedly . . . You can’t. You can’t. It would be a long argument which you ‘[Ilose, Go for it. There are too many co~ntere~amp~es. For examplq, who has the money to purchase evaluations The most proper people in society. You serve only the most proper people in society~ You wouldn’t condone that, . . . (pp. AX-lO2J
Now that’s more of a debate, or at least a non-tepid discussion [laughing]. The fact that participants didn’t often run with ideas like that is what bothered me. Most of the exchange centered more on identifying issues and positions and not fully discussing or debating them, For example, the participants’ introductory comments on “utilization” [pp. 1%231 showcased divergent views on the nature of evaluation, many of which were never again mentioned. T. Schwandt: I would add to your list here of unfinished discussions, Conner’s idea for a metaevaluative Institute of Evaluation and Policy Analysis in the executive branch of government [p. 471, the role of practitioner-evaluators fp. 481, evaluators as promoters of their products fp. 37], and House’s idea of working with practitioner knowl~ge fp. SS-SS]. These are interesting ideas, and I wanted to hear the participants talk more about them. P. Magolda: I think that the unstructured meeting agenda contributed to these problems. Alkin did “sort” the transcript into topics and subtopics to achieve some continuity within and across ideas, but sorting can’t compensate for the underdevelopment of ideas. IIouse prefaced one of his comments with the statement “Without getting into a detailed discussion.” I appreciate his sensitivity to monopolizing the discussion. Yet his comment illuminates the book’s greatest shortcoming, T. Schwandt: Yet, Alkin anticipates and acknowledges this limitation and the participants’ “‘stream of consciousness” presentation style in the first commentary chapter. The four summary chapters compensate in part for the meeting format shortcomings,
103
True. This is the book’s greatest asset. Alkin does a masterful job of synthesizing disjointed conversations, recounting discussants’ comments, articulating the various theoretical positions, including his own, highlighting points of agreement and disagreement, and identifying issues in need of further consideration. It would have been better to use these chapters as the core of the book and supplement them with excerpts from the transcripts. I’ll admit, though, that capturing genuine dialogue about ideas and practices in the format of this book is a refreshing alternative to the traditional publications of academic conference proceedings. T. Schwandt: I agree on all counts. Did you feel you really got to know the points of view of all participants at the meeting? What did Alkin say? [Flipping through book, looking for correct page] Here it is: “this book presents the unusual opportunity for readers to come to know an important group of evaluation professionals and to begin to understand them and their viewpoints more fully” [p. lo]. P. Magolda: I would argue Alkin’s goal was only partially achieved. Several voices seemed to dominate the meeting’s agenda. By the end of the book, I was able to distinguish among Alkin’s [based on his summary chapters], House’s, Patton’s, and Weiss’s evafuation ideologies, but not the other participants. Not having read much of the work of the other participants, it was difficult to get a feel for their philosophies or points of view. I needed more than the two pages allocated to participants’ backgrounds at the beginning of the book and the snippets of conversation to overcome the sense of disembodied voices. What did you think about the inclusion of the previously published papers from Weiss and Patton? T. Schwandt: I’ve used them in class. They provide a nice point-counterpoint introduction to some important ideas about the nature and purpose of evaluation and the notion of evaluation use. They certainly fit the general theme of the book, but it seems that they could be left out without much damage to the book. P. Magoida: You know, not o&y is a fair portion of the book devoted to reprinting these published articles, but some conversaP. Magolda:
Book Reviews
tions are repeated in chapters of the book. For example, the McLaughlin and Patton discussion about the role of technology in evaluation is in the “Framing Questions” section of the “Role of the Evaluator” chapter and in the “Data for Multilevel Use” section of the “Other Methodological Issues” chapter. I was having deja vu all over again! T. Schwandt: Atkin apparently thought that because these conversations applied to several topics discussed in the book, it was necessary to repeat them. But he could just as easily have cross-referenced them in a footnote. Speaking of footnotes . . . What did you think about Hendricks and Ellett’s responses to the conversations in their footnotes? P. Magolda: The inclusion of discussants’ reactions is an excellent idea. Unfortunately, using footnotes as the medium didn’t allow respondents ample space for their critiques. For example, Hendricks, reacting to a participant’s reason for evaluation, stated “I find that we usually do evaluations for lots of different reasons, not just one or two” [p. 481. The reader never learns Hendrick’s reasons. Not only was the amount of response space insufficient, but the location of the comments was distracting. I found myself volleying between the dialogue and the notes. This format undermines the author’s intent of presenting a free-flowing diaIogue. T. Schwandt: I’d characterize the Elfett and Hendrick notes as running random kibitzing. I generally found the kibitzing more annoying than illuminating . . . Much of what we’ve been talking about, it seems, deals with the format and presentation of the book. For me, anyway, the form often inhibited my appreciation of the content. What do you think?
P. Magolda:
Maybe I appreciated the content more than you. The misuse discussion, including Alkin’s misuse categorization model [p. 2931, broadened my understanding of the ways in which evaluation can be misused. I liked the emphasis placed on the evaluator-client relationships and on evaluation planning. I also liked the idea that the text itself modeled the fluid nature of evaluation thinking and practice. T. Schwandt: interesting observations. I’m not sure for whom the book is tarP. Magolda: geted. As a graduate student, I would prefer reading many of the original publications referenced in the discussions, rather than this more fragmented dialogue. In defense of the book, it does provide glimpses into the research agendas and current thinking of an impressive collection of evaluation theoreticians and practitioners. It highlights the diversity of evaluation methodologies and methods. T. Schwandt: I think that graduate students interested in a loosely organized survey of issues in evaluation could benefit from this book provided it was supplemented by an examination and discussion of references of the kind you refer to. I think readers attracted to the title “debates” on evaluation will be somewhat disappointed, and they may not be pleased to find that a portion of the book is devoted to the Weiss and Patton papers. To be sure, important, interesting, and provocative issues that contribute to our understanding of evaluation practice are to be found in the dialogue . . . I need some more coffee; whose idea was it to meet this early, surely not mine. P. Magolda: I suppose you’re going to stick me with the bill . . .
Back to Work: Testing ~eErnpIo~rn~~t Service for DispIaeed Workers by Howard S. Boom. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 1990, 180 pp. Reviewer: Richard H. Price This book is a detailed review of what the author describes as “a rigorously designed and carefully implemented randomized field experiment to study the implementation, impacts, and costs of job-search assistance and retraining services for displaced workers” (p, v). The social importance of such evaluations is substantial. The number of displaced workers, that is, those
who have lost stable, well-paying jobs because of economic changes or changes in technology is estimated to be about 1 million a year, or about 10% of the unemployed. National efforts in the 1980s to respond to this problem were undertaken primarily under the Job Training Partnership Act (JTPA) enacted in 1982 and implemented in 1983. The author of this monograph