- Email: [email protected]
Educational evaluation: Alternative approaches and practical guidelines
Book Reviews and consultants, all knowledgeable about the subject of their chapters. In this review, we offer selective and evaluative comments on several representative chapters rather than a detailed narrative of all the contributions. The first section begins with Gherson and Moore’s description of training in the implementation of corporate strategic change. Training, as an intervention, is compared with other available means of fostering organization change. Alternatives such as additional recruitment or acquisition are compared, and their relative costs and benefits juxtaposed. This chapter answers the question, When is training the best avenue for managing organizational change? The answers are accurate, thoughtful, and concise. In “The Strategic Evaluation of Training,” MacDonald starts off by asking the question, How can training prove its worth? He answers by establishing criteria for determining how this can be accomplished. The usefulness of MacDonald’s insights, however, is weakened by his obscure prose. The last chapter of the first section is perhaps the best in the volume. Ingols looks at the scope of business and industry training programs. In this well-written piece, Ingols reviews the extant literature on management education and other business training programs. Especially useful is a rich bibliography of more than 100 references. In Section II, the authors assume that training is like any other consumable product and thus can be evaluated in the same fashion as any other corporate product. The section begins with Arnoff’s characterization of evaluation’s role in the development of training. He shows that evaluation can strengthen the development process through needs assessments, pilot testing, and program evaluation. May focuses on course improvement through application of quality concepts to training development. He uses four Deming principles: customer satisfaction, focus on prevention, management responsibility for quality, and constant improvement. With each application, he demonstrates how course development can be substantially improved. Excellence in training ensures that what is taught directly reflects the tasks that need to be accomplished. This is the subject of Goldstein’s chapter. Goldstein stresses the development of knowledge, skills, and attitudes from job analysis as a way to ensure this traceability.
Educational Evaluation: Sanders. White Plains, o-8013-0128-9. Reviewers:
Melinda
Alternative Approaches New York: Longman,
L. Costello
415
Clark examines the evaluation of educational media, especially computer-based instruction (CBI). This chapter is confusing and, given Clark’s eminence in the field, somewhat troubling. Clark reviews the meta-analyses on CBI, citing the work done by Kulik and his associates at the University of Michigan. In one of these, the metaanalysis of Japanese CBI studies, Clark incorrectly reports that Shwalb, Shwalb, and Azuma calculated a .41 negative effect size (the effect size is actually positive .41), indicating a substantial advantage for instructor-based training. On the basis of this wrong reporting, he goes on to speculate that Japanese courseware development lags considerably behind that of the United States and thus explains the apparent contradiction. The final section contains two chapters that examine the communication of evaluation results from the perspective of the internal and external consultant. Brandenburg presents a case study of his evaluation of Motorola’s Total Quality Improvement program. The case study illustrates a number of conceptual procedures that consultants can use to increase the usability of their findings and suggestions. Zammit examines the communication of the results of an internal sales training program at Digital Equipment Corporation. She outlines five easily identifiable steps in the communication process, from identifying the audience to conducting follow-up analyses of the reactions to the report. Hers is a concise and well-explicated contribution. Generally of high quality, this book establishes a firm foundation for subsequent evaluation of corporate training. The high caliber of the majority of the chapters is even more striking given the context in which the chapters were written. Corporate personnel who wish to disseminate the results of their evaluation are generally required to get permission from their companies. Often the review process is tedious and difficult because some companies do not wish to offer their competitors any advantages, Consultants are similarly restrained by nondisclosure clauses in their contracts. In companies that do permit easy access to a wider audience, personnel generally find that their efforts go unrewarded. The consequence of these obstacles is that corporate training evaluation is in a dismal state. This book, by contrast, provides some of the best writing on the subject.
and Practical Guidelines, by Blaine R. Worthen and James R. Inc., 1987, xiv + 450 pp. ISBN O-582-28551-8 (pbk) ISBN
and Elisa J. Slee
In Educational Evaluation, Worthen and Sanders combine the theoretical issues, debates, and concepts con-
cerning evaluation with the guidelines, procedures, and helpful hints of conducting evaluations. If an overview
416
Book Reviews
text that combines both is needed, Educational Evaiuation should be considered with an understanding of the limitations and strengths of the book. The book is divided into four parts: Introduction to Evaluation, Alternative Approaches to Educational Evaluation, Practical Guidelines to Planning Evaluations, and Practical Guidelines for Conducting and Using Evaluations. Part One, Introduction to Evaluation, provides a broad overview of the concept of evaluation including the history of evaluation and the role evaluation plays in improving education. In addition to defining basic evaluation concepts such as the purposes, audiences, and goals of evaluation, Worthen and Sanders distinguish between internal and external evaluations. They combine the distinction between internal and external evaluations with a discussion of formative and summative evaluation. Tables display the common roles and purposes which different types of evaluation serve. Although the information in Part One has been presented in other evaluation texts, Worthen and Sander’s discussion is informative and should provide the individual who is new to evaluation with an understanding of the foundations of evaluation. In Part Two, the authors explore the philosophical origins of evaluation approaches and describe the following six alternative approaches to educational evaluations: objectives-oriented, management-oriented, consumeroriented, expertise-oriented, adversary-oriented, and naturalistic- and participant-oriented. One chapter is allocated to each approach and includes a discussion of the strengths and weaknesses of each approach. Part Two includes a discussion of qualitative and quantitative inquiry. The presentation simplifies the current debate regarding qualitative and quantitative approaches and contradicts information Worthen and Sanders present earlier in the text. The authors argue for five pages in Part One that, except for similarities in methods, research and evaluation are not the same, “We agree that research is not equivalent to evaluation, regardless of its modifying adjectives, for, as argued earlier, research and evaluation differ in purpose even when they use the same methods and techniques” (p. 28). However, in Part Two, Worthen and Sanders contradict their own argument when they use examples from research to support their position on the relationship between qualitative and quantitative evaluation. Although the research versus evaluation and the qualitative versus quantitative debates help establish a context in which to read the evaluation model descriptions, such extensive coverage does not seem appropriate in this book. The debates should be introduced but the authors should neither attempt nor purport to solve the debates as they have here. The chapters devoted to each of the six different evaluation approaches are followed by a chapter which con-
tains a comparative analysis of the six alternatives. The greatest strength of this chapter is a figure which provides an excellent review of the approaches. The comparative analysis concludes with a description of an eclectic approach to educational evaluation which is the approach that the authors advocate. Parts Three and Four, described by the authors as the most significant part of the book, include extensive practical guidelines for planning and conducting evaluation studies. Part Three includes everything expected in a cookbook approach for planning an evaluation: clarifying the evaluation request, analyzing the evaluation context, selecting questions, and developing management plans for evaluation. The narrative relates and supplements the numerous charts and checklists included in this section such as checklists which can be used to select an evaluator, to formulate evaluation questions, and to determine when to conduct an evaluation. The topics included in Part Four, Practical Guidelines for Conducting and Using Evaluations, are the political, ethical, and interpersonal considerations of evaluations, along with data analysis and data interpretation. Part Four includes an excellent chapter on collecting and using information which is supplemented by a chart listing and describing the variety of ways information can be collected and how each collection technique can be used. Part Four also presents a list of the 30 Joint Committee evaluation standards, guidelines for using and conducting meta-evaluations, and a summary of the formative and summative uses of evaluation. Programs ~vaiuat~o~ (1983), a text by Brinkerhoff, Brethower, Hluchyj, and Nowakowski, is a natural comparison with Parts Three and Four of Worthen and Sanders. The Brinkerhoff et al. (1983) text is composed of checklists and tables with little of the narrative which is essential to a beginning student. Worthen and Sanders provide valuable linking narrative and also include some information missing from Brinkerhoff et al., such as checklists which can be used to determine whether or not to use an internal or external evaluator, to decide whether or not it is appropriate to do an evaluation, and to select an evaluator. Again, however, the material in this section is for the most part a repeat of material which has already been presented in other texts. The chapter in Part Four on data collection and data analysis in Educational Evaiuation, however, is incomplete. Worthen and Sanders provide no information on data-analysis methods except to refer the reader to other texts such as Brinkerhoff et al. (1983) for, “. . . readable summaries of data-analysis methods” (p. 331). Up to this point in the text, Worthen and Sanders provide enough information about each step of an evaluation so that the book couId be the sole text in an introductory
Book
course. With the omission of even a cursory review of data-analysis procedures, the reader is forced to consult additional sources. The practical guidelines in Parts Three and Four are supplemented with a case study adapted from an earlier publication by Worthen. The authors include the case study, “To help readers apply the content of the chapters . . .” (p. 162). The fictional case study is written in diary format by an evaluator hired to evaluate the Radnor Middle School Humanities program. The case study itself is an excellent idea, however, in its present form, it is one of the most annoying weaknesses of the book. The authors probably thought that the folksy, diary style would be amusing and warm, and would increase the readers’ comfort with new concepts and practices. Unfortunately, they have gone too far. The case study is insulting and irritating. In the case study Mrs. Janson, the evaluator’s contact at the school, is described condescendingly, “. . . Mrs. Janson immediately saw the wisdom and advantage in my suggestion. She seems delightfully perceptive” (p. 202). There are also sarcastic comments about fellow evalucolleagues who ators, “Were I like some clairvoyant seem not to need much information about a program to launch a full-fledged evaluation . . .” (p. 182). The authors frequently stray from details of the evaluation to irrelevant personal anecdotes, “. . . with the addition of the Radnor contract I took on, this year my cup ‘runneth over,’ so to speak. But so did the river-with a flood unlike anything ever seen in this country. The mountain snowfall this winter was over 200 percent of normal, and when the Spring thaw hit, half our valley was under water, or so it seemed” (p. 325). Excessive personal tales unrelated to the Radnor evaluation distract the reader. Worthen and Sanders should have left the case study in its original form (see Worthen, 1981), without the unnecessary personal anecdotes. Both the weakest and strongest elements of the text are found in Parts Three and Four, the last two parts of the book. The strongest element is composed of the checklists, summaries, and supportive text which are practical, helpful, and informative. The weakest ele-
Reviews
417
ment is represented by the distracting personal anecdotes of the case study. One additional strength of the book is the orienting questions at the beginning of each chapter which introduce the reader to important concepts to be covered. The orienting questions are tied to the content of each chapter and effectively summarize each chapter for the reader. The application questions at the end of each chapter might be useful essay exam questions or projects for classroom discussions, but they do not serve as an opportunity for the reader to review the material in the chapter as Worthen and Sanders intended. For example, at the end of the chapter on clarifying the evaluation request, the first application exercise question is, “Develop a list of questions that you can keep by your telephone in case requests for your evaluation services begin to accelerate” (p. 185). Worthen and Sanders intended the text to be a reference book for graduate students in evaluation courses, and related administration, curriculum, or teacher education courses. In addition, they feel that the text would be an appropriate reference book for practicing evaluators, professors, and students desiring an overview of evaluation. Despite their intended audience, the text is most appropriate for a survey course in evaluation designed to provide a broad overview of evaluation in the field of education. The first sections of the book give the beginning student a strong background and understanding of evaluation which sets the stage for planning and conducting evaluations. If evaluations need to be designed or conducted in the context of the course, however, additional references would be required. Worthen and Sanders have combined the models and theoretical issues of evaluation together with the practical guidelines for conducting evaluations in one text. The text does not further existing knowledge in evaluation; it is simply a compilation of information which has been previously presented in two separate volumes. If Educational Evaluation is used as a text, its weaknesses must be compensated for by skipping the distracting elements of the case study, finding other uses for the application questions, and including additional references on data analysis.
REFERENCES BRINKERHOFF, R.O., BRETHOWER, D.M., HLUCHYJ, T., & NOWAKOWSKI, J.R. (1983). Program evaluation: A practitioner’s guide for trainers and educators. Boston: Kluwer-Nijhoff.
WORTHEN, B.R. (1981). Journal entries of an eclectic evaluator. In R.S. Brandt (Ed.), Appliedstrategiesfor curriculum evaluation. Alexandria, VA: Association for Supervision and Curriculum Development.
Educational Evaluation: Sanders. White Plains, o-8013-0128-9. Reviewers:
Melinda
Alternative Approaches New York: Longman,
L. Costello
415
Clark examines the evaluation of educational media, especially computer-based instruction (CBI). This chapter is confusing and, given Clark’s eminence in the field, somewhat troubling. Clark reviews the meta-analyses on CBI, citing the work done by Kulik and his associates at the University of Michigan. In one of these, the metaanalysis of Japanese CBI studies, Clark incorrectly reports that Shwalb, Shwalb, and Azuma calculated a .41 negative effect size (the effect size is actually positive .41), indicating a substantial advantage for instructor-based training. On the basis of this wrong reporting, he goes on to speculate that Japanese courseware development lags considerably behind that of the United States and thus explains the apparent contradiction. The final section contains two chapters that examine the communication of evaluation results from the perspective of the internal and external consultant. Brandenburg presents a case study of his evaluation of Motorola’s Total Quality Improvement program. The case study illustrates a number of conceptual procedures that consultants can use to increase the usability of their findings and suggestions. Zammit examines the communication of the results of an internal sales training program at Digital Equipment Corporation. She outlines five easily identifiable steps in the communication process, from identifying the audience to conducting follow-up analyses of the reactions to the report. Hers is a concise and well-explicated contribution. Generally of high quality, this book establishes a firm foundation for subsequent evaluation of corporate training. The high caliber of the majority of the chapters is even more striking given the context in which the chapters were written. Corporate personnel who wish to disseminate the results of their evaluation are generally required to get permission from their companies. Often the review process is tedious and difficult because some companies do not wish to offer their competitors any advantages, Consultants are similarly restrained by nondisclosure clauses in their contracts. In companies that do permit easy access to a wider audience, personnel generally find that their efforts go unrewarded. The consequence of these obstacles is that corporate training evaluation is in a dismal state. This book, by contrast, provides some of the best writing on the subject.
and Practical Guidelines, by Blaine R. Worthen and James R. Inc., 1987, xiv + 450 pp. ISBN O-582-28551-8 (pbk) ISBN
and Elisa J. Slee
In Educational Evaluation, Worthen and Sanders combine the theoretical issues, debates, and concepts con-
cerning evaluation with the guidelines, procedures, and helpful hints of conducting evaluations. If an overview
416
Book Reviews
text that combines both is needed, Educational Evaiuation should be considered with an understanding of the limitations and strengths of the book. The book is divided into four parts: Introduction to Evaluation, Alternative Approaches to Educational Evaluation, Practical Guidelines to Planning Evaluations, and Practical Guidelines for Conducting and Using Evaluations. Part One, Introduction to Evaluation, provides a broad overview of the concept of evaluation including the history of evaluation and the role evaluation plays in improving education. In addition to defining basic evaluation concepts such as the purposes, audiences, and goals of evaluation, Worthen and Sanders distinguish between internal and external evaluations. They combine the distinction between internal and external evaluations with a discussion of formative and summative evaluation. Tables display the common roles and purposes which different types of evaluation serve. Although the information in Part One has been presented in other evaluation texts, Worthen and Sander’s discussion is informative and should provide the individual who is new to evaluation with an understanding of the foundations of evaluation. In Part Two, the authors explore the philosophical origins of evaluation approaches and describe the following six alternative approaches to educational evaluations: objectives-oriented, management-oriented, consumeroriented, expertise-oriented, adversary-oriented, and naturalistic- and participant-oriented. One chapter is allocated to each approach and includes a discussion of the strengths and weaknesses of each approach. Part Two includes a discussion of qualitative and quantitative inquiry. The presentation simplifies the current debate regarding qualitative and quantitative approaches and contradicts information Worthen and Sanders present earlier in the text. The authors argue for five pages in Part One that, except for similarities in methods, research and evaluation are not the same, “We agree that research is not equivalent to evaluation, regardless of its modifying adjectives, for, as argued earlier, research and evaluation differ in purpose even when they use the same methods and techniques” (p. 28). However, in Part Two, Worthen and Sanders contradict their own argument when they use examples from research to support their position on the relationship between qualitative and quantitative evaluation. Although the research versus evaluation and the qualitative versus quantitative debates help establish a context in which to read the evaluation model descriptions, such extensive coverage does not seem appropriate in this book. The debates should be introduced but the authors should neither attempt nor purport to solve the debates as they have here. The chapters devoted to each of the six different evaluation approaches are followed by a chapter which con-
tains a comparative analysis of the six alternatives. The greatest strength of this chapter is a figure which provides an excellent review of the approaches. The comparative analysis concludes with a description of an eclectic approach to educational evaluation which is the approach that the authors advocate. Parts Three and Four, described by the authors as the most significant part of the book, include extensive practical guidelines for planning and conducting evaluation studies. Part Three includes everything expected in a cookbook approach for planning an evaluation: clarifying the evaluation request, analyzing the evaluation context, selecting questions, and developing management plans for evaluation. The narrative relates and supplements the numerous charts and checklists included in this section such as checklists which can be used to select an evaluator, to formulate evaluation questions, and to determine when to conduct an evaluation. The topics included in Part Four, Practical Guidelines for Conducting and Using Evaluations, are the political, ethical, and interpersonal considerations of evaluations, along with data analysis and data interpretation. Part Four includes an excellent chapter on collecting and using information which is supplemented by a chart listing and describing the variety of ways information can be collected and how each collection technique can be used. Part Four also presents a list of the 30 Joint Committee evaluation standards, guidelines for using and conducting meta-evaluations, and a summary of the formative and summative uses of evaluation. Programs ~vaiuat~o~ (1983), a text by Brinkerhoff, Brethower, Hluchyj, and Nowakowski, is a natural comparison with Parts Three and Four of Worthen and Sanders. The Brinkerhoff et al. (1983) text is composed of checklists and tables with little of the narrative which is essential to a beginning student. Worthen and Sanders provide valuable linking narrative and also include some information missing from Brinkerhoff et al., such as checklists which can be used to determine whether or not to use an internal or external evaluator, to decide whether or not it is appropriate to do an evaluation, and to select an evaluator. Again, however, the material in this section is for the most part a repeat of material which has already been presented in other texts. The chapter in Part Four on data collection and data analysis in Educational Evaiuation, however, is incomplete. Worthen and Sanders provide no information on data-analysis methods except to refer the reader to other texts such as Brinkerhoff et al. (1983) for, “. . . readable summaries of data-analysis methods” (p. 331). Up to this point in the text, Worthen and Sanders provide enough information about each step of an evaluation so that the book couId be the sole text in an introductory
Book
course. With the omission of even a cursory review of data-analysis procedures, the reader is forced to consult additional sources. The practical guidelines in Parts Three and Four are supplemented with a case study adapted from an earlier publication by Worthen. The authors include the case study, “To help readers apply the content of the chapters . . .” (p. 162). The fictional case study is written in diary format by an evaluator hired to evaluate the Radnor Middle School Humanities program. The case study itself is an excellent idea, however, in its present form, it is one of the most annoying weaknesses of the book. The authors probably thought that the folksy, diary style would be amusing and warm, and would increase the readers’ comfort with new concepts and practices. Unfortunately, they have gone too far. The case study is insulting and irritating. In the case study Mrs. Janson, the evaluator’s contact at the school, is described condescendingly, “. . . Mrs. Janson immediately saw the wisdom and advantage in my suggestion. She seems delightfully perceptive” (p. 202). There are also sarcastic comments about fellow evalucolleagues who ators, “Were I like some clairvoyant seem not to need much information about a program to launch a full-fledged evaluation . . .” (p. 182). The authors frequently stray from details of the evaluation to irrelevant personal anecdotes, “. . . with the addition of the Radnor contract I took on, this year my cup ‘runneth over,’ so to speak. But so did the river-with a flood unlike anything ever seen in this country. The mountain snowfall this winter was over 200 percent of normal, and when the Spring thaw hit, half our valley was under water, or so it seemed” (p. 325). Excessive personal tales unrelated to the Radnor evaluation distract the reader. Worthen and Sanders should have left the case study in its original form (see Worthen, 1981), without the unnecessary personal anecdotes. Both the weakest and strongest elements of the text are found in Parts Three and Four, the last two parts of the book. The strongest element is composed of the checklists, summaries, and supportive text which are practical, helpful, and informative. The weakest ele-
Reviews
417
ment is represented by the distracting personal anecdotes of the case study. One additional strength of the book is the orienting questions at the beginning of each chapter which introduce the reader to important concepts to be covered. The orienting questions are tied to the content of each chapter and effectively summarize each chapter for the reader. The application questions at the end of each chapter might be useful essay exam questions or projects for classroom discussions, but they do not serve as an opportunity for the reader to review the material in the chapter as Worthen and Sanders intended. For example, at the end of the chapter on clarifying the evaluation request, the first application exercise question is, “Develop a list of questions that you can keep by your telephone in case requests for your evaluation services begin to accelerate” (p. 185). Worthen and Sanders intended the text to be a reference book for graduate students in evaluation courses, and related administration, curriculum, or teacher education courses. In addition, they feel that the text would be an appropriate reference book for practicing evaluators, professors, and students desiring an overview of evaluation. Despite their intended audience, the text is most appropriate for a survey course in evaluation designed to provide a broad overview of evaluation in the field of education. The first sections of the book give the beginning student a strong background and understanding of evaluation which sets the stage for planning and conducting evaluations. If evaluations need to be designed or conducted in the context of the course, however, additional references would be required. Worthen and Sanders have combined the models and theoretical issues of evaluation together with the practical guidelines for conducting evaluations in one text. The text does not further existing knowledge in evaluation; it is simply a compilation of information which has been previously presented in two separate volumes. If Educational Evaluation is used as a text, its weaknesses must be compensated for by skipping the distracting elements of the case study, finding other uses for the application questions, and including additional references on data analysis.
REFERENCES BRINKERHOFF, R.O., BRETHOWER, D.M., HLUCHYJ, T., & NOWAKOWSKI, J.R. (1983). Program evaluation: A practitioner’s guide for trainers and educators. Boston: Kluwer-Nijhoff.
WORTHEN, B.R. (1981). Journal entries of an eclectic evaluator. In R.S. Brandt (Ed.), Appliedstrategiesfor curriculum evaluation. Alexandria, VA: Association for Supervision and Curriculum Development.