- Email: [email protected]
Transforming qualitative data: Description, analysis and interpretation
Book Reviews
dominantly by categorizing aspects of evaluation and offering brief descriptions and definitions of these categories. The subsequent chapters elaborate upon the following steps in the process of evaluation. Through the use of short sentences, numbering, and listing, the content is presented in a clear and simple style. The last chapter of the book focuses on planning and budgeting, often overlooked topics in evaluation texts, and is a good conclusion. The book’s strength in presenting a great deal of complex material in a simple, straightforward manner is also its downfall. First, its conciseness and organized outline make it a useful desk reference for advanced students, program evaluators and those being evaluated. However, this same conciseness makes it useful only for those who already have a background in program evaluation. It would be appropriate as an introductory text to program evaluation only if it were supplemented with other readings, particularly those that focus on other equally important approaches to evaluation. Additionally, while the organization is tight and focused, thereby guiding the reader through complex materials, the organization of the chapters and the transitions between chapters suggest that program evaluation is a linear process, presenting it as somewhat simplistic and task-oriented. Also, although this book is strong on the “how-to” for putting together an experimental evaluation study, it is weak on the “how-to” for qualitative/participatory evaluation and the “whyto” for all types of evaluation. There is no discussion of the meaning of program evaluation. There is no discussion about the role of the evaluator. Also, there
Transforming
275
is no discussion of the social implications or responsibilities of program evaluation, and no sense of the evaluation taking place through human interactions for the purpose of enhancing our quality of life and contributing to social betterment. Therefore, this book would not be appropriate for a class that focuses on understanding theories, concepts and philosophy of program evaluation. Finally, although the book is comprehensive in the terms of covering the experimental perspective of evaluation, it seems the book is more about design and methods than about evaluation, per se. Particularly when the book deals with sampling design and statistical analysis (for example which statistic to use), there is no clear sense of adapting these tools specifically to program evaluation applications. Other than the first chapter and the examples, the book does not seem to be, specifically, about evaluation-it could be about doing experimental research in educational and human service contexts. “The book is designed to cover the field and respect the needs of students in education and psychology, as well as those of teachers, counsellors, experts in curriculum and instruction, testing specialists, and anyone else interested in understanding and improving teaching and learning”, (p, xi). In fact, the text does not cover the field, but does a pretty good job of presenting one perspective in the field of program evaluation, and will appeal to individuals who share this perspective. The book is certainly introductory level material, but most likely it would have to be supplemented with other readings to be adoptable for courses. The book is a nice desk reference on an experimental approach to program evaluation.
Qualitative Data: Description, Analysis and Interpretation, by H. F. Wolcott, Thousand
Oaks, CA:
Sage, 1994,433 pp. Reviewer:
Linda Mabry
Most if not all practicing ethnographers and teachers of research or evaluation methodology would concede that the toughest part of qualitative work is to describe data analysis for students, for a proposal, or for the methods section of a report. Often it is harder to describe qualitative data analysis than it is to do it. How do we find meaning in interview transcripts, observation notes, or the minutes of meetings? What is the nature, the ontology, the process of insight? As a friend who will “tell it like it is”, Harry Wolcott strides into this thorniest thicket of qualitative work,
the “black box” of what really happens between data collection (or “fieldwork”, his preferred term) and completed research report. In Transforming Qualitative Data: Description,
Analysis,
and Interpretation
(I 994),
he eschews the scientistic, step-by-step accounts found in some textbooks and articles of such procedures as categorization as a precursor to identification of trends, or development of typologies as a phase in theory generation, or enumeration of coded data to refine or verify hypotheses. Despite their perfunctory admonitions to the contrary, advocates of more-or-less specific actions to be taken at more-or-less specific points in a post-data collection timetable strongly suggest that qualitative
276
Book Reviews
data analysis is a sequential macro-activity specifiable in detail in advance. But does this help us understand or evaluate the phenomena we study or the perceptions and meanings of those who operate within them, or merely help us respond to our obligations to describe the ineffable process of meaning-making? Wolcott offers an alternative, while understanding the impetus for magniloquent formulae: Because qualitative data gathering is conducted through such everyday techniques as participant observation and interviewing, it is comforting to employ a term like “analysis” to suggest that in what we do with the data we are able to wrest them from their humble origins and transform them into something grand enough to pass for science (pp. 24-25).
How grand is qualitative data analysis? Consider his distilled advice to doctoral students: “Tell the story. Then tell how that happened to be the way you told it” (p. 16). Wolcott concedes the things we study are dissimilar, as are the kinds and depths of understandings we reach, as are we researchers ourselves. Exactly what we do in the effort to apprehend the fundamental nature of these things necessarily varies. Adaptation of methods to phenomena, to the data available, to researcher preference and experience, and more preclude detailed advance specification of interpretive processes. How our minds work in developing our understandings may not be describable except in broad strokes. Wolcott offers a general outline and a cornucopia of advice. To the triumvirate procedures of qualitative data gathering-observation, interview, and documents review-Wolcott adds three similarly broad and flexible categories to describe the process of data “transformation”: description, analysis, and interpretation. (Wolcott prefers the term “transformation” to “analysis” as the overarching term for the entire process because he stipulates a more constrained definition for analysis as one of the three processes within transformation. This stipulation, which differs from widespread current usage, unfortunately invites confusion.) These three processes may be used exclusively or in combination. Transformation of data by description produces a report characterized largely by data presentation with relatively little (or perhaps relatively implicit) interpretation. Separate from or building upon description, transformation may involve analysis, “some careful, systematic way to identify key factors and relationships among them” (p. 10). From either or both description and analysis, or separate from them, data may be interpreted “to reach out for understanding or explanation beyond the limits of what can be explained with the degree of certainty usually associated with analysis” (pp. 10-l 1). Wolcott properly notes that distinctions about the nature of these three processes are decidedly blurred and that, when using all three, they will undoubtedly overlap in a rough-and-tumble dia-
lectic rather than progress in stately and predictable linearity. Another variability: qualitative data analysis is judgment-intensive and, consequently, idiosyncratic. Early on, Wolcott warns his readers of the absence of definitive techniques or professional secrets: Do qualitative researchers have some special way of describing things and rendering those descriptions in final accounts? Not that I know of.. (p. 14) Every decision about the appropriate level of detail returns ultimately to the immediate purposes being addressed and to over-riding considerations that map a course between extremes of too-selective reporting or hopeless obfuscation. (p. 14) If I could provide the magic formula for recognizing, remembering, recording, and subsequently transforming relevant data [into descriptive accounts], here is where I would reveal it. But the only advice I offer is to make judgments in terms of sufficiency and to screen all description in those terms, the data to be excluded as well as the data to be included. (emphasis in the original, pp. 14-15) Just as some researchers prefer to build their cases essentially through the descriptive account, others appear obviously impatient to get beyond “mere description” . ‘ (p. 15).
Some may consider Wolcott’s move away from recipe-like delineation of process a retreat into chaos. What, then, are we to tell our students and potential funders and clients about how we deal with qualitative data? For the evaluator, as for the researcher, there is benefit from Wolcott’s unsettling admissions that valid interpretation is not guaranteed by certain actions, and that faithful adherence to particular procedures does not ensure trustworthy findings. Rather, we must strive for empirically well-warranted and logically or theoretically coherent renderings. Instead of issuing directives, Wolcott shares suggestions gleaned from his professional life: ten ways one might tell a story or narrative, eleven ways to approach interpretation, four strategies for ethnographic observation, and more. Even where Wolcott enumerates, he is quick to announce that his lists are non-exhaustive. Following didactic sections are examples of his own work, nine completed studies which have appeared earlier in some published form, including such longstanding favorites as “Sneaky Kid”, “The Man in the Principal’s Office”, and “The Teacher as an Enemy”. While there is little explicit attention directed toward the analysis of qualitative data for the purposes of program evaluation in this volume’, there is much that is useful to program evaluators and to students of program evaluation, partly because the field has increasingly utilized qualitative methods. The volume would
‘See pp. 179-180. ‘See, for example, LeCompte, M. D. & Preissle, J. (1993). Ethnography and qualitative design in educational research (2nd ed.), NY: Academic Press; and Miles, M. B. & Huberman, A. M. (1984). Qualitative data analysis: A sourcebook ofnew methods. Beverly Hills, CA: Sage Publications.
Book Reviews have greater appeal to this audience if it included an account or two which comprise or supplement evaluation reports, but many practicing evaluators have access to such documents-and Wolcott is not an evaluator by profession. Perhaps particularly in evaluation where market and political forces intrude into inquiry
277
and demand slick credibility, advocates of a more delineated approach to qualitative data analysis have exerted strong appea12. Wolcott offers an important counterpoint, one which would make a fine supplementary reading to a course on evaluation methods.
dominantly by categorizing aspects of evaluation and offering brief descriptions and definitions of these categories. The subsequent chapters elaborate upon the following steps in the process of evaluation. Through the use of short sentences, numbering, and listing, the content is presented in a clear and simple style. The last chapter of the book focuses on planning and budgeting, often overlooked topics in evaluation texts, and is a good conclusion. The book’s strength in presenting a great deal of complex material in a simple, straightforward manner is also its downfall. First, its conciseness and organized outline make it a useful desk reference for advanced students, program evaluators and those being evaluated. However, this same conciseness makes it useful only for those who already have a background in program evaluation. It would be appropriate as an introductory text to program evaluation only if it were supplemented with other readings, particularly those that focus on other equally important approaches to evaluation. Additionally, while the organization is tight and focused, thereby guiding the reader through complex materials, the organization of the chapters and the transitions between chapters suggest that program evaluation is a linear process, presenting it as somewhat simplistic and task-oriented. Also, although this book is strong on the “how-to” for putting together an experimental evaluation study, it is weak on the “how-to” for qualitative/participatory evaluation and the “whyto” for all types of evaluation. There is no discussion of the meaning of program evaluation. There is no discussion about the role of the evaluator. Also, there
Transforming
275
is no discussion of the social implications or responsibilities of program evaluation, and no sense of the evaluation taking place through human interactions for the purpose of enhancing our quality of life and contributing to social betterment. Therefore, this book would not be appropriate for a class that focuses on understanding theories, concepts and philosophy of program evaluation. Finally, although the book is comprehensive in the terms of covering the experimental perspective of evaluation, it seems the book is more about design and methods than about evaluation, per se. Particularly when the book deals with sampling design and statistical analysis (for example which statistic to use), there is no clear sense of adapting these tools specifically to program evaluation applications. Other than the first chapter and the examples, the book does not seem to be, specifically, about evaluation-it could be about doing experimental research in educational and human service contexts. “The book is designed to cover the field and respect the needs of students in education and psychology, as well as those of teachers, counsellors, experts in curriculum and instruction, testing specialists, and anyone else interested in understanding and improving teaching and learning”, (p, xi). In fact, the text does not cover the field, but does a pretty good job of presenting one perspective in the field of program evaluation, and will appeal to individuals who share this perspective. The book is certainly introductory level material, but most likely it would have to be supplemented with other readings to be adoptable for courses. The book is a nice desk reference on an experimental approach to program evaluation.
Qualitative Data: Description, Analysis and Interpretation, by H. F. Wolcott, Thousand
Oaks, CA:
Sage, 1994,433 pp. Reviewer:
Linda Mabry
Most if not all practicing ethnographers and teachers of research or evaluation methodology would concede that the toughest part of qualitative work is to describe data analysis for students, for a proposal, or for the methods section of a report. Often it is harder to describe qualitative data analysis than it is to do it. How do we find meaning in interview transcripts, observation notes, or the minutes of meetings? What is the nature, the ontology, the process of insight? As a friend who will “tell it like it is”, Harry Wolcott strides into this thorniest thicket of qualitative work,
the “black box” of what really happens between data collection (or “fieldwork”, his preferred term) and completed research report. In Transforming Qualitative Data: Description,
Analysis,
and Interpretation
(I 994),
he eschews the scientistic, step-by-step accounts found in some textbooks and articles of such procedures as categorization as a precursor to identification of trends, or development of typologies as a phase in theory generation, or enumeration of coded data to refine or verify hypotheses. Despite their perfunctory admonitions to the contrary, advocates of more-or-less specific actions to be taken at more-or-less specific points in a post-data collection timetable strongly suggest that qualitative
276
Book Reviews
data analysis is a sequential macro-activity specifiable in detail in advance. But does this help us understand or evaluate the phenomena we study or the perceptions and meanings of those who operate within them, or merely help us respond to our obligations to describe the ineffable process of meaning-making? Wolcott offers an alternative, while understanding the impetus for magniloquent formulae: Because qualitative data gathering is conducted through such everyday techniques as participant observation and interviewing, it is comforting to employ a term like “analysis” to suggest that in what we do with the data we are able to wrest them from their humble origins and transform them into something grand enough to pass for science (pp. 24-25).
How grand is qualitative data analysis? Consider his distilled advice to doctoral students: “Tell the story. Then tell how that happened to be the way you told it” (p. 16). Wolcott concedes the things we study are dissimilar, as are the kinds and depths of understandings we reach, as are we researchers ourselves. Exactly what we do in the effort to apprehend the fundamental nature of these things necessarily varies. Adaptation of methods to phenomena, to the data available, to researcher preference and experience, and more preclude detailed advance specification of interpretive processes. How our minds work in developing our understandings may not be describable except in broad strokes. Wolcott offers a general outline and a cornucopia of advice. To the triumvirate procedures of qualitative data gathering-observation, interview, and documents review-Wolcott adds three similarly broad and flexible categories to describe the process of data “transformation”: description, analysis, and interpretation. (Wolcott prefers the term “transformation” to “analysis” as the overarching term for the entire process because he stipulates a more constrained definition for analysis as one of the three processes within transformation. This stipulation, which differs from widespread current usage, unfortunately invites confusion.) These three processes may be used exclusively or in combination. Transformation of data by description produces a report characterized largely by data presentation with relatively little (or perhaps relatively implicit) interpretation. Separate from or building upon description, transformation may involve analysis, “some careful, systematic way to identify key factors and relationships among them” (p. 10). From either or both description and analysis, or separate from them, data may be interpreted “to reach out for understanding or explanation beyond the limits of what can be explained with the degree of certainty usually associated with analysis” (pp. 10-l 1). Wolcott properly notes that distinctions about the nature of these three processes are decidedly blurred and that, when using all three, they will undoubtedly overlap in a rough-and-tumble dia-
lectic rather than progress in stately and predictable linearity. Another variability: qualitative data analysis is judgment-intensive and, consequently, idiosyncratic. Early on, Wolcott warns his readers of the absence of definitive techniques or professional secrets: Do qualitative researchers have some special way of describing things and rendering those descriptions in final accounts? Not that I know of.. (p. 14) Every decision about the appropriate level of detail returns ultimately to the immediate purposes being addressed and to over-riding considerations that map a course between extremes of too-selective reporting or hopeless obfuscation. (p. 14) If I could provide the magic formula for recognizing, remembering, recording, and subsequently transforming relevant data [into descriptive accounts], here is where I would reveal it. But the only advice I offer is to make judgments in terms of sufficiency and to screen all description in those terms, the data to be excluded as well as the data to be included. (emphasis in the original, pp. 14-15) Just as some researchers prefer to build their cases essentially through the descriptive account, others appear obviously impatient to get beyond “mere description” . ‘ (p. 15).
Some may consider Wolcott’s move away from recipe-like delineation of process a retreat into chaos. What, then, are we to tell our students and potential funders and clients about how we deal with qualitative data? For the evaluator, as for the researcher, there is benefit from Wolcott’s unsettling admissions that valid interpretation is not guaranteed by certain actions, and that faithful adherence to particular procedures does not ensure trustworthy findings. Rather, we must strive for empirically well-warranted and logically or theoretically coherent renderings. Instead of issuing directives, Wolcott shares suggestions gleaned from his professional life: ten ways one might tell a story or narrative, eleven ways to approach interpretation, four strategies for ethnographic observation, and more. Even where Wolcott enumerates, he is quick to announce that his lists are non-exhaustive. Following didactic sections are examples of his own work, nine completed studies which have appeared earlier in some published form, including such longstanding favorites as “Sneaky Kid”, “The Man in the Principal’s Office”, and “The Teacher as an Enemy”. While there is little explicit attention directed toward the analysis of qualitative data for the purposes of program evaluation in this volume’, there is much that is useful to program evaluators and to students of program evaluation, partly because the field has increasingly utilized qualitative methods. The volume would
‘See pp. 179-180. ‘See, for example, LeCompte, M. D. & Preissle, J. (1993). Ethnography and qualitative design in educational research (2nd ed.), NY: Academic Press; and Miles, M. B. & Huberman, A. M. (1984). Qualitative data analysis: A sourcebook ofnew methods. Beverly Hills, CA: Sage Publications.
Book Reviews have greater appeal to this audience if it included an account or two which comprise or supplement evaluation reports, but many practicing evaluators have access to such documents-and Wolcott is not an evaluator by profession. Perhaps particularly in evaluation where market and political forces intrude into inquiry
277
and demand slick credibility, advocates of a more delineated approach to qualitative data analysis have exerted strong appea12. Wolcott offers an important counterpoint, one which would make a fine supplementary reading to a course on evaluation methods.