- Email: [email protected]
Toward the justification of claims in evaluation research
Evaluation and Program Planning, Vol. 10, pp. 309-314, 1987 Printed in the USA. All rights reserved.
0149-7189/87 $3.00 + .00 Copyright © 1987 Pergamon Journals Ltd
T O W A R D THE JUSTIFICATION OF CLAIMS IN E V A L U A T I O N R E S E A R C H
NICK L. SMITH Syracuse University
ABSTRACT Four types o f claims generally arise in evaluation research: research claims, policy claims, evaluation claims, and management claims. The relationships among these types o f claims can be seen both in the similarity o f the questions under the types, and in their interrelatedness in making evaluation recommendations. By construing these claims as question-answer propositions, it is possible to identify the specific kinds o f questions associated with each o f the four types o f claims. Under this view, evaluation research is predominantly a form o f applied inquiry, and the justification o f evaluation claims becomes the validation o f questionanswer propositions.
The justification of claims in evaluation research is a difficult issue to examine in part because there are several different types of claims commonly made in evaluation practice: there are research claims, policy claims, evaluation claims, and management claims. As a first step in understanding better how one might justify claims in the practice of evaluation, then, I attempt in the following discussion to clarify further these types of claims and thereby provide a conceptual structure for considering their justification.
Clarification of and discrimination between the various types of questions which evaluations undertake to answer is absolutely fundamental to getting a useful answer at all . . . (Scriven, 1980, p. 45) The practice of evaluation research can be characterized as the making of justifiable claims concerning the object of study. The discussion which follows is based on the premise that these claims are essentially answers to questions of interest. That is, the process of evaluation research is fundamentally a process of answering q u e s t i o n s - i n q u i r y is the core of evaluation research.
CLAIMS AS QUESTION-ANSWER PROPOSITIONS The summary of evaluation research claims presented here builds on Dillon's (1984a) suggestion that the evaluation process can be conceived of as the development of question-answer propositions. A claim in evaluation research is therefore an assertion that a particular question-answer proposition is both meaningful and valid. Many difficulties and inadequacies of performance arise in evaluation due to insufficient attention to this question-answer connection: evaluators and clients pose unanswerable questions, the questions posed are not
meaningful or important, the questions posed are never answered, or the answers provided do not address the questions posed (as when the answers at the end of the report do not relate to the questions stated in the abstract). Dillon (1984a) recommends four procedures for improving evaluation practice in this regard. First, before identifying the question to investigate, classify the questions that can be asked. The nature of the answer depends on the nature of the question. To get a different k i n d of answer, one must ask a different k i n d of question. Classification of possible questions is a use-
The analysis presented here draws heavilyon what 1 consider to be several significant, but largelyoverlooked, contributions to our understanding of questions and answers in evaluation research. I am especially indebted to the prior work of Jim Dillon, Bob Gowin, and Tom Green. Any misinterpretations or other injustices done here to their original contributions remain, of course, solelymy responsibility. Requests for reprints should be sent to Nick L. Smith, School of Education, Syracuse University, Syracuse, New York, 13244-2340. 309
310
NICK L. SMITH
ful way to find and formulate the most significant questions for the evaluative study. Second, before posing the question, analyze it by examining the presuppositions of the question and the presumptions of the questioner. This step prevents one from arriving at untrue answers to invalid questions, and from seeking an answer to a question not truly offered. Third, before addressing the question, construct a dummy answer and submit it for review by the client and audiences to identify which kinds of answers will be considered unwelcome, irrelevant, interesting, or influential. Finally, before stating the answer, state the question. Just as a question may have many answers, an answer may fit many questions. The answers we arrive at have meaning only within a given question-answer proposition.
If we conceive of claims in evaluation research as question-answer propositions, then perhaps, as Dillon (1984a) suggests, we should proceed by classifying types of questions. I will depart from the strict erotetic logic of question formation, however, by not classifying all questions that could be asked in evaluation research, but instead focusing only on those types of questions that seem to be commonly asked in evaluation practice. I do this because my intent is to consider the justification of evaluation claims commonly made, rather than all possible evaluation claims. The discussion which follows does not attempt to treat exhaustively all types of questions, but only those most frequently arising in day to day practice.
TYPES OF E V A L U A T I O N R E S E A R C H C L A I M S Categories of Claims There are four categories of claims which include the majority of assertions made in the practice of evaluation research: research claims, policy claims, evaluation claims, and management claims (cf. Smith, 1980). Research and policy claims are concerned with classes of events, while evaluation and management claims are concerned with specific events. Research deals with fact claims, policy deals with general value claims, evaluation deals with specific value attributions or claims, and management deals with action claims. The general characteristics of the questions and answers under each of these four types of claims are summarized below in Table 1. The kinds of questions frequently found in the question-answer propositions under each of these four
TABLE 1 CATEGORIES OF CLAIMS IN EVALUATION RESEARCH Primary Questions Research What is the state of affairs generally? Why is the state of affairs the way it is? (explanation)
Form of Answers
Nature of Answers
X is true
Fact claim.
X is true because of Y.
Explanation.
Policy What is the desirable state of affairs?
X's are good.
General value claim.
Evaluation Is this specific event desirable?
This X is good.
Specific value attribution.
Do X here.
Action claim.
Management What needs to be done in this situation to achieve the desirable state of affairs?
Note. Adapted from Smith (1980).
types of claims are addressed in greater detail in the sections which follow. I have previously suggested a range of the types of causal and noncausal questions that should be addressed to understand fully the nature of an educational program (Smith, 1981). Those questions are reproduced below in Table 2. In this revised display, I have added a more generalized form of each question and have resequenced them in order to illustrate their coverage of the four types of evaluation research claims. In a subsequent investigation (Dillon, 1984b), it was estimated that the classification of types of questions in Table 2 accounted for 80°70 of the empirical questions and 70°7o of the empirical and theoretical questions in a collection of 924 research questions found in a range of nine education journals. This finding suggests that the sample questions in Table 2 cover a majority of the types of questions commonly posed in educational research. I would expect the percent of coverage to be even higher if the literature reviewed were restricted to the narrower field of evaluation research within education. Research Claims Research claims, and therefore research questions, are a central part of evaluation research. Indeed, if there is a difference between the use of the terms "evaluation" and "evaluation research" to describe the work of professional evaluators, it is that the latter label indicates a greater emphasis on the role of inquiry in the evaluation process. Some theorists, for example Rossi and Freeman (1985), emphasize the research aspects so heavily that the evaluative components of evaluation research are dramatically overshadowed. Other approaches, for example Eisner's (1983), highlight the interpretive and judgmental aspects of evaluation research with greatly diminished attention to descriptive or analytic inquiry. All commonly employed forms of
Justification TABLE 2 SAMPLE QUESTIONS FOR EVALUATION RESEARCH
Project: Program for providing special education children with reading instruction by using CAI materials located in remedial centers. Sample Questions Causal--Research Do children read better as a result of this program? Do children read better in this program as compared with other programs? NoncausaI--Research What is the daily experience of the children participating in this program? Are the remedial centers located in the areas of primary need? NoncausaI--Policy What do we mean by "special education children" and "remediation"? Is this program receiving support from state and local officials for political rather than educational reasons? Noncausal-- Evaluation What are the characteristics of the best CAI materials being used? How do the various minority groups view this program and judge its quality? NoncausaI--Management What is the cost-effectiveness of the program compared with other programs? How can we maximize the scheduling of classes at the centers with minimum expense?
General Form
Does X cause Y? Does X cause more of Y than Z causes of Y? What is X? Is X located where Y is lowest? What does " Y " mean? Why does S support X?
What makes W good? Does T value X?
311 TABLE 3
A CLASSIFICATION OF RESEARCH QUESTIONS Category of Question Zero order O. Rhetorical First order: Properties 1. Existence/affirmation-negation 2. Instance/identification 3. Substance/definition 4. Character/description 5. Function/application 6. Rationale/explication Second order: Comparisons 7. Concomitance 8. Equivalence 9. Difference Third order: Contingencies 10. Relation 11. Correlation 1 2. Conditionality 1 3. Biconditionality Extra order: Other 14. Deliberation 1 5. Unspecified 1 6. Unclear
Note. From "The Classification of Research Questions" by J.T. Dillon, 1984, Review of Educational Research, 54, p. 330, copyright 1984 by American Educational Research Association. Reprinted by permission.
Is X more costeffective than Z? How are U maximized and V minimized simultaneously?
Note. Adapted from Smith (1 981 ).
evaluation or evaluation research, however, make some research claims, to a greater or lesser extent. Only a few categorizations of research questions are available. Dillon (1984b) reviewed 12 such categorization schemes, including several which not only classified research questions, but identified research methods associated with each. Dillon provided an integrated classification of 17 categories of research questions designed to encompass all scholarly inquiry. His system classified questions by the logic of their propositions and arranged the questions by priority and increment of knowledge. The 17 major categories of his system (there are also 12 subcategories) are listed in Table 3. This broader, more comprehensive classification system accounted for 99°-/o and 90°7o respectively of the empirical and empirical and theoretical literature of the nine education journals mentioned above. If Dillon's system does indeed encompass all scholarly inquiry, then each
of the categories of questions under all four of the types of evaluation research claims discussed here may fit within his single system. If that is the case, then his classification system is the more powerful tool for identifying all logically possible question-answer propositions for evaluation research and thereby perhaps expanding the range of inquiry in the evaluation profession. The focus of this discussion, however, is on those types of questions that seem to have arisen naturally in various aspects of evaluation research practice. The incidence of these more cc,~amonplace types of questions could be verified empirically and justification of the resultant claims could be strengthened to improve professional practice. The justification of questionanswer propositions that seldom arise offers little for the improvement of practice. The categories of questions suggested below for each of the four types of evaluation research claims (research, policy, evaluation, and management) are intended only as preliminary lists to be used as a basis for empirical study of the actual questions raised in these four aspects of evaluation research practice. Studies of practice are needed to identify the most consistently raised questions and the resultant claims with the weakest justifications. A preliminary list of the types of research questions often arising in evaluation research
312
NICK L. SMITH
is presented below in Table 4. The seven suggested questions are categorized in three groups: definitional, descriptive, and instrumental. Policy Claims It was suggested above that the primary policy question is "Which P's are good?" As with the other categories, this is, of course, a gross oversimplification, useful as a general heuristic, but not reflective of the complexity of the questions raised in policy problems. Further, the display in Table 1 does not illustrate the ways in which the types of questions raised may overlap the categories. Green defined a policy question as " . . . a request for a line of action aimed at securing an optimal resolution of a conflict between different goods, all of which must be accepted, but which, taken together, cannot all be maximized" (1982, p. 144). More formally, this might be stated as "What action P will simultaneously optimize X, Y, and Z, given that X, Y, and Z are all individually valued but cannot be simultaneously maximized?" In other words, what P's are, in general, good to do; or, what should one do? Green argued that the need for policy claims arises because of scarcity of goods and the fact that, in principle, some goods cannot be simultaneously maximized (e.g., equality and liberty). Policy questions are practical, not theoretical, questions whose answers are practical action claims. Policy questions are thus like management questions in that they involve action claims, but at a broader level of application; they are like evaluation questions in that they involve value claims, but again at a more general level. Finally, policy questions are related to research questions in that the selection of proper action involves the assessment of the anticipated consequences of possible actions (a research claim). A summary of the major types of policy questions arising in evaluation research is presented in Table 5 below. Evaluation Claims The making of value claims is the defining characteristic that differentiates evaluation research from other
TABLE 4 RESEARCH QUESTIONS ARISING IN EVALUATION RESEARCH Definitional 1. What is "P"? 2. Does " P " exist? Descriptive 3. Is this P an instance of "P"? 4. What constitutes this P? Instrumental 5. What does P do? 6. Does P cause Y? 7. How does P cause Y?
TABLE 5 POLICY QUESTIONS ARISING IN EVALUATION RESEARCH 1. What X, Y, and Z are in conflict? 2. What P is the best possible optimization of X, Y, and Z? a. What are the possible consequences of P? b. What are the marginal effects of optimization P ever optimization Q? 3. How do we reach P? a. Who will support P? (political analysis)
forms of applied social research. In spite of this, most professional attention has focused on the inquiry aspects of evaluation rather than on the value aspects. In this instance, evaluation theory and practice have been especially hampered by the general unwillingness of the social sciences to deal with issues of value. To some extent this is changing. As Novak and Gowin noted (1984, p. 174): "Classical empiricist and positivist philosophies denied that value questions and value claims had a legitimate place in a rational enterprise of knowledge construction. Contemporary philosophies, on the other hand, see value questions as crucial to the advance of our understanding in any discipline." Scriven (1980) has provided a lengthy discussion of types of questions which an evaluator may be called upon to answer, including pre-evaluative, evaluative, and quasi-evaluative questions. A summary listing of his 17 questions and subquestions is contained below in Table 6.
TABLE 6 EVALUATION QUESTIONS ARISING IN EVALUATION RESEARCH A. Pre-Evaluative Questions 1. What is X? 2. Is X valued? a. X valued by whom? b. Is X wanted, preferred, valenced? c. Is X believed to be valuable? B. Evaluative Questions 1. Is X good? 2. Is X better than Y? 3. How good is X? a. How much better is X than Y? 4. How much is X worth? a. Is X worth more than Y, and why? 5. What mixture of X and Y is worth most? 6. In what respects is X good? 7. What parts or components of X are good? C. Quasi-Evaluative Questions 1. What makes X good? 2. What would make X better? 3. Will X improve? Note. From The Logic of Evaluation (pp. 4 6 - 8 9 ) by M. Scriven, 1980, Pt. Reyes, CA: Edge press. Copyright 1980 by M. Scriven. Reprinted by permission.
Justification Gowin (in Gowin and Green, 1980, and Novak and Gowin, 1984) also has argued that value claims are answers to value questions, and that the various types of value questions are related to knowledge questions and to each other. He suggested five types of value questions (summarized below in Table 7), including intrinsic, instrumental, comparative, decision, and idealization. Again note the overlap among these questions and those in the other three categories; for example, the similarity with the comparative question in the research category, and the decision claim with a like question in the management category.
Management Claims Because evaluation in the United States is performed most frequently as a service for managers and administrators, evaluation researchers must often deal with management questions. Even though management claims are not directly a part of the inquiry or valuing aspects of evaluation research, they represent the ultimate ends to which these other claims are usually directed. Management claims are pragmatic and applied; they are action claims. The primary question is of the form "Should X be done here?" Local conditions and events make all the difference in answering management questions. The answers are specific to the local context in that with any given combination of research, policy, and evaluation claims, the answers to the management questions will still differ depending on local factors. The prevalent types of management questions are summarized below in Table 8. The relationships between
TABLE 7 M O R E EVALUATION QUESTIONS ARISING IN EVALUATION R E S E A R C H 1. Is X good? What is the good of X? (Intrinsic value claim) 2. Is X good for Z? (Instrumental value claim) 3. Is X better than Y? (Comparative value claim) 4. Is X right? Ought we to choose X? (Decision claim) 5. Is the good of X something that can be made better? (Idealization claim) Note. From The Evaluation Document: Philosophic Structure (p. 13) by D.B. Gowin and T. Green, 1980, Portland, OR: Northwest Regional Educational Laboratory, ROEP Rep. No. 38. Copyright 1980 by Northwest Regional Educational Laboratory. Reprinted by permission.
JUSTIFICATION This discussion so far has left the issue of the justification of claims largely unaddressed. This was because
313 TABLE 8 MANAGEMENT QUESTIONS ARISING IN EVALUATION RESEARCH 1. 2. 3. 4.
Should X be done here? Can X be done here? What are the costs of doing X here? What are the benefits of doing X here?
management claims and the prior three types are demonstrated in the following section.
Recommendations in Evaluation Research It is in the activity of making recommendations in evaluation research that we best see how the four types of claims may interrelate. Consider the kinds of claims used in the following example to support the recommendation that additional teacher aides should be assigned to a remedial reading program which serves students transferring from small rural schools to a slightly academically stronger suburban junior high school. 1. The suburban district accepts the responsibility to prepare adequately its students for entry into high school. (policy claim) 2. The current rural transfer students, after one year in the remedial program, are scoring at the 4Jth percentile on the district standardized reading test. (research claim) 3. This performance shows improvement over the students' entry level of the 37th percentile, but is unacceptably low for probable success in high school, based on the current high school entry average of the 63rd percentile. (evaluation claim) 4. Increases in individual attention by teacher aides and the resultant increased time on task have been shown to increase reading performance. (research claim) 5. We should increase the number of teacher aides in our remedial reading program, and we have the resources and opportunity to do so this next year. (management claim) This is an oversimplified example, but it illustrates the interconnections of research, policy, evaluation, and management claims in evaluation research. The interconnections are also suggested by the various questions which overlap two or more categories. Perhaps a single classification scheme (an adaptation of Table 3?) could be constructed to formally integrate the several question schemes described above (several of them do not represent formal classifications, only lists). OF CLAIMS of the need to first answer the prior question of what are the types of claims (question-answer propositions)
314
N I C K L. S M I T H
whose justification is of interest. One could argue simply that establishing the answer to the question constitutes the justification for the claim. The methods of justification then would be the question answering methods, whether they be logical analysis, theoretical research, or empirical research. While perhaps a reasonable starting point, this argument seems an oversimplification on both practical and methodological grounds. In practice, evaluation research claims are not made as isolated propositions, but as part of a larger interconnected argument, as in the example above. The ultimate justification for a particular proposition, therefore, depends in part on the strengths and weaknesses of the total argument within which it is embedded. Further, the meaningfulness of a particular claim also depends on the nature of the methodological system used to address it. Since different question answering methods embody different ontological and epistemological assumptions, and since there are different methods available for answering a single question, the justification of a particular question-answer proposition must necessarily include the justification of the method used to answer it. There are, of course, other ways to conceive of evaluation research than as inquiry. If we do not base our conception of evaluation research on the premise that it is a form of inquiry involving the development of question-answer propositions, then claims may be justified through non-inquiry means. For example, perhaps policy claims are justified through persuasion, negotiation, power, and influence; evaluation claims
are justified through recognized connoisseurship, and compelling criticism; and management claims are justified through systems development, historical precedent, and experienced intuition. All these forms of justification are perhaps better construed as something other than question-answer propositions. I believe that the justification of any of these viewpoints rests, however, to some extent on their usefulness in clarifying our thinking about evaluation research (evaluation theory), and in improving our practice. While certainly not the only conception, the view of evaluation research as the development of question-answer propositions does suggest a useful categorization of claims within which to consider alternative forms of justification in evaluation research. Guba (1987) has recently identified some of the diverse forms of naturalistic evaluation currently popular. Fetterman (1986) has argued that these various forms all have their own sets of standards by which the quality of their work should be judged. Are there many acceptable ways of justifying conclusions in naturalistic evaluation, or only one fundamental way? Can all forms of naturalistic evaluation be usefully construed as alternative procedures for the development of question-answer propositions? If so, then the justification of claims may center on the validity of the questionanswer propositions each form of naturalistic evaluation constructs. In reading the following four articles of this special issue, consider the types of claims being made and the methods advocated for their justification.
REFERENCES DILLON, J.T. (1984a). Finding the question f o r evaluation research (ROEP Rep. No. 102). Portland, OR: Northwest Regional Educational Laboratory.
GREEN, T.F. (1982). Policy and evaluation: A conceptual study. In N.L. Smith & D. N. Caulley (Eds.), The interaction o f evaluation and policy. Portland, OR: Northwest Regional Educational Laboratory.
DILLON, J.T. (1984b). The classification of research questions. Review o f Educational Research, 54, 327-361.
GUBA, E.G. (1987). What have we learned about naturalistic evaluation? Evaluation Practice, 8, 23-43.
EISNER, E.W. (1983). Educational connoisseurship and criticism: Their form and functions in educational evaluation. In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (Eds.), Evaluation models. Boston: Kluwer-Nijhoff.
FETTERMAN, D.M. (1986). Conceptual crossroads: Methods and ethics in ethnographic evaluation. In D. D. Williams (Ed.), Naturalistic evaluation: Vol. 30. New directions in program evaluation, San Francisco: Jossey-Bass. GOWIN, D.B., & GREEN, T. (1980). The evaluation document: Philosophic structure (ROEP Rep. No. 38). Portland, OR: Northwest Regional Educational Laboratory.
NOVAK, J.D., & GOWIN, D.B. (1984). Learning how to learn. New York: Cambridge University Press. ROSSI, P. H., & FREEMAN, H.E. (1985). Evaluation: A systematic approach. Beverly Hills, CA: Sage Publications. SCRIVEN, M. (1980). The logic o f evaluation. Pt. Reyes, CA: Edge press. SMITH, N.L. (1980). Federal research on evaluation methods in health, criminal justice, and the military. Educational Evaluation and Policy Analysis, 2, 53-59. SMITH, N.L. (1981). Noncausal inquiry in education. Educational Researcher, 10, 23.
0149-7189/87 $3.00 + .00 Copyright © 1987 Pergamon Journals Ltd
T O W A R D THE JUSTIFICATION OF CLAIMS IN E V A L U A T I O N R E S E A R C H
NICK L. SMITH Syracuse University
ABSTRACT Four types o f claims generally arise in evaluation research: research claims, policy claims, evaluation claims, and management claims. The relationships among these types o f claims can be seen both in the similarity o f the questions under the types, and in their interrelatedness in making evaluation recommendations. By construing these claims as question-answer propositions, it is possible to identify the specific kinds o f questions associated with each o f the four types o f claims. Under this view, evaluation research is predominantly a form o f applied inquiry, and the justification o f evaluation claims becomes the validation o f questionanswer propositions.
The justification of claims in evaluation research is a difficult issue to examine in part because there are several different types of claims commonly made in evaluation practice: there are research claims, policy claims, evaluation claims, and management claims. As a first step in understanding better how one might justify claims in the practice of evaluation, then, I attempt in the following discussion to clarify further these types of claims and thereby provide a conceptual structure for considering their justification.
Clarification of and discrimination between the various types of questions which evaluations undertake to answer is absolutely fundamental to getting a useful answer at all . . . (Scriven, 1980, p. 45) The practice of evaluation research can be characterized as the making of justifiable claims concerning the object of study. The discussion which follows is based on the premise that these claims are essentially answers to questions of interest. That is, the process of evaluation research is fundamentally a process of answering q u e s t i o n s - i n q u i r y is the core of evaluation research.
CLAIMS AS QUESTION-ANSWER PROPOSITIONS The summary of evaluation research claims presented here builds on Dillon's (1984a) suggestion that the evaluation process can be conceived of as the development of question-answer propositions. A claim in evaluation research is therefore an assertion that a particular question-answer proposition is both meaningful and valid. Many difficulties and inadequacies of performance arise in evaluation due to insufficient attention to this question-answer connection: evaluators and clients pose unanswerable questions, the questions posed are not
meaningful or important, the questions posed are never answered, or the answers provided do not address the questions posed (as when the answers at the end of the report do not relate to the questions stated in the abstract). Dillon (1984a) recommends four procedures for improving evaluation practice in this regard. First, before identifying the question to investigate, classify the questions that can be asked. The nature of the answer depends on the nature of the question. To get a different k i n d of answer, one must ask a different k i n d of question. Classification of possible questions is a use-
The analysis presented here draws heavilyon what 1 consider to be several significant, but largelyoverlooked, contributions to our understanding of questions and answers in evaluation research. I am especially indebted to the prior work of Jim Dillon, Bob Gowin, and Tom Green. Any misinterpretations or other injustices done here to their original contributions remain, of course, solelymy responsibility. Requests for reprints should be sent to Nick L. Smith, School of Education, Syracuse University, Syracuse, New York, 13244-2340. 309
310
NICK L. SMITH
ful way to find and formulate the most significant questions for the evaluative study. Second, before posing the question, analyze it by examining the presuppositions of the question and the presumptions of the questioner. This step prevents one from arriving at untrue answers to invalid questions, and from seeking an answer to a question not truly offered. Third, before addressing the question, construct a dummy answer and submit it for review by the client and audiences to identify which kinds of answers will be considered unwelcome, irrelevant, interesting, or influential. Finally, before stating the answer, state the question. Just as a question may have many answers, an answer may fit many questions. The answers we arrive at have meaning only within a given question-answer proposition.
If we conceive of claims in evaluation research as question-answer propositions, then perhaps, as Dillon (1984a) suggests, we should proceed by classifying types of questions. I will depart from the strict erotetic logic of question formation, however, by not classifying all questions that could be asked in evaluation research, but instead focusing only on those types of questions that seem to be commonly asked in evaluation practice. I do this because my intent is to consider the justification of evaluation claims commonly made, rather than all possible evaluation claims. The discussion which follows does not attempt to treat exhaustively all types of questions, but only those most frequently arising in day to day practice.
TYPES OF E V A L U A T I O N R E S E A R C H C L A I M S Categories of Claims There are four categories of claims which include the majority of assertions made in the practice of evaluation research: research claims, policy claims, evaluation claims, and management claims (cf. Smith, 1980). Research and policy claims are concerned with classes of events, while evaluation and management claims are concerned with specific events. Research deals with fact claims, policy deals with general value claims, evaluation deals with specific value attributions or claims, and management deals with action claims. The general characteristics of the questions and answers under each of these four types of claims are summarized below in Table 1. The kinds of questions frequently found in the question-answer propositions under each of these four
TABLE 1 CATEGORIES OF CLAIMS IN EVALUATION RESEARCH Primary Questions Research What is the state of affairs generally? Why is the state of affairs the way it is? (explanation)
Form of Answers
Nature of Answers
X is true
Fact claim.
X is true because of Y.
Explanation.
Policy What is the desirable state of affairs?
X's are good.
General value claim.
Evaluation Is this specific event desirable?
This X is good.
Specific value attribution.
Do X here.
Action claim.
Management What needs to be done in this situation to achieve the desirable state of affairs?
Note. Adapted from Smith (1980).
types of claims are addressed in greater detail in the sections which follow. I have previously suggested a range of the types of causal and noncausal questions that should be addressed to understand fully the nature of an educational program (Smith, 1981). Those questions are reproduced below in Table 2. In this revised display, I have added a more generalized form of each question and have resequenced them in order to illustrate their coverage of the four types of evaluation research claims. In a subsequent investigation (Dillon, 1984b), it was estimated that the classification of types of questions in Table 2 accounted for 80°70 of the empirical questions and 70°7o of the empirical and theoretical questions in a collection of 924 research questions found in a range of nine education journals. This finding suggests that the sample questions in Table 2 cover a majority of the types of questions commonly posed in educational research. I would expect the percent of coverage to be even higher if the literature reviewed were restricted to the narrower field of evaluation research within education. Research Claims Research claims, and therefore research questions, are a central part of evaluation research. Indeed, if there is a difference between the use of the terms "evaluation" and "evaluation research" to describe the work of professional evaluators, it is that the latter label indicates a greater emphasis on the role of inquiry in the evaluation process. Some theorists, for example Rossi and Freeman (1985), emphasize the research aspects so heavily that the evaluative components of evaluation research are dramatically overshadowed. Other approaches, for example Eisner's (1983), highlight the interpretive and judgmental aspects of evaluation research with greatly diminished attention to descriptive or analytic inquiry. All commonly employed forms of
Justification TABLE 2 SAMPLE QUESTIONS FOR EVALUATION RESEARCH
Project: Program for providing special education children with reading instruction by using CAI materials located in remedial centers. Sample Questions Causal--Research Do children read better as a result of this program? Do children read better in this program as compared with other programs? NoncausaI--Research What is the daily experience of the children participating in this program? Are the remedial centers located in the areas of primary need? NoncausaI--Policy What do we mean by "special education children" and "remediation"? Is this program receiving support from state and local officials for political rather than educational reasons? Noncausal-- Evaluation What are the characteristics of the best CAI materials being used? How do the various minority groups view this program and judge its quality? NoncausaI--Management What is the cost-effectiveness of the program compared with other programs? How can we maximize the scheduling of classes at the centers with minimum expense?
General Form
Does X cause Y? Does X cause more of Y than Z causes of Y? What is X? Is X located where Y is lowest? What does " Y " mean? Why does S support X?
What makes W good? Does T value X?
311 TABLE 3
A CLASSIFICATION OF RESEARCH QUESTIONS Category of Question Zero order O. Rhetorical First order: Properties 1. Existence/affirmation-negation 2. Instance/identification 3. Substance/definition 4. Character/description 5. Function/application 6. Rationale/explication Second order: Comparisons 7. Concomitance 8. Equivalence 9. Difference Third order: Contingencies 10. Relation 11. Correlation 1 2. Conditionality 1 3. Biconditionality Extra order: Other 14. Deliberation 1 5. Unspecified 1 6. Unclear
Note. From "The Classification of Research Questions" by J.T. Dillon, 1984, Review of Educational Research, 54, p. 330, copyright 1984 by American Educational Research Association. Reprinted by permission.
Is X more costeffective than Z? How are U maximized and V minimized simultaneously?
Note. Adapted from Smith (1 981 ).
evaluation or evaluation research, however, make some research claims, to a greater or lesser extent. Only a few categorizations of research questions are available. Dillon (1984b) reviewed 12 such categorization schemes, including several which not only classified research questions, but identified research methods associated with each. Dillon provided an integrated classification of 17 categories of research questions designed to encompass all scholarly inquiry. His system classified questions by the logic of their propositions and arranged the questions by priority and increment of knowledge. The 17 major categories of his system (there are also 12 subcategories) are listed in Table 3. This broader, more comprehensive classification system accounted for 99°-/o and 90°7o respectively of the empirical and empirical and theoretical literature of the nine education journals mentioned above. If Dillon's system does indeed encompass all scholarly inquiry, then each
of the categories of questions under all four of the types of evaluation research claims discussed here may fit within his single system. If that is the case, then his classification system is the more powerful tool for identifying all logically possible question-answer propositions for evaluation research and thereby perhaps expanding the range of inquiry in the evaluation profession. The focus of this discussion, however, is on those types of questions that seem to have arisen naturally in various aspects of evaluation research practice. The incidence of these more cc,~amonplace types of questions could be verified empirically and justification of the resultant claims could be strengthened to improve professional practice. The justification of questionanswer propositions that seldom arise offers little for the improvement of practice. The categories of questions suggested below for each of the four types of evaluation research claims (research, policy, evaluation, and management) are intended only as preliminary lists to be used as a basis for empirical study of the actual questions raised in these four aspects of evaluation research practice. Studies of practice are needed to identify the most consistently raised questions and the resultant claims with the weakest justifications. A preliminary list of the types of research questions often arising in evaluation research
312
NICK L. SMITH
is presented below in Table 4. The seven suggested questions are categorized in three groups: definitional, descriptive, and instrumental. Policy Claims It was suggested above that the primary policy question is "Which P's are good?" As with the other categories, this is, of course, a gross oversimplification, useful as a general heuristic, but not reflective of the complexity of the questions raised in policy problems. Further, the display in Table 1 does not illustrate the ways in which the types of questions raised may overlap the categories. Green defined a policy question as " . . . a request for a line of action aimed at securing an optimal resolution of a conflict between different goods, all of which must be accepted, but which, taken together, cannot all be maximized" (1982, p. 144). More formally, this might be stated as "What action P will simultaneously optimize X, Y, and Z, given that X, Y, and Z are all individually valued but cannot be simultaneously maximized?" In other words, what P's are, in general, good to do; or, what should one do? Green argued that the need for policy claims arises because of scarcity of goods and the fact that, in principle, some goods cannot be simultaneously maximized (e.g., equality and liberty). Policy questions are practical, not theoretical, questions whose answers are practical action claims. Policy questions are thus like management questions in that they involve action claims, but at a broader level of application; they are like evaluation questions in that they involve value claims, but again at a more general level. Finally, policy questions are related to research questions in that the selection of proper action involves the assessment of the anticipated consequences of possible actions (a research claim). A summary of the major types of policy questions arising in evaluation research is presented in Table 5 below. Evaluation Claims The making of value claims is the defining characteristic that differentiates evaluation research from other
TABLE 4 RESEARCH QUESTIONS ARISING IN EVALUATION RESEARCH Definitional 1. What is "P"? 2. Does " P " exist? Descriptive 3. Is this P an instance of "P"? 4. What constitutes this P? Instrumental 5. What does P do? 6. Does P cause Y? 7. How does P cause Y?
TABLE 5 POLICY QUESTIONS ARISING IN EVALUATION RESEARCH 1. What X, Y, and Z are in conflict? 2. What P is the best possible optimization of X, Y, and Z? a. What are the possible consequences of P? b. What are the marginal effects of optimization P ever optimization Q? 3. How do we reach P? a. Who will support P? (political analysis)
forms of applied social research. In spite of this, most professional attention has focused on the inquiry aspects of evaluation rather than on the value aspects. In this instance, evaluation theory and practice have been especially hampered by the general unwillingness of the social sciences to deal with issues of value. To some extent this is changing. As Novak and Gowin noted (1984, p. 174): "Classical empiricist and positivist philosophies denied that value questions and value claims had a legitimate place in a rational enterprise of knowledge construction. Contemporary philosophies, on the other hand, see value questions as crucial to the advance of our understanding in any discipline." Scriven (1980) has provided a lengthy discussion of types of questions which an evaluator may be called upon to answer, including pre-evaluative, evaluative, and quasi-evaluative questions. A summary listing of his 17 questions and subquestions is contained below in Table 6.
TABLE 6 EVALUATION QUESTIONS ARISING IN EVALUATION RESEARCH A. Pre-Evaluative Questions 1. What is X? 2. Is X valued? a. X valued by whom? b. Is X wanted, preferred, valenced? c. Is X believed to be valuable? B. Evaluative Questions 1. Is X good? 2. Is X better than Y? 3. How good is X? a. How much better is X than Y? 4. How much is X worth? a. Is X worth more than Y, and why? 5. What mixture of X and Y is worth most? 6. In what respects is X good? 7. What parts or components of X are good? C. Quasi-Evaluative Questions 1. What makes X good? 2. What would make X better? 3. Will X improve? Note. From The Logic of Evaluation (pp. 4 6 - 8 9 ) by M. Scriven, 1980, Pt. Reyes, CA: Edge press. Copyright 1980 by M. Scriven. Reprinted by permission.
Justification Gowin (in Gowin and Green, 1980, and Novak and Gowin, 1984) also has argued that value claims are answers to value questions, and that the various types of value questions are related to knowledge questions and to each other. He suggested five types of value questions (summarized below in Table 7), including intrinsic, instrumental, comparative, decision, and idealization. Again note the overlap among these questions and those in the other three categories; for example, the similarity with the comparative question in the research category, and the decision claim with a like question in the management category.
Management Claims Because evaluation in the United States is performed most frequently as a service for managers and administrators, evaluation researchers must often deal with management questions. Even though management claims are not directly a part of the inquiry or valuing aspects of evaluation research, they represent the ultimate ends to which these other claims are usually directed. Management claims are pragmatic and applied; they are action claims. The primary question is of the form "Should X be done here?" Local conditions and events make all the difference in answering management questions. The answers are specific to the local context in that with any given combination of research, policy, and evaluation claims, the answers to the management questions will still differ depending on local factors. The prevalent types of management questions are summarized below in Table 8. The relationships between
TABLE 7 M O R E EVALUATION QUESTIONS ARISING IN EVALUATION R E S E A R C H 1. Is X good? What is the good of X? (Intrinsic value claim) 2. Is X good for Z? (Instrumental value claim) 3. Is X better than Y? (Comparative value claim) 4. Is X right? Ought we to choose X? (Decision claim) 5. Is the good of X something that can be made better? (Idealization claim) Note. From The Evaluation Document: Philosophic Structure (p. 13) by D.B. Gowin and T. Green, 1980, Portland, OR: Northwest Regional Educational Laboratory, ROEP Rep. No. 38. Copyright 1980 by Northwest Regional Educational Laboratory. Reprinted by permission.
JUSTIFICATION This discussion so far has left the issue of the justification of claims largely unaddressed. This was because
313 TABLE 8 MANAGEMENT QUESTIONS ARISING IN EVALUATION RESEARCH 1. 2. 3. 4.
Should X be done here? Can X be done here? What are the costs of doing X here? What are the benefits of doing X here?
management claims and the prior three types are demonstrated in the following section.
Recommendations in Evaluation Research It is in the activity of making recommendations in evaluation research that we best see how the four types of claims may interrelate. Consider the kinds of claims used in the following example to support the recommendation that additional teacher aides should be assigned to a remedial reading program which serves students transferring from small rural schools to a slightly academically stronger suburban junior high school. 1. The suburban district accepts the responsibility to prepare adequately its students for entry into high school. (policy claim) 2. The current rural transfer students, after one year in the remedial program, are scoring at the 4Jth percentile on the district standardized reading test. (research claim) 3. This performance shows improvement over the students' entry level of the 37th percentile, but is unacceptably low for probable success in high school, based on the current high school entry average of the 63rd percentile. (evaluation claim) 4. Increases in individual attention by teacher aides and the resultant increased time on task have been shown to increase reading performance. (research claim) 5. We should increase the number of teacher aides in our remedial reading program, and we have the resources and opportunity to do so this next year. (management claim) This is an oversimplified example, but it illustrates the interconnections of research, policy, evaluation, and management claims in evaluation research. The interconnections are also suggested by the various questions which overlap two or more categories. Perhaps a single classification scheme (an adaptation of Table 3?) could be constructed to formally integrate the several question schemes described above (several of them do not represent formal classifications, only lists). OF CLAIMS of the need to first answer the prior question of what are the types of claims (question-answer propositions)
314
N I C K L. S M I T H
whose justification is of interest. One could argue simply that establishing the answer to the question constitutes the justification for the claim. The methods of justification then would be the question answering methods, whether they be logical analysis, theoretical research, or empirical research. While perhaps a reasonable starting point, this argument seems an oversimplification on both practical and methodological grounds. In practice, evaluation research claims are not made as isolated propositions, but as part of a larger interconnected argument, as in the example above. The ultimate justification for a particular proposition, therefore, depends in part on the strengths and weaknesses of the total argument within which it is embedded. Further, the meaningfulness of a particular claim also depends on the nature of the methodological system used to address it. Since different question answering methods embody different ontological and epistemological assumptions, and since there are different methods available for answering a single question, the justification of a particular question-answer proposition must necessarily include the justification of the method used to answer it. There are, of course, other ways to conceive of evaluation research than as inquiry. If we do not base our conception of evaluation research on the premise that it is a form of inquiry involving the development of question-answer propositions, then claims may be justified through non-inquiry means. For example, perhaps policy claims are justified through persuasion, negotiation, power, and influence; evaluation claims
are justified through recognized connoisseurship, and compelling criticism; and management claims are justified through systems development, historical precedent, and experienced intuition. All these forms of justification are perhaps better construed as something other than question-answer propositions. I believe that the justification of any of these viewpoints rests, however, to some extent on their usefulness in clarifying our thinking about evaluation research (evaluation theory), and in improving our practice. While certainly not the only conception, the view of evaluation research as the development of question-answer propositions does suggest a useful categorization of claims within which to consider alternative forms of justification in evaluation research. Guba (1987) has recently identified some of the diverse forms of naturalistic evaluation currently popular. Fetterman (1986) has argued that these various forms all have their own sets of standards by which the quality of their work should be judged. Are there many acceptable ways of justifying conclusions in naturalistic evaluation, or only one fundamental way? Can all forms of naturalistic evaluation be usefully construed as alternative procedures for the development of question-answer propositions? If so, then the justification of claims may center on the validity of the questionanswer propositions each form of naturalistic evaluation constructs. In reading the following four articles of this special issue, consider the types of claims being made and the methods advocated for their justification.
REFERENCES DILLON, J.T. (1984a). Finding the question f o r evaluation research (ROEP Rep. No. 102). Portland, OR: Northwest Regional Educational Laboratory.
GREEN, T.F. (1982). Policy and evaluation: A conceptual study. In N.L. Smith & D. N. Caulley (Eds.), The interaction o f evaluation and policy. Portland, OR: Northwest Regional Educational Laboratory.
DILLON, J.T. (1984b). The classification of research questions. Review o f Educational Research, 54, 327-361.
GUBA, E.G. (1987). What have we learned about naturalistic evaluation? Evaluation Practice, 8, 23-43.
EISNER, E.W. (1983). Educational connoisseurship and criticism: Their form and functions in educational evaluation. In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (Eds.), Evaluation models. Boston: Kluwer-Nijhoff.
FETTERMAN, D.M. (1986). Conceptual crossroads: Methods and ethics in ethnographic evaluation. In D. D. Williams (Ed.), Naturalistic evaluation: Vol. 30. New directions in program evaluation, San Francisco: Jossey-Bass. GOWIN, D.B., & GREEN, T. (1980). The evaluation document: Philosophic structure (ROEP Rep. No. 38). Portland, OR: Northwest Regional Educational Laboratory.
NOVAK, J.D., & GOWIN, D.B. (1984). Learning how to learn. New York: Cambridge University Press. ROSSI, P. H., & FREEMAN, H.E. (1985). Evaluation: A systematic approach. Beverly Hills, CA: Sage Publications. SCRIVEN, M. (1980). The logic o f evaluation. Pt. Reyes, CA: Edge press. SMITH, N.L. (1980). Federal research on evaluation methods in health, criminal justice, and the military. Educational Evaluation and Policy Analysis, 2, 53-59. SMITH, N.L. (1981). Noncausal inquiry in education. Educational Researcher, 10, 23.