- Email: [email protected]
Identifying and measuring unintended outcomes
Evaluation and Program Planning, Vol. I, pp. 27-34, Printed in the USA. All rights reserved.
IDENTIFYING
1984 Copyright
AND MEASURING
UNINTENDED
0149-7189/84$3.00 + .OO 0 1984 Pergamon Press Ltd
OUTCOMES
SAM SHERRILL University
of Cincinnati
ABSTRACT Unintended outcomes of governmentaI actions have received little attention from evaluators. The argument made here is that outcome evaluations should routinely include efforts to identifr and measure unintended outcomes. A systems perspective is presented which treats governmental actions as disequilibrating intrusions into reacting systems that produce an array of outcomes, only some of which are intended. Intended and unintended outcomes can be valued monetarily, in terms of human rights, or both; specifically, outcomes having monetary value alter both the level and distribution of income while outcomes with rights value alter the level and distribution of rights. More needs to be done on unintended outcomes. The systems approach offered here requires further elaboration. Design modifications that increase the chances of identifying and measuring unanticipated outcomes are needed, especially for designs based on non-equivalent groups and time-series data. Instrumentation is needed that will more accurately and sensitively register unanticipated outcomes. In the meantime, evaluators should pay more attention to theoretical predictions, the experiences of those involved with programs similar to the ones being evaluated, intercultural differences, and to unintended outcomes that originate within government itself.
INTRODUCTION Identifying and measuring intended outcomes or goals of governmental programs receives a great deal of attention from evaluators. However, relatively little attention has been accorded unintended outcomes. This lack of balance easily conveys the impression that evaluators need be concerned only with the goals of governmental programs and need not be concerned with what programs could do that no one intended or expected. Perhaps this imbalance arises from a combination of the propensity to judge programs by the degree they do what is expected of them and the convenient way most outcome evaluation methodologies, especially the experimental approach, seem to facilitate such judgments. Regardless of the reasons, the view taken here is that outcome evaluations should consist of balanced efforts to identify and measure both intended and unintended outcomes. This means that outcome evaluations confined solely to intended outcomes are neither complete nor adequate. As discussed in the following, unintended outcomes are almost certain to occur (so we are not talking about rare events) and are no more likely to be wanted than unwanted. Unintended outcomes that are unwanted are especially important Requests
for reprints
should
be addressed
to Sam Sherrill,
PhD,
to both governmental decision makers and to those unexpectedly and adversely affected, perhaps even more important than the intended outcomes. Because of evaluation’s growing importance to decision making and to the larger public interest, evaluators must begin devoting more attention to unintended outcomes. The subject has not been totally ignored. Building on the work of Ellul and Forrester, Meyers (1981) describes unintended outcomes as the unexpected or counterintuitive consequences of governmental actions that arise from a preoccupation with techniques as solutions and from a failure to see consequences as reactions of complex and interrelated social systems and subsystems. Techniques are described as standardized and efficient means for achieving predetermined results. The problem is not so much with the techniques themselves as it is with the narrow perspective that often guides their use by government. When solutions are equated with predetermined results, government seeks to impose techniques on social problems instead of solving them, and it fails to see the difference. In addition, a preoccupation with predetermined results leaves government unprepared to foresee unintended outcomes as social systemic reactions. Meyers would School
21
of Planning,
University
of Cincinnati,
Cincinnati,
OH 45221.
SAM SHERRILL
28
argue for a broader more flexible systems approach, a point developed in this article. In an equivalent fashion, Striven (1976; cited in Salasin, 1974) argues that preoccupation with the goals of governmental actions prevents evaluators from discovering unintended outcomes. His well-known solution, goal-free evaluation, would have evaluators look for outcomes without any prior knowledge of goals. Innocent of prior knowledge, outcomes are more easily discerned. Although I am certainly sympathetic with the intent of goal-free evaluation, I do not feel such innocence to be of much advantage, or even feasible in most cases. See Meyers (1981) for a more detailed critique of goal-free evaluation that expands on this point. Economists have long been concerned with a narrow class of unintended outcomes labeled externalities. Among consumers, externalities arise when one’s own consumption of an economic good affects the economic well-being of others. For example, when at THE
MEANING
AND
IMPORTANCE
Unintended outcomes are the side effects of governmental policies and programs, not the outcomes these actions are suppose to produce. Unintended outcomes are anticipated when accurately predicted in advance
.
my own expense I am immunized against a communicable disease, 1 have also improved the well-being of others because I am now one less person who can pass the disease along. Externalities arise in production when companies avoid paying all of their costs of production by imposing some of the costs on the genera1 public. For example, when a company pollutes the air and water, it imposes costs on the consumers of both instead of on the consumers of its products. From an externalities are considered economic perspective, market failures, thus occasions for governmental intervention. My interest here is in unintended outcomes created by governmental actions, including governmental reactions to externalities. The article begins with a discussion of the importance of unintended outcomes. A systems approach is then offered as both an explanation of why unintended outcomes occur and as a perspective that could be useful in identifying them. Ways of identifying and measuring them are also presented. OF UNINTENDED
OUTCOMES
and are unanticipated when they occur unexpectedly. Whether they are anticipated or not, they are either positive, neutral, or negative in value. This is shown in Figure 1. Excluded from this definition are unintended
. Governmental Intervention
Figure 1. Intended
and Unintended
Outcomes
of Governmental
Actions.
Identifying
and Measuring Unintended
extremes of intended outcomes, all effects associated with the conduct of the evaluation itself (for example, the measurement effect), as well as those associated with the presence of the intervention itself (such as the placebo effect). Labeling unintended outcomes as positive or negative (Suchman, 1967), desirable or undesirable (Merton, 1936), or as good or bad (Campbell [cited in Salasin, 19731; Greenberg, 1971) clearly implies that some are wanted and others are not. To my knowledge, no attention has been devoted to the meaning of such labels. They are simply applied with no accompanying explanation of what distinguishes one from the other, what is meant by such terms as positive and negative, or to whom such effects are either good or bad. Using Okun’s (1975) discussion of the relationship between rights and dollars as a starting point, 1 have argued elsewhere (Sherrill, in press) that outcomes can be valued not only monetarily but in human rights terms as well. For evaluation, this means that intended and unintended outcomes can have monetary value, human rights value, or both. These respective values can be either positive or negative. Outcomes with positive monetary value raise the level of a nation’s income, whereas outcomes with negative value (i.e., those that impose costs) lower the level of income. In equivalent fashion, intended and unintended outcomes having positive rights value raise the level of human rights, while those with negative value lower the level of rights. Changes in the level of income and rights will not affect all segments of the population equally. Some may be adversely affected while others benefit. Even among those helped and those hurt, the effect will vary: Some will be greatly benefited while others will be aided only slightly; the cost will be burdensome for some and minor for others. In short, changes in the level of income and rights are certain to be followed by distributional changes as well. Thus, whether an unintended outcome is positive, neutral, or negative depends on what it does to the level of national income, the level of human rights, or both, and on what it does to the distribution of income and rights. From this perspective, it seems highly unlikely that an unintended (or intended) outcome could ever be neutral, because this would require equality between monetary value and costs, equality between human rights value and loss, and a proportionally equal impact on income, rights, or both, throughout the population. (The addition of other values would diminish the likelihood even further.) An unintended outcome could be neutral if it had neither monetary nor rights value, but then it would be immaterial to decision making and irrelevant to evaluation (unless other values were involved - in which case,
Outcomes
29
we are back to the unlikely prospect that both level and distribution are unaffected). In short, unintended outcomes having either monetary or rights value are going to be important to some segments of the population and to the governmental policymakers that represent those segments, especially when monetary costs and rights losses are involved. In general, the impact of unintended outcomes that impose monetary costs or rights losses, especially on responsive segments of the population, will be greater on decision making than outcomes that bestow one or both values. Given the occurrence of positive intended outcomes, additions to income or rights would mostly reinforce previous decisions favoring the responsible program or policy. Positive unintended outcomes are bonuses or serendipities, and do not ordinarily present problems that would lead decision makers to reconsider earlier choices. Although extra values are not a problem, extra costs and losses are. What otherwise might have been a justifiable policy or program may be harder to defend in the face of extra and unanticipated monetary costs or rights losses. Indeed, much of the controversy over specific policies and programs concerns the likelihood that costly side effects will occur, how great they might be, and who will be hurt. It is not unusual for such controversies to overshadow discussions of positive intended outcomes. For example, the current controversy over nuclear power centers less on the intended outcome of nuclear plants (the production of electricity) and anticipated but unintended outcomes that are positive (reduced oil imports) than on the possible negative unintended effect of radiation on public health, especially the health of those living near such plants. Offshore drilling for oil along American coastlines will provide more oil (the intended outcome), reduce oil imports (unintended but positive outcome), but may also damage shorelines, wildlife, and oceanrelated businesses if a blowout or spill occurs. An exemption to the minimum wage law for teenagers may provide them with more jobs (intended) but may also adversely affect employment among adults who are being paid to do similar work at or just above the minimum wage. Reductions in Medicaid and Medicare coverage might reduce federal health care costs (intended) but place vital care out of the reach of those who need but cannot afford it (presumably an unintended effect). Even though unintended outcomes, especially those having negative values, can be important to decision making, this influence would not be nearly so great if the outcomes occurred rarely or even just occasionally. To the contrary, governmental actions will almost always produce unintended outcomes because such actions are not simply isolated events that produce singular consequences, but are disequilibrating intru-
SAM SHERRILL
30
sions into complex reactive systems. From a systems analysis perspective (Kuhn 1963, 1974; Kuhn & Beam, 1982), these acts-as-intrusions produce adjustments and readjustments throughout the web of relationships that tie the elements of systems together and that link different systems and subsystems. The relationships are the conduits that transport the influence of the initial intrusion to the system’s most remote parts as well as to other systems. The resulting adjustments and readjustments are the outcomes, only one or at best a few of which are intended. Governments are the most likely of society’s institutions to be confronted with
THE OUTCOMES
OF IGNORING
Inadequate theoretical knowledge about a system and the lack of prior experience with new interventions can substantially limit the ability of any government to predict the unintended outcomes of its actions (cf., Merton, 1936, pp. 898-903). However, neither knowledge, prior experience, nor the two together, guarantee immunity to unanticipated outcomes when governments single-mindedly pursue only one or a limited set of goals. This can be plainly seen in Heyneman’s (1979) description of the unwanted outcomes in Africa of dams built for irrigation and the production of electric power. The principal justification of these projects was the substantial increase in the standard of living (read level of national income) these dams would produce for the participating nations. The dams were constructed with little or no consideration for other consequences. Although dams like the High Aswan in Egypt have eliminated drought and do provide power (though not as much as expected), they also have created severe health problems for people living near them and who use their waters for irrigation. Because of the permanent presence of water, infectious diseases such as malaria and river blindness have increased dramatically. And so has snail fever or schistosomiasis, caused by parasitic flatworms for whom the snails are temporary hosts. Before the Aswan was built, the annual drought killed nearly all of the snails and their parasitic passengers. Only 5% of the population had the disease then. Now the infection rate is about 85% among those living and working near the irrigation channels and, according to Owen (1975), the rate is expected to reach 100%. The disease is difficult to detect and costly to treat. When treated with Western techniques, success is achieved only within limited areas and only so long as the techniques are in use. As Heyneman points out, “when the experts leave . . . the snails return” (P. 56). The preceding example illustrates three important points regarding unintended outcomes. First, preoccupation with a single goal, or just a few limited goals, increases the likelihood that unintended outcomes will be unanticipated because those involved
what Fuller (Horowitz, 1977) calls “polycentric problems.” These are problems that are “so ‘many-centered’ that a pull at any one point changes an entire web of relationships” (p. 59). Whether or not the social system was in equilibrium, governmental actions will force readjustments to occur. Indeed, we can say that all outcomes are systemic adjustments induced by governmental actions. As long as governmental acts are unbalancing intrusions, there will be ,a variety of adjustments-as-outcomes, only some of which were the goals of those acts.
UNINTENDED
OUTCOMES
are unlikely to consider any outcomes other than the one(s) being pursued. In some cases, the High Aswan seems to be one, the outcomes being pursued are so highly valued that there may be a willful disinterest in unintended outcomes, and a concomitant hostility toward anyone who raises the possibility that such negative outcomes might occur. Second, even familiar projects, such as dams, are likely to produce unintended outcomes, at least some of which will be unanticipated. This suggests that unanticipated outcomes are even more likely with newer untried actions where prior experience is lacking. Appropriately, more attention should be devoted to the unintended outcomes of these programs, perhaps to the point of giving them more initial emphasis than intended outcomes. Third, interventions can be expected to beget more interventions when inadequate attention is given to unintended outcomes. The Aswan has altered the course of the Nile River enough to produce erosion at the bases of the down river bridges. According to Heyneman (1979), to prevent further and irreparable damage, ten new barrier dams will have to be built at a total cost of about one half of the original dam. This clearly illustrates the fix governments get into when they find that several more compensatory interventions are needed to offset the negative effects of the initial action. In turn, these subsequent interventions can themselves have unintended negative outcomes that then require even more compensatory actions. Thus, the initial intervention produces a chain reaction of negative outcomes requiring governmental responses that begin to grow at a geometric rate. This is, of course, a most extreme possibility that is unlikely to occur each time governments act. However, a remote possibility becomes more probable when unintended outcomes are ignored. Although not illustrated by it, the preceding example does raise the point that governmental actions have a cultural character that cannot be ignored. What is culturally appropriate in the West may be wholly out of place in non-Western countries. Culturally displaced
Identifying
and Measuring Unintended
actions are all the more likely to produce unanticipated effects, a point that anthropologists have been making for a long time. This is especially likely when the government carrying out the intervention fails to recognize or willfully ignores cultural differences. The same is true of intended outcomes as well. Heyneman (1979) recounts how efforts to find local solutions to snail fever are proving more successful than imported Western medical technology. Ethiopian researchers discovered that the berries of an indigenous plant, often used by Africans in powdered form as soap, were effective in killing the snails. As an intervention,
PREDICTING
31
it is more likely to succeed because it was an intrinsic part of the environment and compatible with the local culture. Neither of these points are veiled suggestions that intercultural differences will be the downfall of every intervention that originates in one culture and is applied in another. Rather, the suggestion is that evaluators and foreign policymakers alike should be mindful of the special effect of intercultural differences on interventions and especially mindful of the possible occurrence of unintended negative outcomes arising specifically out of intercultural differences.
UNANTICIPATED
There are several ways to reduce the number of unanticipated outcomes. An obvious place to start is with the theories and research relevant to an intervention. The theory used to predict that an intervention should produce a particular intended outcome can be the source of predictions about unanticipated outcomes as well. For example, economic theory would predict that tobacco price supports should keep the income of tobacco farmers about on par with incomes in other sectors of the economythe intended outcome of the price support program. The same theory suggests that such supports would also keep tobacco prices higher than they would be otherwise, making cigarette prices higher than they would be without the supports. Higher cigarette prices discourage smoking which, in turn, has positive health effects. Thus, the theory predicted both the intended outcome and a positive unintended outcome. In a related fashion, the theoretical basis for the intervention could point to other theories or areas of research from which inferences about unintended outcomes might be drawn. Economic theory has nothing to say about the psychological problems that might be created by state lotteries. But, economic studies of the spending patterns of those who frequently participate in the lotteries could point to special problems created for those who tend to gamble compulsively. Nothing in economic theory would be of much use in addressing such a problem, but it could certainly be a useful source of inferences about a potential unintended outcome of lotteries. It is not always necessary to invoke a particular theory when simple logic may do. One does not have to be an expert in any field to realize the potential negative effects of the indiscriminate storage and dumping of toxic wastes near sources of drinking water. The variety and incidence of negative outcomes are better predicted by experts, but common sense alone leads one to expect such outcomes are likely to occur. Seeing interventions as intrusions into reacting systems is not only a useful way of explaining the ex-
Outcomes
OUTCOMES
istence of unintended outcomes, it is also useful for identifying outcomes that would otherwise go unexpected. This is not so much a theoretical approach as it is an angle of approach (systems analysis is not substantive theory but a conceptual latticework into which one can map theoretical substance). From this perspective, the evaluator is required to look at an intervention as a disturbance to a system whose consequences spread via the interrelationships comprising the system. Having identified particular adjustments as intended outcomes, the evaluator can look for other systemic adjustments that are unintended and, perhaps. otherwise unanticipated. Campbell (1969; cited in Salasin, 1973) recommends the opponents of interventions as sources of information and speculation about negative effects. Because opponents obviously do not favor the policy, they can be expected to focus their attention more on what is apt to go wrong than right with it. This could add realism to the flush of enthusiasm and over-optimism that sometimes accompanies major legislative acts (as an example, see the description by Primack & von Hippel, 1974, of how opponents of the SST emphasized the plane’s potentially adverse effects on the stratosphere as well as the noise pollution it would create at supersonic speeds). Although lining up opponents against proponents can provide a more balanced view of a policy, and some idea about what kinds of unintended outcomes might occur, such debates can be counterproductive. The fervor of the debate and the desire for a political victory (or the equally strong desire to avoid a public defeat) can create confusion and needless controversy that eclipse the whole subject of outcomes. Riecken and Boruch (1974) suggest client advisory groups and the administrators of programs as possible sources of information on unintended outcomes because both are closely and directly involved. In addition, members of experimental groups can report on changes other than those measured by evaluators. Where possible, reports from friends, family, and co-workers might also prove useful. For example, reports from
SAM SHERRILL
32
family members could be especially helpful in identifying the unintended outcomes of mental health interventions. Hyman and Wright (197 1) recommend an examination of the records and experiences of identical or similar programs to see what kind of unforeseen outcomes occurred. They also emphasize the importance of examining the relevant literature for insights. The ongoing work of others could be especially important in cases where there is a lag between the beginning of the program and the occurrence of unintended outcomes. As noted below, delays between programs and outcomes, especially unintended ones, pose special design problems. Some unintended outcomes can be identified in advance by reviewing the relevant law and governmental regulations. Because government is part of the system affected by its own actions, it is reasonable to search for unintended outcomes originating within government itself. An action taken by one part or level of government could produce unintended outcomes in the form of responses from other parts or levels (one of the purposes of interagency committees is to avoid such outcomes by coordinating policies that have potential government-wide implications). For example, a decision to close a military base could increase civilian unemployment in the affected area, requiring mandatory increases in unemployment benefits and family assistance payments. Because such benefits are required, they are predictable on the basis of the laws that require them. As another example, federal efforts to discourage cigarette smoking could reduce the level of demand for tobacoo. If the supply of tobacco remains steady or does not drop as quickly as demand, tobacco prices will fall and obligatory price support programs will be activated to support the incomes of tobacco farmers. Thus, an intervention initiated by the Department of Health and Human Services could produce unintended outcomes in the form of an obligatory response by the Department of Agriculture. In general, unintended outcomes that originate within government itself should be easier to identify and predict because the laws and regulations that require them are known and open to examination.
ESTABLISHING AND
Finally, attention should be paid to the composition of the intervention itself and to changes in that composition. Although often labeled with a single name or title, a governmental program can actually be a cluster of actions that share the same general policy objective but are also individually capable of producing different unintended outcomes. A prime example is the Model Cities program created under the Johnson Administration. The general objective of Model Cities was to improve the quality of life in the low income areas of American cities (thereby preventing further riots in those areas). This objective was to be achieved by a collection of programs to expand the range of social services available to Model Neighborhood residents, provide more training and employment opportunities, improve housing, as well as to create opportunities for greater participation by residents in local government. Applying the methods just described to a program like Model Cities would require a careful examination of each of its many component actions; clearly, such composite programs cannot be treated as if they were the monolithic interventions implied by the X in typical design statements. Interventions can evolve (or mutate, depending on raising the possibility of one’s point of view), unintended outcomes arising from the evolved intervention that would not have arisen from the original version. In general, evolutionary change is more likely to be important in evaluations based on longitudinal designs where enough time passes for evolutionary changes to occur. More than others, mental health interventions may be prone to more rapid change as the therapist modifies his or her approach during the course of therapy. Thus, in longitudinal evaluations of mental health programs, special attention might be devoted to unintended outcomes arising rather quickly from changes in the nature of the intervention itself. Because time is required for evolutionary changes to occur, a common characteristic of all unintended outcomes arising from evolving interventions is a lag in time between the initial occurrence of the intervention and the occurrence of the unintended outcomes.
THE RELATIONSHIP UNANTICIPATED
Designs used to link governmental actions with intended outcomes can also be used to link unanticipated outcomes to the same actions. The following design is presented as an example. Though untried so far as I know, it seems about as feasible as experimental designs in general.
BETWEEN OUTCOMES (R) O,,.... O’i,... (R) OcI . . . . O’,, . . . This
Oiz.... 0’12.~. O,> . . . . o’cz.. .
ACTIONS
O,J....X.... O’,,...(X)... 0,. . . . . . . . . . . . o’c3.. . . . . . . . .
is a longitudinal,
Oi,.... O’ib... 0c4 . . . . O’&. . .
experimental
Oi5..** 0’is.s. 0,5 . . . . O’,... . .
Oie O’ih Oc6 O’&
design
with
Identifying
and Measuring Unintended
multiple pretest measures. Measures Oi, through Oi3 are pretest and 0, through Oi, are posttest measures of the intended outcome for the experimental group. The corresponding measures of the intended outcome for the control group are O,, through 0C3 and O,, through OC6, respectively. Evidence of another possible outcome comes from a comparison of the posttest measures O’id through O’i6 to 0’c4 to O’c6.. These will not be precise measures because the part of the instrument that records them could not be focused in advance to register a specific kind of change. Instead, these will be trace measures of something that will have to be more specifically identified and measured in subsequent evaluations; for example, these measures could be changes informally reported by subjects or members of their families, or their responses in a survey to an open-ended question about other effects. Although imprecise and of uncertain validity, these measures may provide enough to conclude that the intervention produced something other than what was intended and expected. If the comparison reveals no significant difference, then the inference that X produced an unanticipated outcome is harder to accept, at least on the basis of initial evidence. Whether additional evaluations ought to be conducted to explore the possibility further would depend on whether the suspected outcome is positive or negative. If the initial finding provided little evidence of an unanticipated outcome, but the outcome was potentially serious, life-threatening for example, then the need for additional investigation is far more compelling than it would be if the outcome were positive or neutral in value. Of course, if the comparison revealed a significant difference, then further investigation would be warranted, especially if the outcome was potentially serious. The comparisons just described could be carried out without additional pretests. However, when time and resources allow them to be added, additional pretests would enable evaluators to make an additional comparison to further test tentative conclusions drawn from comparisons of posttest data. They could com-
Outcomes
33
pare the pretest-to-posttest trend of the experimental group (O’il-O’,,) to the trend for the control group (O’c,-O’c6). If the slope of the trend line for the control group continues unchanged while the trend line for the experimental group shifts just subsequent to the occurrence of the intervention, then the conclusion that there is a significant difference between the posttest measures for the two groups has greater credence. Conversely, a comparison of posttest measures revealing no significant difference is reinforced if the trend for the experimental group, as compared to the control group trend, seemed unaltered by the intervention. The evaluator has to be fortunate in several ways to even tentatively establish a relationship between an action and its unanticipated outcomes. Using an experimental design is not always feasible. When some lesser design is used, it is correspondingly harder to link actions and unanticipated outcomes. In addition, intended and unanticipated outcomes may start at different times after the intervention. These differential time lags will further complicate attempts to use posttest comparisons or comparisons of trends because both rest on the assumption that intended and unanticipated outcomes occur simultaneously. Worse yet, the basis for a comparison between intended and unanticipated outcomes is lost if the action does not produce the intended outcome during the time period covered by the evaluation. The preceding difficulties are compounded when multiple unanticipated outcomes are involved. Instrumentation that detects what appears to be one unanticipated outcome may be partially registering several of these outcomes at once. Because advance preparation of instruments to record different unanticipated outcomes will be difficult, catchall kinds of measures are likely to sweep several unanticipated outcomes at once. When a program is composed of several different actions, as is often the case, accurate reproduction of each one is important because one or more of the actions could have been responsible for one or more unanticipated outcomes.
SUMMARY From a systems perspective, governmental actions are disequilibrating intrusions into reacting systems that produce an array of outcomes, only some of which are intended. Intended and unintended outcomes can be valued monetarily, in terms of human rights, or both; specifically, outcomes having monetary value alter both the level and distribution of income, whereas outcomes with rights value alter the level and distribution of rights. Despite the importance of unintended outcomes and the near-certainty of their occurrence, eval-
uators have all but ignored them. The argument made here is that outcome evaluations are inadequate unless they include efforts to identify and measure unintended outcomes. Although no specific balance of effort between the two is recommended, the current division is unacceptable because so little attention is devoted to unintended outcomes. Much more needs to be done on this subject. The systems approach briefly described here requires further elaboration. More ways to modify designs that in-
34
SAM SHERRILL
crease the chances of identifying and measuring unanticipated outcomes need to be found, especially for designs based on non-equivalent groups and timeseries data. Further refinements in instrumentation are also needed. The validity and reliability of measures of these outcomes need examining from the ground up. In the meantime, evaluators might pay more attention to theoretical predictions, the experiences of those involved with programs similar to the ones being eval-
uated, and to unintended outcomes that originate within and between levels of government. Most of all, the scope of outcome evaluation should be permanently expanded to routinely include all outcomes, not just those related to goals. This is not an argument for goal-free evaluation, but an argument for an outcome evaluation that affords intended and unintended outcomes roughly equal status.
REFERENCES CAMPBELL,
D.
T.
Reforms
as
experiments.
American
Psychologist, 1969, 24, 409-429. GREENBERG, 9. G. Evaluation of social programs. In F. G. Caro (Ed.), Readings in evaluation research. New York: Russell Sage Foundation, 1971. HEYNEMAN,
D. Dams
and disease.
Human Nature, 1979, 2(2),
MEYERS, W. R. The evaluation enterprise. San Francisco: Bass, 198 1. OKUN, A. M. Equality Washington, DC: Brookings
and efficiency: Institution,
Jossey-
The big tradeoff.
1975.
OWEN, 0. National resource conservation (2nd ed.). MacMillian Press, 1975.
New York:
50-57. J. & VON HIPPEL, F. Advice and dissent: Scientists in the political arena. New York: Basic Books, 1974.
PRIMACK, HOROWITZ, The Brookings
D. L. The courts and socialpolicy. Institution, 1977.
Washington,
DC:
HYMAN, H. H., & WRIGHT, C. R. Evaluating social action programs. In F. G. Caro (Ed.), Readings in evaluation research. New York: Russell Sage Foundation, 1971.
KUHN, A. The study ofsociety: IL: Dorsey Press, 1963.
A unified approach. Homewood,
KUHN, A. The logic of social systems: A unified, deductive, system-based approach to social science. San Francisco: JosseyBass,
1974.
RIECKEN, H. W., & BORUCH R. F. Social experimentation: A method for phnning and evaluating social interventions. New York: Academic SALASIN, Campbell].
Press,
1974.
S. Experimentation
revisited
[interview
SALASIN, S. Exploring goal-free evaluation Michael Striven]. Evaluation, 1974, 2(l), 9- 16. SCRIVEN,
with
D. T.
Evaluation, 1973, l(3), 7-13.
M. Payoffs
from
evaluation.
[interview
In C. C. Abt (Ed.),
with
The
evaluution of social programs. Beverly Hills, CA: Sage, 1976.
KUHN, A. & BEAM, R. D. The logic of organization. San Francisco: Jossey-Bass, 1982.
SHERRILL, S. Outcomes, values, and costs in the evaluation of governmentalprograms. Glenview, IL: Scott, Foresman, (in press).
MERTON, R. K. The unanticipated consequences of purposive social action. American Sociological Review, 1936, I, 894-904.
SUCHMAN, Foundation,
E. A. Evaluative research. New York: 1967.
Russell
Sage
IDENTIFYING
1984 Copyright
AND MEASURING
UNINTENDED
0149-7189/84$3.00 + .OO 0 1984 Pergamon Press Ltd
OUTCOMES
SAM SHERRILL University
of Cincinnati
ABSTRACT Unintended outcomes of governmentaI actions have received little attention from evaluators. The argument made here is that outcome evaluations should routinely include efforts to identifr and measure unintended outcomes. A systems perspective is presented which treats governmental actions as disequilibrating intrusions into reacting systems that produce an array of outcomes, only some of which are intended. Intended and unintended outcomes can be valued monetarily, in terms of human rights, or both; specifically, outcomes having monetary value alter both the level and distribution of income while outcomes with rights value alter the level and distribution of rights. More needs to be done on unintended outcomes. The systems approach offered here requires further elaboration. Design modifications that increase the chances of identifying and measuring unanticipated outcomes are needed, especially for designs based on non-equivalent groups and time-series data. Instrumentation is needed that will more accurately and sensitively register unanticipated outcomes. In the meantime, evaluators should pay more attention to theoretical predictions, the experiences of those involved with programs similar to the ones being evaluated, intercultural differences, and to unintended outcomes that originate within government itself.
INTRODUCTION Identifying and measuring intended outcomes or goals of governmental programs receives a great deal of attention from evaluators. However, relatively little attention has been accorded unintended outcomes. This lack of balance easily conveys the impression that evaluators need be concerned only with the goals of governmental programs and need not be concerned with what programs could do that no one intended or expected. Perhaps this imbalance arises from a combination of the propensity to judge programs by the degree they do what is expected of them and the convenient way most outcome evaluation methodologies, especially the experimental approach, seem to facilitate such judgments. Regardless of the reasons, the view taken here is that outcome evaluations should consist of balanced efforts to identify and measure both intended and unintended outcomes. This means that outcome evaluations confined solely to intended outcomes are neither complete nor adequate. As discussed in the following, unintended outcomes are almost certain to occur (so we are not talking about rare events) and are no more likely to be wanted than unwanted. Unintended outcomes that are unwanted are especially important Requests
for reprints
should
be addressed
to Sam Sherrill,
PhD,
to both governmental decision makers and to those unexpectedly and adversely affected, perhaps even more important than the intended outcomes. Because of evaluation’s growing importance to decision making and to the larger public interest, evaluators must begin devoting more attention to unintended outcomes. The subject has not been totally ignored. Building on the work of Ellul and Forrester, Meyers (1981) describes unintended outcomes as the unexpected or counterintuitive consequences of governmental actions that arise from a preoccupation with techniques as solutions and from a failure to see consequences as reactions of complex and interrelated social systems and subsystems. Techniques are described as standardized and efficient means for achieving predetermined results. The problem is not so much with the techniques themselves as it is with the narrow perspective that often guides their use by government. When solutions are equated with predetermined results, government seeks to impose techniques on social problems instead of solving them, and it fails to see the difference. In addition, a preoccupation with predetermined results leaves government unprepared to foresee unintended outcomes as social systemic reactions. Meyers would School
21
of Planning,
University
of Cincinnati,
Cincinnati,
OH 45221.
SAM SHERRILL
28
argue for a broader more flexible systems approach, a point developed in this article. In an equivalent fashion, Striven (1976; cited in Salasin, 1974) argues that preoccupation with the goals of governmental actions prevents evaluators from discovering unintended outcomes. His well-known solution, goal-free evaluation, would have evaluators look for outcomes without any prior knowledge of goals. Innocent of prior knowledge, outcomes are more easily discerned. Although I am certainly sympathetic with the intent of goal-free evaluation, I do not feel such innocence to be of much advantage, or even feasible in most cases. See Meyers (1981) for a more detailed critique of goal-free evaluation that expands on this point. Economists have long been concerned with a narrow class of unintended outcomes labeled externalities. Among consumers, externalities arise when one’s own consumption of an economic good affects the economic well-being of others. For example, when at THE
MEANING
AND
IMPORTANCE
Unintended outcomes are the side effects of governmental policies and programs, not the outcomes these actions are suppose to produce. Unintended outcomes are anticipated when accurately predicted in advance
.
my own expense I am immunized against a communicable disease, 1 have also improved the well-being of others because I am now one less person who can pass the disease along. Externalities arise in production when companies avoid paying all of their costs of production by imposing some of the costs on the genera1 public. For example, when a company pollutes the air and water, it imposes costs on the consumers of both instead of on the consumers of its products. From an externalities are considered economic perspective, market failures, thus occasions for governmental intervention. My interest here is in unintended outcomes created by governmental actions, including governmental reactions to externalities. The article begins with a discussion of the importance of unintended outcomes. A systems approach is then offered as both an explanation of why unintended outcomes occur and as a perspective that could be useful in identifying them. Ways of identifying and measuring them are also presented. OF UNINTENDED
OUTCOMES
and are unanticipated when they occur unexpectedly. Whether they are anticipated or not, they are either positive, neutral, or negative in value. This is shown in Figure 1. Excluded from this definition are unintended
. Governmental Intervention
Figure 1. Intended
and Unintended
Outcomes
of Governmental
Actions.
Identifying
and Measuring Unintended
extremes of intended outcomes, all effects associated with the conduct of the evaluation itself (for example, the measurement effect), as well as those associated with the presence of the intervention itself (such as the placebo effect). Labeling unintended outcomes as positive or negative (Suchman, 1967), desirable or undesirable (Merton, 1936), or as good or bad (Campbell [cited in Salasin, 19731; Greenberg, 1971) clearly implies that some are wanted and others are not. To my knowledge, no attention has been devoted to the meaning of such labels. They are simply applied with no accompanying explanation of what distinguishes one from the other, what is meant by such terms as positive and negative, or to whom such effects are either good or bad. Using Okun’s (1975) discussion of the relationship between rights and dollars as a starting point, 1 have argued elsewhere (Sherrill, in press) that outcomes can be valued not only monetarily but in human rights terms as well. For evaluation, this means that intended and unintended outcomes can have monetary value, human rights value, or both. These respective values can be either positive or negative. Outcomes with positive monetary value raise the level of a nation’s income, whereas outcomes with negative value (i.e., those that impose costs) lower the level of income. In equivalent fashion, intended and unintended outcomes having positive rights value raise the level of human rights, while those with negative value lower the level of rights. Changes in the level of income and rights will not affect all segments of the population equally. Some may be adversely affected while others benefit. Even among those helped and those hurt, the effect will vary: Some will be greatly benefited while others will be aided only slightly; the cost will be burdensome for some and minor for others. In short, changes in the level of income and rights are certain to be followed by distributional changes as well. Thus, whether an unintended outcome is positive, neutral, or negative depends on what it does to the level of national income, the level of human rights, or both, and on what it does to the distribution of income and rights. From this perspective, it seems highly unlikely that an unintended (or intended) outcome could ever be neutral, because this would require equality between monetary value and costs, equality between human rights value and loss, and a proportionally equal impact on income, rights, or both, throughout the population. (The addition of other values would diminish the likelihood even further.) An unintended outcome could be neutral if it had neither monetary nor rights value, but then it would be immaterial to decision making and irrelevant to evaluation (unless other values were involved - in which case,
Outcomes
29
we are back to the unlikely prospect that both level and distribution are unaffected). In short, unintended outcomes having either monetary or rights value are going to be important to some segments of the population and to the governmental policymakers that represent those segments, especially when monetary costs and rights losses are involved. In general, the impact of unintended outcomes that impose monetary costs or rights losses, especially on responsive segments of the population, will be greater on decision making than outcomes that bestow one or both values. Given the occurrence of positive intended outcomes, additions to income or rights would mostly reinforce previous decisions favoring the responsible program or policy. Positive unintended outcomes are bonuses or serendipities, and do not ordinarily present problems that would lead decision makers to reconsider earlier choices. Although extra values are not a problem, extra costs and losses are. What otherwise might have been a justifiable policy or program may be harder to defend in the face of extra and unanticipated monetary costs or rights losses. Indeed, much of the controversy over specific policies and programs concerns the likelihood that costly side effects will occur, how great they might be, and who will be hurt. It is not unusual for such controversies to overshadow discussions of positive intended outcomes. For example, the current controversy over nuclear power centers less on the intended outcome of nuclear plants (the production of electricity) and anticipated but unintended outcomes that are positive (reduced oil imports) than on the possible negative unintended effect of radiation on public health, especially the health of those living near such plants. Offshore drilling for oil along American coastlines will provide more oil (the intended outcome), reduce oil imports (unintended but positive outcome), but may also damage shorelines, wildlife, and oceanrelated businesses if a blowout or spill occurs. An exemption to the minimum wage law for teenagers may provide them with more jobs (intended) but may also adversely affect employment among adults who are being paid to do similar work at or just above the minimum wage. Reductions in Medicaid and Medicare coverage might reduce federal health care costs (intended) but place vital care out of the reach of those who need but cannot afford it (presumably an unintended effect). Even though unintended outcomes, especially those having negative values, can be important to decision making, this influence would not be nearly so great if the outcomes occurred rarely or even just occasionally. To the contrary, governmental actions will almost always produce unintended outcomes because such actions are not simply isolated events that produce singular consequences, but are disequilibrating intru-
SAM SHERRILL
30
sions into complex reactive systems. From a systems analysis perspective (Kuhn 1963, 1974; Kuhn & Beam, 1982), these acts-as-intrusions produce adjustments and readjustments throughout the web of relationships that tie the elements of systems together and that link different systems and subsystems. The relationships are the conduits that transport the influence of the initial intrusion to the system’s most remote parts as well as to other systems. The resulting adjustments and readjustments are the outcomes, only one or at best a few of which are intended. Governments are the most likely of society’s institutions to be confronted with
THE OUTCOMES
OF IGNORING
Inadequate theoretical knowledge about a system and the lack of prior experience with new interventions can substantially limit the ability of any government to predict the unintended outcomes of its actions (cf., Merton, 1936, pp. 898-903). However, neither knowledge, prior experience, nor the two together, guarantee immunity to unanticipated outcomes when governments single-mindedly pursue only one or a limited set of goals. This can be plainly seen in Heyneman’s (1979) description of the unwanted outcomes in Africa of dams built for irrigation and the production of electric power. The principal justification of these projects was the substantial increase in the standard of living (read level of national income) these dams would produce for the participating nations. The dams were constructed with little or no consideration for other consequences. Although dams like the High Aswan in Egypt have eliminated drought and do provide power (though not as much as expected), they also have created severe health problems for people living near them and who use their waters for irrigation. Because of the permanent presence of water, infectious diseases such as malaria and river blindness have increased dramatically. And so has snail fever or schistosomiasis, caused by parasitic flatworms for whom the snails are temporary hosts. Before the Aswan was built, the annual drought killed nearly all of the snails and their parasitic passengers. Only 5% of the population had the disease then. Now the infection rate is about 85% among those living and working near the irrigation channels and, according to Owen (1975), the rate is expected to reach 100%. The disease is difficult to detect and costly to treat. When treated with Western techniques, success is achieved only within limited areas and only so long as the techniques are in use. As Heyneman points out, “when the experts leave . . . the snails return” (P. 56). The preceding example illustrates three important points regarding unintended outcomes. First, preoccupation with a single goal, or just a few limited goals, increases the likelihood that unintended outcomes will be unanticipated because those involved
what Fuller (Horowitz, 1977) calls “polycentric problems.” These are problems that are “so ‘many-centered’ that a pull at any one point changes an entire web of relationships” (p. 59). Whether or not the social system was in equilibrium, governmental actions will force readjustments to occur. Indeed, we can say that all outcomes are systemic adjustments induced by governmental actions. As long as governmental acts are unbalancing intrusions, there will be ,a variety of adjustments-as-outcomes, only some of which were the goals of those acts.
UNINTENDED
OUTCOMES
are unlikely to consider any outcomes other than the one(s) being pursued. In some cases, the High Aswan seems to be one, the outcomes being pursued are so highly valued that there may be a willful disinterest in unintended outcomes, and a concomitant hostility toward anyone who raises the possibility that such negative outcomes might occur. Second, even familiar projects, such as dams, are likely to produce unintended outcomes, at least some of which will be unanticipated. This suggests that unanticipated outcomes are even more likely with newer untried actions where prior experience is lacking. Appropriately, more attention should be devoted to the unintended outcomes of these programs, perhaps to the point of giving them more initial emphasis than intended outcomes. Third, interventions can be expected to beget more interventions when inadequate attention is given to unintended outcomes. The Aswan has altered the course of the Nile River enough to produce erosion at the bases of the down river bridges. According to Heyneman (1979), to prevent further and irreparable damage, ten new barrier dams will have to be built at a total cost of about one half of the original dam. This clearly illustrates the fix governments get into when they find that several more compensatory interventions are needed to offset the negative effects of the initial action. In turn, these subsequent interventions can themselves have unintended negative outcomes that then require even more compensatory actions. Thus, the initial intervention produces a chain reaction of negative outcomes requiring governmental responses that begin to grow at a geometric rate. This is, of course, a most extreme possibility that is unlikely to occur each time governments act. However, a remote possibility becomes more probable when unintended outcomes are ignored. Although not illustrated by it, the preceding example does raise the point that governmental actions have a cultural character that cannot be ignored. What is culturally appropriate in the West may be wholly out of place in non-Western countries. Culturally displaced
Identifying
and Measuring Unintended
actions are all the more likely to produce unanticipated effects, a point that anthropologists have been making for a long time. This is especially likely when the government carrying out the intervention fails to recognize or willfully ignores cultural differences. The same is true of intended outcomes as well. Heyneman (1979) recounts how efforts to find local solutions to snail fever are proving more successful than imported Western medical technology. Ethiopian researchers discovered that the berries of an indigenous plant, often used by Africans in powdered form as soap, were effective in killing the snails. As an intervention,
PREDICTING
31
it is more likely to succeed because it was an intrinsic part of the environment and compatible with the local culture. Neither of these points are veiled suggestions that intercultural differences will be the downfall of every intervention that originates in one culture and is applied in another. Rather, the suggestion is that evaluators and foreign policymakers alike should be mindful of the special effect of intercultural differences on interventions and especially mindful of the possible occurrence of unintended negative outcomes arising specifically out of intercultural differences.
UNANTICIPATED
There are several ways to reduce the number of unanticipated outcomes. An obvious place to start is with the theories and research relevant to an intervention. The theory used to predict that an intervention should produce a particular intended outcome can be the source of predictions about unanticipated outcomes as well. For example, economic theory would predict that tobacco price supports should keep the income of tobacco farmers about on par with incomes in other sectors of the economythe intended outcome of the price support program. The same theory suggests that such supports would also keep tobacco prices higher than they would be otherwise, making cigarette prices higher than they would be without the supports. Higher cigarette prices discourage smoking which, in turn, has positive health effects. Thus, the theory predicted both the intended outcome and a positive unintended outcome. In a related fashion, the theoretical basis for the intervention could point to other theories or areas of research from which inferences about unintended outcomes might be drawn. Economic theory has nothing to say about the psychological problems that might be created by state lotteries. But, economic studies of the spending patterns of those who frequently participate in the lotteries could point to special problems created for those who tend to gamble compulsively. Nothing in economic theory would be of much use in addressing such a problem, but it could certainly be a useful source of inferences about a potential unintended outcome of lotteries. It is not always necessary to invoke a particular theory when simple logic may do. One does not have to be an expert in any field to realize the potential negative effects of the indiscriminate storage and dumping of toxic wastes near sources of drinking water. The variety and incidence of negative outcomes are better predicted by experts, but common sense alone leads one to expect such outcomes are likely to occur. Seeing interventions as intrusions into reacting systems is not only a useful way of explaining the ex-
Outcomes
OUTCOMES
istence of unintended outcomes, it is also useful for identifying outcomes that would otherwise go unexpected. This is not so much a theoretical approach as it is an angle of approach (systems analysis is not substantive theory but a conceptual latticework into which one can map theoretical substance). From this perspective, the evaluator is required to look at an intervention as a disturbance to a system whose consequences spread via the interrelationships comprising the system. Having identified particular adjustments as intended outcomes, the evaluator can look for other systemic adjustments that are unintended and, perhaps. otherwise unanticipated. Campbell (1969; cited in Salasin, 1973) recommends the opponents of interventions as sources of information and speculation about negative effects. Because opponents obviously do not favor the policy, they can be expected to focus their attention more on what is apt to go wrong than right with it. This could add realism to the flush of enthusiasm and over-optimism that sometimes accompanies major legislative acts (as an example, see the description by Primack & von Hippel, 1974, of how opponents of the SST emphasized the plane’s potentially adverse effects on the stratosphere as well as the noise pollution it would create at supersonic speeds). Although lining up opponents against proponents can provide a more balanced view of a policy, and some idea about what kinds of unintended outcomes might occur, such debates can be counterproductive. The fervor of the debate and the desire for a political victory (or the equally strong desire to avoid a public defeat) can create confusion and needless controversy that eclipse the whole subject of outcomes. Riecken and Boruch (1974) suggest client advisory groups and the administrators of programs as possible sources of information on unintended outcomes because both are closely and directly involved. In addition, members of experimental groups can report on changes other than those measured by evaluators. Where possible, reports from friends, family, and co-workers might also prove useful. For example, reports from
SAM SHERRILL
32
family members could be especially helpful in identifying the unintended outcomes of mental health interventions. Hyman and Wright (197 1) recommend an examination of the records and experiences of identical or similar programs to see what kind of unforeseen outcomes occurred. They also emphasize the importance of examining the relevant literature for insights. The ongoing work of others could be especially important in cases where there is a lag between the beginning of the program and the occurrence of unintended outcomes. As noted below, delays between programs and outcomes, especially unintended ones, pose special design problems. Some unintended outcomes can be identified in advance by reviewing the relevant law and governmental regulations. Because government is part of the system affected by its own actions, it is reasonable to search for unintended outcomes originating within government itself. An action taken by one part or level of government could produce unintended outcomes in the form of responses from other parts or levels (one of the purposes of interagency committees is to avoid such outcomes by coordinating policies that have potential government-wide implications). For example, a decision to close a military base could increase civilian unemployment in the affected area, requiring mandatory increases in unemployment benefits and family assistance payments. Because such benefits are required, they are predictable on the basis of the laws that require them. As another example, federal efforts to discourage cigarette smoking could reduce the level of demand for tobacoo. If the supply of tobacco remains steady or does not drop as quickly as demand, tobacco prices will fall and obligatory price support programs will be activated to support the incomes of tobacco farmers. Thus, an intervention initiated by the Department of Health and Human Services could produce unintended outcomes in the form of an obligatory response by the Department of Agriculture. In general, unintended outcomes that originate within government itself should be easier to identify and predict because the laws and regulations that require them are known and open to examination.
ESTABLISHING AND
Finally, attention should be paid to the composition of the intervention itself and to changes in that composition. Although often labeled with a single name or title, a governmental program can actually be a cluster of actions that share the same general policy objective but are also individually capable of producing different unintended outcomes. A prime example is the Model Cities program created under the Johnson Administration. The general objective of Model Cities was to improve the quality of life in the low income areas of American cities (thereby preventing further riots in those areas). This objective was to be achieved by a collection of programs to expand the range of social services available to Model Neighborhood residents, provide more training and employment opportunities, improve housing, as well as to create opportunities for greater participation by residents in local government. Applying the methods just described to a program like Model Cities would require a careful examination of each of its many component actions; clearly, such composite programs cannot be treated as if they were the monolithic interventions implied by the X in typical design statements. Interventions can evolve (or mutate, depending on raising the possibility of one’s point of view), unintended outcomes arising from the evolved intervention that would not have arisen from the original version. In general, evolutionary change is more likely to be important in evaluations based on longitudinal designs where enough time passes for evolutionary changes to occur. More than others, mental health interventions may be prone to more rapid change as the therapist modifies his or her approach during the course of therapy. Thus, in longitudinal evaluations of mental health programs, special attention might be devoted to unintended outcomes arising rather quickly from changes in the nature of the intervention itself. Because time is required for evolutionary changes to occur, a common characteristic of all unintended outcomes arising from evolving interventions is a lag in time between the initial occurrence of the intervention and the occurrence of the unintended outcomes.
THE RELATIONSHIP UNANTICIPATED
Designs used to link governmental actions with intended outcomes can also be used to link unanticipated outcomes to the same actions. The following design is presented as an example. Though untried so far as I know, it seems about as feasible as experimental designs in general.
BETWEEN OUTCOMES (R) O,,.... O’i,... (R) OcI . . . . O’,, . . . This
Oiz.... 0’12.~. O,> . . . . o’cz.. .
ACTIONS
O,J....X.... O’,,...(X)... 0,. . . . . . . . . . . . o’c3.. . . . . . . . .
is a longitudinal,
Oi,.... O’ib... 0c4 . . . . O’&. . .
experimental
Oi5..** 0’is.s. 0,5 . . . . O’,... . .
Oie O’ih Oc6 O’&
design
with
Identifying
and Measuring Unintended
multiple pretest measures. Measures Oi, through Oi3 are pretest and 0, through Oi, are posttest measures of the intended outcome for the experimental group. The corresponding measures of the intended outcome for the control group are O,, through 0C3 and O,, through OC6, respectively. Evidence of another possible outcome comes from a comparison of the posttest measures O’id through O’i6 to 0’c4 to O’c6.. These will not be precise measures because the part of the instrument that records them could not be focused in advance to register a specific kind of change. Instead, these will be trace measures of something that will have to be more specifically identified and measured in subsequent evaluations; for example, these measures could be changes informally reported by subjects or members of their families, or their responses in a survey to an open-ended question about other effects. Although imprecise and of uncertain validity, these measures may provide enough to conclude that the intervention produced something other than what was intended and expected. If the comparison reveals no significant difference, then the inference that X produced an unanticipated outcome is harder to accept, at least on the basis of initial evidence. Whether additional evaluations ought to be conducted to explore the possibility further would depend on whether the suspected outcome is positive or negative. If the initial finding provided little evidence of an unanticipated outcome, but the outcome was potentially serious, life-threatening for example, then the need for additional investigation is far more compelling than it would be if the outcome were positive or neutral in value. Of course, if the comparison revealed a significant difference, then further investigation would be warranted, especially if the outcome was potentially serious. The comparisons just described could be carried out without additional pretests. However, when time and resources allow them to be added, additional pretests would enable evaluators to make an additional comparison to further test tentative conclusions drawn from comparisons of posttest data. They could com-
Outcomes
33
pare the pretest-to-posttest trend of the experimental group (O’il-O’,,) to the trend for the control group (O’c,-O’c6). If the slope of the trend line for the control group continues unchanged while the trend line for the experimental group shifts just subsequent to the occurrence of the intervention, then the conclusion that there is a significant difference between the posttest measures for the two groups has greater credence. Conversely, a comparison of posttest measures revealing no significant difference is reinforced if the trend for the experimental group, as compared to the control group trend, seemed unaltered by the intervention. The evaluator has to be fortunate in several ways to even tentatively establish a relationship between an action and its unanticipated outcomes. Using an experimental design is not always feasible. When some lesser design is used, it is correspondingly harder to link actions and unanticipated outcomes. In addition, intended and unanticipated outcomes may start at different times after the intervention. These differential time lags will further complicate attempts to use posttest comparisons or comparisons of trends because both rest on the assumption that intended and unanticipated outcomes occur simultaneously. Worse yet, the basis for a comparison between intended and unanticipated outcomes is lost if the action does not produce the intended outcome during the time period covered by the evaluation. The preceding difficulties are compounded when multiple unanticipated outcomes are involved. Instrumentation that detects what appears to be one unanticipated outcome may be partially registering several of these outcomes at once. Because advance preparation of instruments to record different unanticipated outcomes will be difficult, catchall kinds of measures are likely to sweep several unanticipated outcomes at once. When a program is composed of several different actions, as is often the case, accurate reproduction of each one is important because one or more of the actions could have been responsible for one or more unanticipated outcomes.
SUMMARY From a systems perspective, governmental actions are disequilibrating intrusions into reacting systems that produce an array of outcomes, only some of which are intended. Intended and unintended outcomes can be valued monetarily, in terms of human rights, or both; specifically, outcomes having monetary value alter both the level and distribution of income, whereas outcomes with rights value alter the level and distribution of rights. Despite the importance of unintended outcomes and the near-certainty of their occurrence, eval-
uators have all but ignored them. The argument made here is that outcome evaluations are inadequate unless they include efforts to identify and measure unintended outcomes. Although no specific balance of effort between the two is recommended, the current division is unacceptable because so little attention is devoted to unintended outcomes. Much more needs to be done on this subject. The systems approach briefly described here requires further elaboration. More ways to modify designs that in-
34
SAM SHERRILL
crease the chances of identifying and measuring unanticipated outcomes need to be found, especially for designs based on non-equivalent groups and timeseries data. Further refinements in instrumentation are also needed. The validity and reliability of measures of these outcomes need examining from the ground up. In the meantime, evaluators might pay more attention to theoretical predictions, the experiences of those involved with programs similar to the ones being eval-
uated, and to unintended outcomes that originate within and between levels of government. Most of all, the scope of outcome evaluation should be permanently expanded to routinely include all outcomes, not just those related to goals. This is not an argument for goal-free evaluation, but an argument for an outcome evaluation that affords intended and unintended outcomes roughly equal status.
REFERENCES CAMPBELL,
D.
T.
Reforms
as
experiments.
American
Psychologist, 1969, 24, 409-429. GREENBERG, 9. G. Evaluation of social programs. In F. G. Caro (Ed.), Readings in evaluation research. New York: Russell Sage Foundation, 1971. HEYNEMAN,
D. Dams
and disease.
Human Nature, 1979, 2(2),
MEYERS, W. R. The evaluation enterprise. San Francisco: Bass, 198 1. OKUN, A. M. Equality Washington, DC: Brookings
and efficiency: Institution,
Jossey-
The big tradeoff.
1975.
OWEN, 0. National resource conservation (2nd ed.). MacMillian Press, 1975.
New York:
50-57. J. & VON HIPPEL, F. Advice and dissent: Scientists in the political arena. New York: Basic Books, 1974.
PRIMACK, HOROWITZ, The Brookings
D. L. The courts and socialpolicy. Institution, 1977.
Washington,
DC:
HYMAN, H. H., & WRIGHT, C. R. Evaluating social action programs. In F. G. Caro (Ed.), Readings in evaluation research. New York: Russell Sage Foundation, 1971.
KUHN, A. The study ofsociety: IL: Dorsey Press, 1963.
A unified approach. Homewood,
KUHN, A. The logic of social systems: A unified, deductive, system-based approach to social science. San Francisco: JosseyBass,
1974.
RIECKEN, H. W., & BORUCH R. F. Social experimentation: A method for phnning and evaluating social interventions. New York: Academic SALASIN, Campbell].
Press,
1974.
S. Experimentation
revisited
[interview
SALASIN, S. Exploring goal-free evaluation Michael Striven]. Evaluation, 1974, 2(l), 9- 16. SCRIVEN,
with
D. T.
Evaluation, 1973, l(3), 7-13.
M. Payoffs
from
evaluation.
[interview
In C. C. Abt (Ed.),
with
The
evaluution of social programs. Beverly Hills, CA: Sage, 1976.
KUHN, A. & BEAM, R. D. The logic of organization. San Francisco: Jossey-Bass, 1982.
SHERRILL, S. Outcomes, values, and costs in the evaluation of governmentalprograms. Glenview, IL: Scott, Foresman, (in press).
MERTON, R. K. The unanticipated consequences of purposive social action. American Sociological Review, 1936, I, 894-904.
SUCHMAN, Foundation,
E. A. Evaluative research. New York: 1967.
Russell
Sage