- Email: [email protected]
Experimental and contextual models of program evaluation
Evaluation and Program Planning, Vol. 1, pp. 229-234 (1978)
0149-7189/78/0701-0229502.00/0
Pergamon Press. Printed in the U.S.A.
Copyright © 1978 Pergamon Press
EXPERIMENTAL AND C O N T E X T U A L MODELS OF PROGRAM E V A L U A T I O N GERALD M . B ~ T A N Department
of Anthropology
Northwestern
University
ABSTRACT
Evaluation research is one o f the most rapidly evolving fields o f applied behavioralscience. As demand f o r program assessment has increased, the number o f alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines f o r the choice o f appropriate evaluation strategies. The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses o f discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology o f program characteristics (breadth o f goals, scope o f treatment, specificity o f results, and clarity o f theory) is developed and linked to the appropriateness o f experimental and contextual evaluation.
As t h e n u m b e r o f social a c t i o n p r o g r a m s has g r o w n a n d public faith in t h e i r efficacy has declined, t h e d e m a n d for f o r m a l p r o g r a m evaluations has rapidly increased. Yet b e h a v i o r a l scientists w h o are involved in e v a l u a t i o n research face a c o n f u s i n g variety o f c o m p e t i n g assessment strategies a n d have few criteria for c h o o s i n g m e t h o d s a n d m o d e l s a p p r o p r i a t e to t h e p r o g r a m s e t t i n g at h a n d . T h e y sorely n e e d a clearer a n d m o r e u n i f i e d u n d e r s t a n d i n g of alternatives. F r o m the p e r s p e c t i v e o f a n t h r o p o l o g y , h o w e v e r , the a p p a r e n t diversity o f e v a l u a t i o n m o d e l s seems t o r e p r e s e n t a basic t h e o r e t i c a l split -- a d i s t i n c t i o n b e t w e e n " a n a l y t i c a l " a p p r o a c h e s t h a t focus o n discrete causes a n d effects as o p p o s e d to " h o l i s t i c " a p p r o a c h e s t h a t focus o n complex webs o f social a n d c u l t u r a l i n t e r a c t i o n (e.g., B e n n e t t ,
EXPERIMENTAL A c t i o n p r o g r a m s are c r e a t e d t o facilitate desired social change. W h a t e v e r t h e i r s u b s t a n t i v e focus -- h e a l t h , education, law, or a n y t h i n g else -- all a c t i o n p r o g r a m s follow the same f u n d a m e n t a l logic. A f o r m a l o r g a n i z a t i o n , such as a hospital, school, or g o v e r n m e n t agency, perceives a p r o b l e m w i t h i n its j u r i s d i c t i o n . A p r o g r a m is d e v e l o p e d w h i c h a d m i n i s t e r s a t r e a t m e n t i n t e n d e d t o alleviate t h e p r o b l e m . This t r e a t m e n t , in t u r n , affects a c l i e n t population.
1976; Weiss, 1966). Thus, c o m m o n l y used s u m m a t i v e e v a l u a t i o n s are b a s e d o n an e x p e r i m e n t a l m o d e l t h a t simplifies a c t i o n p r o g r a m realities a n d infers cause a n d e f f e c t links t h r o u g h rigorously c o n t r o l l e d c o m p a r i s o n s . O t h e r a p p r o a c h e s , such as systems, process, a n d e t h n o g r a p h i c e v a l u a t i o n , are b a s e d o n a c o n t e x t u a l m o d e l w h i c h seeks to c o m p r e h e n d t h e m u l t i - d i m e n s i o n a l realities of e v e r y d a y program operations. This p a p e r articulates the d i f f e r e n c e s b e t w e e n experim e n t a l a n d c o n t e x t u a l e v a l u a t i o n s a n d e x a m i n e s t h e i r implications. It argues t h a t each a p p r o a c h p r e s e n t s advantages a n d disadvantages, a n d t h a t each is a p p r o p r i a t e to d i f f e r e n t e v a l u a t i o n settings. A p r e l i m i n a r y t y p o l o g y is suggested w h i c h relates t h e use o f e x p e r i m e n t a l a n d cont e x t u a l m o d e l s to p a r t i c u l a r p r o g r a m characteristics.
EVALUATION
MODELS
T h e m o s t c o m m o n way o f assessing such an a c t i o n program is t h r o u g h a s u m m a t i v e e v a l u a t i o n (Scriven, 1969). A s u m m a t i v e e v a l u a t i o n assumes t h a t explicit p r o g r a m goals can b e i s o l a t e d a n d t h e results of t r e a t m e n t c a n b e q u a l i t a t i v e l y m e a s u r e d . These m e a s u r e m e n t s are c o m p a r e d w i t h t h e i n t e n d e d results, t h e results o f a l t e r n a t i v e treatm e n t s , or t h e results o f n o t r e a t m e n t a t all in o r d e r to r e a c h a j u d g e m e n t a b o u t p r o g r a m effectiveness. A l t h o u g h t h e n a t u r e o f t r e a t m e n t is n o t e d , t h e process o f p r o g r a m
Requests for reprints should be sent to Gerald M. Britan, Department of Anthropology, Northwestern University, Evanston, Illinois 60201. 229
230
G E R A L D M. B R I T A N
o p e r a t i o n is n o t a m a j o r c o n c e r n . T r e a t m e n t is viewed as a d i s c r e t e a n d isolatable i n p u t w h i c h can b e i n f e r e n t i a l l y l i n k e d to m e a s u r a b l e o u t c o m e s . T h e m a j o r m e t h o d o l o g i c a l p r o b l e m faced b y such an a p p r o a c h to e v a l u a t i o n is i n s u r i n g i n t e r n a l validity (Campbell, 1 9 6 3 ) ; p r o g r a m results m u s t b e u n e q u i v o c a l l y tied to p r o g r a m t r e a t m e n t a n d n o t to the effects of e x o g e n o u s variables. T h e s o l u t i o n is clear: s u m m a t i v e e v a l u a t i o n s s h o u l d be designed, as m u c h as possible, as e x p e r i m e n t s w h i c h c o n t r a s t s i t u a t i o n s d i f f e r i n g o n l y in the fact o f t r e a t m e n t (e.g., C a m p b e l l & Stanley, 1 9 6 6 ; R i e c k e n & B o r u c h , 1 9 7 4 ; Weiss, 1972). O f t e n , h o w e v e r , " p u r e " exp e r i m e n t s are impossible. Ethical c o n s i d e r a t i o n s m a y restrict r a n d o m a s s i g n m e n t o f subjects to critically n e e d e d p r o g r a m t r e a t m e n t s , legislative m a n d a t e s m a y require t h a t a p r o g r a m b e available to all qualified p a r t i c i p a n t s , or evalu a t i o n m a y b e g i n well a f t e r a p r o g r a m has started, elimin a t i n g any p o s s i b i l i t y o f rigorous c o n t r o l . While t h e r e is n o way to c o m p l e t e l y e l i m i n a t e t h r e a t s to i n t e r n a l validity in such s i t u a t i o n s , q u a s i - e x p e r i m e n t a l designs have b e e n d e v e l o p e d (Campbell, 1974), such as " m a t c h e d p a i r s " or " e x t e n d e d time series," t h a t m i n i m i z e e x o g e n o u s effects a n d p r o v i d e a clear u n d e r s t a n d i n g o f sources o f error. T h e e x p e r i m e n t a l m o d e l , h o w e v e r , r e m a i n s t h e ideal; non-exp e r i m e n t a l " c a s e s t u d i e s " or " a f t e r - o n l y c o m p a r i s o n s , " are seen as last resorts in w h i c h cause a n d e f f e c t is diffic u l t to isolate (Weiss, 1972). Y e t the p r o b l e m s f a c e d b y s u m m a t i v e e v a l u a t i o n trans-
CONTEXTUAL
c e n d issues o f research design a n d h i g h l i g h t the l i m i t a t i o n s of an e x p e r i m e n t a l m o d e l ' s u n d e r l y i n g a s s u m p t i o n s . Summ a t i v e e v a l u a t i o n s t r e a t a c t i o n p r o g r a m s as if t h e y were l a b o r a t o r i e s o f social c h a n g e in w h i c h specific t r e a t m e n t s are always e x p e c t e d to yield p a r t i c u l a r results. B u t social a c t i o n p r o g r a m s are n o t l a b o r a t o r y e x p e r i m e n t s ; goals are n o t always distinct, b u t are o f t e n m u l t i p l e a n d c o n t r a d i c t o r y ; t r e a t m e n t s are n o t single o c c u r r e n c e s , b u t c o n t i n u ing processes; results n e e d n o t be easily m e a s u r a b l e , b u t are o f t e n i n e x a c t a n d m u l t i - d i m e n s i o n a l . D i f f e r e n t program t r e a t m e n t s n o t o n l y p r o d u c e d i f f e r e n t degrees of success, b u t d i f f e r e n t k i n d s o f success, w h i c h m a y b e impossible to c o m p a r e q u a n t i t a t i v e l y . M a n y a c t i o n p r o g r a m s aim at q u a l i t a t i v e changes in i n s t i t u t i o n s and c o m m u n i t i e s ; n o t easily m e a s u r a b l e results, b u t r a t h e r altered c o n t e x t s for social life. T h e e x p e r i m e n t a l m o d e l simplifies a c t i o n p r o g r a m dyn a m i c s i n t o an explicit analysis o f cause a n d effect. Often, howe~er, p r o g r a m o u t c o m e s c a n n o t b e u n d e r s t o o d without considering the procedures through which treatments are applied, the m e a n i n g o f t r e a t m e n t for p a r t i c i p a n t s , a n d the social a n d c u l t u r a l c o n t e x t in w h i c h a c t i o n is embedded. Alternative evaluation approaches which consider the c o m p l e x i t y of such p r o g r a m settings do n o t r e p r e s e n t w a t e r e d - d o w n e x p e r i m e n t a l studies, b u t a t t e m p t s to und e r s t a n d m u l t i - d i m e n s i o n a l causes a n d effects t h r o u g h holistic analyses o f p r o g r a m c o n t e x t .
EVALUATION
C o n t e x t u a l e v a l u a t i o n treats a c t i o n p r o g r a m s as o n g o i n g social realities b y d i r e c t l y s t u d y i n g t h e i r e v e r y d a y activities. Goals, t r e a t m e n t s , a n d results t h e r e f o r e evolve f r o m c o n t i n u t i n g i n t e r a c t i o n s a m o n g p r o g r a m p a r t i c i p a n t s . By u n d e r s t a n d i n g these processes, a c o n t e x t u a l e v a l u a t i o n att e m p t s to explain h o w a p r o g r a m has d e v e l o p e d , w h a t it does, a n d h o w it can b e altered. C o n t e x t u a l e v a l u a t i o n o f t e n p r o v i d e s a basis for later e x p e r i m e n t a l assessment b y explicitly d e l i n e a t i n g the m e a s u r a b l e goals a n d results n e e d e d for logical inference. S o m e p r o g r a m s , h o w e v e r , m a y n e v e r b y suitable for exp e r i m e n t a l s t u d y . T h e i r o p e r a t i o n s involve so m a n y m u t u ally i n t e r a c t i n g variables t h a t causality c a n n o t b e separ a t e d f r o m t h e specific s e t t i n g in w h i c h results occur. In s u c h situations, o n l y c o n t e x t u a l u n d e r s t a n d i n g can provide m e a n i n g f u l p r o g r a m a s s e s s m e n t a n d p l a n n i n g . Several existing e v a l u a t i o n a p p r o a c h e s are implicitly b a s e d on a c o n t e x t u a l m o d e l . S y s t e m s e v a l u 9 t i o n (Etzioni, 1960; S c h u l b e r g & Baker, 1 9 6 8 ; S u c h m a n , 1970), for example, focuses o n t h e o r g a n i z a t i o n a l s e t t i n g m w h i c h program activities d e v e l o p a n d u n f o l d , c o n s i d e r i n g the syst e m i c i n t e r p l a y a m o n g o r g a n i z a t i o n a l e n v i r o n m e n t , program s t r u c t u r e a n d t r e a t m e n t i m p l e m e n t a t i o n . E t h n o graphic e v a l u a t i o n ( C e n t e r for New Schools, 1974) conc e n t r a t e s o n social a n d c u l t u r a l milieu a n d the way in w h i c h p r o g r a m p a r t i c i p a n t s develop d i f f e r e n t p e r c e p t i o n s and understandings of program operations and outcomes. Clinical e v a l u a t i o n (Glazer & Back, 1972, 1973) focuses o n d a y - t o - d a y i m p l e m e n t a t i o n , viewing t r e a t m e n t as a
MODELS
m u l t i - d i m e n s i o n a l process with c h a n g i n g o u t c o m e s over time. Political e v a l u a t i o n , o n the o t h e r h a n d , considers c o m p e t i t i v e r e l a t i o n s h i p s in p r o g r a m a d m i n i s t r a t i o n a n d implementation the process t h r o u g h w h i c h goals, procedures a n d e x p e c t a t i o n s are d e t e r m i n e d (Krause & H o w a r d , 1975). A t t e m p t s have also b e e n m a d e t o f o r m a l i z e particular m e t h o d o l o g i e s , such as adversary e v a l u a t i o n (Kourilsky, 1973) a n d subjective values m e a s u r e m e n t ( G u t t e n tag, 1973). I n s t e a d o f isolating specific causes a n d effects, such c o n t e x t u a l e v a l u a t i o n s try t o c o m p r e h e n d larger systems a n d relationships. T h e y do n o t seek unicausal links a m o n g one d i m e n s i o n a l variables, b u t ask a m o r e basic q u e s t i o n : " W h a t is going on h e r e ? " ( R a p o p o r t , 1963, p. 1900). While an e x p e r i m e n t a l a p p r o a c h assumes t h a t a c t i o n prograins can test generalizable t h e o r e t i c a l p r o p o s i t i o n s , cont e x t u a l e v a l u a t i o n s are, first o f all, e x p l o r a t o r y a n d assume t h a t valid t h e o r e t i c a l q u e s t i o n s t h e m s e l v e s m u s t still be f o r m u l a t e d . A t h e o r e t i c a l a n d m e t h o d o l o g i c a l basis for c o n t e x t u a l e v a l u a t i o n s c a n b e f o u n d in F o s t e r ' s ( 1 9 6 9 ) m o r e general discussion of d i r e c t e d social change. T o u n d e r s t a n d such change, F o s t e r argues, one m u s t c o n s i d e r t h e n a t u r e o f b o t h i n n o v a t i n g o r g a n i z a t i o n s a n d target groups as well as t h e s e t t i n g in w h i c h t h e y i n t e r a c t . T h e f o r m a t for research is a d e t a i l e d case s t u d y w h i c h a n a l y z e s t h e t o t a l s p e c t r u m of events associated w i t h p r o g r a m i m p l e m e n t a t i o n . W h a t is n e e d e d is: • . . an u n u s u a l l y b r o a d a n d flexible field research
E x p e r i m e n t a l and C o n t e x t u a l Models
m e t h o d o l o g y based on a holistic view of society and culture . . . . Where the technical, social, cultural, economic, psychological, and o t h e r p e r t i n e n t factors are nearly infinite, and usually n o t recognized in advance, this exploratory quality is e n o r m o u s l y advantageous. It vastly increases the investigator's chances of hitting u p o n the critical element in any situation. (Foster, 1969, p. 57-58) Foster's goal is to identify the implicit cultural " p r e m ises" of b o t h target groups and innovating organizations and to understand their implications for behavior in action program settings. Such a c o n t e x t u a l evaluation provides a detailed historical account of the factors affecting program operations and results. Research is inductive: issues are defined on the basis o f empirical findings, and answers to preliminary questions are the questions in the n e x t stage o f research. The first step in c o n t e x t u a l evaluation is a description o f w h a t actually happens in program treatment. Analysis then considers w h y such t r e a t m e n t occurs and h o w it relates to formal rules, informal goals, varying participant understandings, external pressures, and multiple program results. Finally, variations in program results are linked to variations in the t r e a t m e n t process and provide a basis for future program improvements. Contextual evaluation uses multiple m e t h o d s and data sources to gain a holistic perspective on program activities. Procedures can vary greatly from case to case and are best illustrated by a concrete example. In 1975, the National A c a d e m y of Sciences asked me to evaluate a federal program which developed coopera-
THE PURPOSES
OF EXPERIMENTAL
Experimental evaluations relate program treatments to program o u t c o m e s w i t h o u t directly examining causal processes. J u d g m e n t s about program effectiveness are made through controlled comparisons of results which assume uni-dimensional program goals and treatments. As a result, findings are clear and precise, but sometimes overly simplified. Contextual evaluations investigate causal relationships b e t w e e n program treatments and o u t c o m e s by directly examining the processes through which results are achieved. Program goals, treatments, and results are seen as multidimensional, and differences in program success are expected. Thus, c o n t e x t u a l evaluations provide in-depth (though sometimes overly subjective) findings. C o n t e x t u a l and experimental approaches not only utilize different kinds of data, but also provide alternative frameworks for understanding which address very different evaluation purposes. The quantitative comparisons of experimental evaluation provide decision makers with a precise basis for choices among program alternatives. Thus, experimental evaluation is especially useful for ex post facto decisions about the c o n t i n u a t i o n and expansion of pilot projects or o t h e r programs of a limited duration. Yet since experimental evaluations tend to ignore the de-
231
tive policy experiments with o t h e r government agencies. Nearly thirty projects had already been developed, which aimed at private sector e c o n o m i c and technological benefits. Their specific results were the subject o f internal review. My research, however, asked a far b r o a d e r question: To what e x t e n t were cooperative policy experiments a feasible strategy for change? I followed a c o n t e x t u a l approach. The first step was to analyze program d o c u m e n t s and identify formal goals and procedures. These were then c o m p a r e d with first-hand observations and discussions of everyday procedures. Differences were n o t e d and these became the subject of intensive interviews. It soon became clear that formal mandates were greatly m o d i f i e d by informal understandings about what kinds o f policy changes were possible and which w o u l d be m o s t beneficial to the program and to individual staff members. A quantitative analysis o f project characteristics d e t e r m i n e d the choice of eleven m o r e intensive case studies. These cases provided additional data on the informal objectives of agency clients, problems of project development, and factors affecting i m p l e m e n t a t i o n success. These materials were c o m b i n e d in an analysis o f program process which yielded specific r e c o m m e n d a t i o n s for changes in policy emphasis, organizational structure, and operating procedures (Britan, 1977, 1978, in press). Many of these r e c o m m e n d a t i o n s have already been implemented. The kinds of i n f o r m a t i o n and analysis provided by such a c o n t e x t u a l study are very different from an experimental appraisal of results. It is not, therefore, surprising that the two evaluation models are associated with rather different evaluation purposes.
AND CONTEXTUAL
EVALUATION
tails o f t r e a t m e n t i m p l e m e n t a t i o n , they provide little formative feedback for the d e v e l o p m e n t o f program improvements (Scriven, 1969). This is precisely the strong p o i n t o f a c o n t e x t u a l study. E x p e r i m e n t a l and c o n t e x t u a l approaches are o f t e n useful c o m p l e m e n t s . C o n t e x t u a l research can define measurable results for later experimental assessment, just as exp e r i m e n t a t i o n can test specific propositions within a broader c o n t e x t u a l framework. The difference b e t w e e n the two approaches, however, is far m o r e than the difference b e t w e e n qualitative and quantitative measurement. Experimental proofs can be inferred as easily on the basis o f qualitative distinctions - - w h e t h e r an effect occurs at all - as quantitative ones. At the same time, c o n t e x t u a l studies measure variables and results whenever appropriate. The approaches differ m o r e p r o f o u n d l y in their assumptions of h o w program dynamics and results can best be understood. The purpose of an evaluation, and the appropriateness of these alternative evaluation models, depends greatly on the nature of the program being considered. Next, a typology of program characteristics is developed which provides preliminary guidelines for the choice of evaluation strategies.
232
G E R A L D M. B R I T A N
EVALUATION
MODELS
AND
A l t h o u g h all a c t i o n p r o g r a m s have a similar logic in facilit a t i n g social change, t h e i r p a r t i c u l a r c h a r a c t e r i s t i c s vary greatly. A n e w e d u c a t i o n a l p r o g r a m , for e x a m p l e , c o u l d involve a single change in an o n g o i n g c u r r i c u l u m or the e s t a b l i s h m e n t of an entire a l t e r n a t i v e school. It is n o t very surprising t h a t e v a l u a t i o n s o f these t w o p r o g r a m s w o u l d b e very different. The choice of contextual and experimental evaluation strategies c a n be l i n k e d to an u n d e r s t a n d i n g o f such program differences. A few basic p r o g r a m c h a r a c t e r i s t i c s seem especially crucial: specificity of p r o g r a m goals, b r e a d t h o f p r o g r a m t r e a t m e n t , clarity of u n d e r l y i n g t h e o r y , a n d scope of p r o g r a m results ( S u c h m a n , 1 9 7 0 ; T h o m p s o n , 1967; Weiss, 1972; Weiss & Rein, 1974).
Program Goals Program goals are the i n t e n d e d results of p r o g r a m treatm e n t . T h e y m a y b e e x t r e m e l y specific, as in a medical ino c u l a t i o n p r o g r a m , or t h e y m a y b e diffuse, as in a h e a l t h p r o g r a m t h a t aims at i m p r o v i n g the general well-being o f a target c o m m u n i t y . In a d d i t i o n , p r o g r a m goals m a y be singular or m u l t i p l e , f o r m a l l y s t a t e d or i n f o r m a l l y u n d e r s t o o d , agreed u p o n or controversial.
Program Treatment and Program T h e o r y P r o g r a m t r e a t m e n t s can be single i n p u t s (requiring reckless drivers to a t t e n d a special l e c t u r e ) or m u l t i p l e i n p u t s (a m a n y - f a c e t e d driver safety p r o g r a m ) ; t r e a t m e n t s can o c c u r at o n e p o i n t in time or t h e y can involve an o n g o i n g p r o g r a m ; t r e a t m e n t s can address a single p r o b l e m or a n u m b e r o f d i f f e r e n t ones. In principle, p r o g r a m t r e a t m e n t is b a s e d on u n d e r l y i n g theory an u n d e r s t a n d i n g o f the cause a n d e f f e c t relat i o n s h i p b e t w e e n p r o g r a m t r e a t m e n t a n d the a c h i e v e m e n t of p r o g r a m goals, t"ew p r o g r a m s , h o w e v e r , have a clear a n d precise t h e o r e t i c a l basis a n d t h e r e f o r e p r o g r a m treatm e n t s o f t e n involve m u l t i p l e , fuzzy, or even c o n t r a d i c t o r y u n d e r s t a n d i n g s of t h e p r o b l e m to b e solved. Program Results A l t h o u g h p r o g r a m results are i n t e n d e d to fulfill p r o g r a m goals, actual results m a y vary f r o m t h e e x p e c t e d a n d una n t i c i p a t e d results m a y b e far m o r e i m p o r t a n t t h a n int e n d e d ones. While s o m e p r o g r a m results, such as the t e a c h i n g of m a t h e m a t i c a l skills, m a y be specific a n d m e a s u r a b l e , others, such as i m p r o v e m e n t s in the quality of life, are m o r e diffuse.
Other Program Characteristics O t h e r i m p o r t a n t p r o g r a m c h a r a c t e r i s t i c s m i g h t also be identified. A p r o g r a m ' s o r g a n i z a t i o n a l locus as p a r t o f an e s t a b l i s h e d agency or an i n d e p e n d e n t e n t i t y is o f t e n an i m p o r t a n t c o n s i d e r a t i o n . Program scale w h e t h e r activities o c c u r at an i n s t i t u t i o n a l , c o m m u n i t y , or n a t i o n a l level c o u l d also b e c o n s i d e r e d . R a t h e r t h a n c o m p l i c a t e the issue at this stage, h o w e v e r , we will simply assume t h a t such factors are s u b s u m e d b y p r o g r a m goala, p r o g r a m results, a n d p r o g r a m t h e o r y a n d t r e a t m e n t .
PROGRAM
CHARACTERISTICS
Program T y p e s T h e s e p r o g r a m c h a r a c t e r i s t i c s can b e d i c h o t o m i z e d in t e r m s o f n o m i n a l d i s t i n c t i o n s a l o n g three m a j o r d i m e n sions. Thus, p r o g r a m goals can be n a r r o w or b r o a d , program results can b e specific or diffuse, and p r o g r a m t h e o r y can b e clear or fuzzy (simple or c o m p l i c a t e d treatm e n t s are seen to reflect t h e o r y ) . A total o f eight differe n t p r o g r a m t y p e s are possible. First we will c o n s i d e r the t w o e x t r e m e cases, w h e r e t h e c h o i c e b e t w e e n e x p e r i m e n tal a n d c o n t e x t u a l e v a l u a t i o n is clear. 1. Narrow Goals, Clear Theory, Specific Results. This t y p e o f p r o g r a m nearly a p p r o x i m a t e s t h e l a b o r a t o r y ideal a n d is, t h e r e f o r e , the m o s t a p p r o p r i a t e for e x p e r i m e n t a l assessment. For e x a m p l e , w h e n a p u b l i c h e a l t h p r o g r a m is testing the efficacy of a n e w vaccine, t r e a t m e n t can be una m b i g u o u s l y l i n k e d to o u t c o m e a n d a precise j u d g m e n t of effectiveness can b e made. Such t r e a t m e n t s are b a s e d o n u n i - d i m e n s i o n a l cause a n d e f f e c t a s s u m p t i o n s . While social p r o g r a m s are rarely this clear, w h e n t r e a t m e n t is simple a n d results precise, e x p e r i m e n t a l e v a l u a t i o n is a p p r o p r i a t e . S u p p l e m e n t a r y c o n t e x t u a l research m i g h t also b e used to verify t h r e a t m e n t o c c u r r e n c e , elicit a l t e r n a t i v e goals, or assess u n i n t e n d e d results. 2. Broad Goals, Fuzzy Theory, Diffuse Results. Focusing o n discrete causes a n d effects m a k e s little sense w h e n program o p e r a t i o n is so p o o r l y u n d e r s t o o d . W h e n evaluating a c o m m u n i t y m e n t a l h e a l t h c e n t e r , for e x a m p l e , a res e a r c h e r faces m u t l i p l e goals, diverse u n d e r s t a n d i n g s , varied t r e a t m e n t s , and results t h a t range f r o m changes in individual p s y c h o l o g y to overall effects on e n t i r e neighb o r h o o d s . A d e t a i l e d e x a m i n a t i o n of p r o g r a m c o n t e x t explains hove these a n d o t h e r factors i n t e r r e l a t e in an ongoing p r o g r a m process. This w o u l d not, o f course, preclude the use of later e x p e r i m e n t a l validation, if the clear d e f i n i t i o n s n e e d e d for logical i n f e r e n c e e v e n t u a l l y emerge. Most a c t i o n p r o g r a m s fall s o m e w h e r e b e t w e e n these e x t r e m e s o f specificity a n d diffuseness, m a k i n g the c h o i c e o f an assessment strategy m o r e c o m p l i c a t e d . E i t h e r an exp e r i m e n t a l or c o n t e x t u a l e v a l u a t i o n can o f t e n be applied, b u t each involves d i s t i n c t a d v a n t a g e s and disadvantages.
3. Broad Goals, Clear Theory, Specific Results. S c h o o l b r e a k f a s t p r o g r a m s are p r o v i d e d to p r o m o t e b r o a d changes in h e a l t h , e m o t i o n a l a d j u s t m e n t , a n d a c a d e m i c p e r f o r m a n c e o f s t u d e n t s . Before e x p e r i m e n t a l research can p r o c e e d , these b r o a d f o r m a l a n d i n f o r m a l goals m u s t be b e t t e r specified. Once this is d o n e , actual results, such as d e c l i n i n g illness rates, can b e e x p e r i m e n t a l l y verified. T h e reasons w h y some e x p e c t e d effects do n o t o c c u r w o u l d , h o w e v e r , b e a s u b j e c t for c o n t e x t u a l study.
4. Narrow Goals, Clear Theory, Diffuse Results. A comp e n s a t o r y m a t h e m a t i c s p r o g r a m is designed to i m p r o v e s t u d e n t skills t h r o u g h r e m e d i a l training. T h e effectiveness of this t r a i n i n g can be easily t e s t e d t h r o u g h an experim e n t a l c o m p a r i s o n of s t a n d a r d i z e d test scores. Such an a p p r o a c h , h o w e v e r , m a y ignore equally i m p o r t a n t supple-
E x p e r i m e n t a l a n d C o n t e x t u a l Models
233
m e n t a r y results -- i m p r o v e m e n t s in o t h e r subject areas, e n h a n c e d s t u d e n t a d j u s t m e n t , or altered career interests. Moreover, if the results of c o m p e n s a t o r y t r a i n i n g v a r y s u b s t a n t i a l l y b e t w e e n classrooms or schools (as is t h e case in at least o n e m a j o r M i d w e s t e r n city), c o n t e x t u a l research can find reasons in t h e v a r i a t i o n in p r o g r a m implementation.
p e r i m e n t a l e v a l u a t i o n w o u l d tell us little a b o u t w h y t h e y were achieved or h o w t h e y c o u l d b e i m p r o v e d . Such an e v a l u a t i o n w o u l d also ignore o t h e r p o t e n t i a l l y i m p o r t a n t p r o g r a m effects, such as i m p r o v e m e n t s in p r e - r e t i r e m e n t career s a t i s f a c t i o n or p o s t - r e t i r e m e n t a d j u s t m e n t . A cont e x t u a l s t u d y seems w a r r a n t e d to d e t e r m i n e w h a t the club actually does a n d w h a t effects it has.
5. Narrow Goals, Fuzzy Theory, Specific Results. If goals
7o Broad Goals, Fuzzy Theory, Specific Results. A comm u n i t y d e v e l o p m e n t p r o g r a m can fulfill a m u l t i t u d e o f purposes. W i t h o u t a c o n t e x t u a l s t u d y o f these goals a n d t h e m e t h o d s t h r o u g h w h i c h t h e y are achieved, t h e r e are few guidelines for e x p e r i m e n t a l assessment, even if actual results are quite specific. C o n t e x t u a l e v a l u a t i o n m a y provide a basis for l a t e r e x p e r i m e n t a l analysis, b u t it also adds d a t a t h a t is d i r e c t l y r e l e v a n t to f u t u r e p r o g r a m p l a n n i n g .
are n a r r o w a n d results specific, e x p e r i m e n t a l i n f e r e n c e can p r o v i d e a clear basis for j u d g i n g p r o g r a m effectiveness. It c a n n o t , h o w e v e r , clarify the reasons for p r o g r a m failure or success. This k n o w l e d g e is especially i m p o r t a n t if p r o g r a m results are i n c o n s i s t e n t . F o r e x a m p l e , a m a j o r A m e r i c a n c o m p a n y r e c e n t l y i n t r o d u c e d n e w w o r d - p r o c e s s i n g techn o l o g y a i m e d at e n h a n c i n g business office efficiency. In a b o u t 80% o f the cases, office efficiency did in fact imp r o v e ; b u t in 20% it actually declined. T h e c o m p a n y , t h e r e f o r e , c o m m i s s i o n e d a c o n t e x t u a l s t u d y to e x a m i n e t h e i m p a c t o f its e q u i p m e n t o n office r o u t i n e s a n d to learn w h y efficiency s o m e t i m e s d r o p p e d . These findings will b e i n c o r p o r a t e d in i m p r o v e d t e c h n o l o g y in the future.
6. Narrow Goals, Fuzzy Theory, Diffuse Results. A comp a n y m a y i n a u g u r a t e a p r e - r e t i r e m e n t club for the relatively n a r r o w p u r p o s e o f i m p r o v i n g the financial p l a n n i n g o f e m p l o y e e s . Such a p r o g r a m m i g h t b e e x p e r i m e n t a l l y e v a l u a t e d simply t h r o u g h c o m p a r i s o n s o f the financial s t a t u s o f retirees. Yet, if results were positive, such an ex-
8. Broad Goals, Clear Theory, Diffuse Results. It seems u n l i k e l y t h a t m a n y p r o g r a m s w i t h clearly d e f i n e d t h e o r y a n d t r e a t m e n t will also have b r o a d goals and diffuse results. Such situations, however, w o u l d call for c o n t e x t u a l appraisal a n d a clearer u n d e r s t a n d i n g o f goals a n d results before experimental study could proceed. In general, e x p e r i m e n t a l e v a l u a t i o n clearly judges p r o g r a m results b u t adds l i t d e u n d e r s t a n d i n g o f causal process, while c o n t e x t u a l research elucidates causal processes b u t provides little basis for explicit j u d g m e n t . Dep e n d i n g o n e v a l u a t i o n p u r p o s e s , the t w o a p p r o a c h e s can o f t e n b e usefully c o m b i n e d .
DISCUSSION A l t h o u g h applied social scientists are faced w i t h a growing n u m b e r of c o m p e t i n g a p p r o a c h e s to p r o g r a m evaluation, few c h o i c e guidelines have b e e n developed. This p a p e r has argued t h a t t h e a p p a r e n t variety o f e v a l u a t i o n strategies r e p r e s e n t s t w o u n d e r l y i n g m o d e l s t h a t are b a s e d o n d i f f e r e n t epistemological a s s u m p t i o n s . E x p e r i m e n t a l evalu a t i o n s isolate discrete causes a n d effects t h r o u g h controlled c o m p a r i s o n s of specific p r o g r a m results. T h e goal is an explicit j u d g m e n t of p r o g r a m success. C o n t e x t u a l evaluation, on the o t h e r h a n d , tries t o u n d e r s t a n d the c o m p l e x i t y o f a c t i o n p r o g r a m goals, t r e a t m e n t s , a n d results as it u n f o l d s in actual p r o g r a m processes. J u d g m e n t s o f success c a n n o t be e x p r e s s e d along a single dim e n s i o n a n d t h e view of d i f f e r e n t individuals a n d institut i o n s can b e e x p e c t e d to vary. In a practical sense, e x p e r i m e n t a l and c o n t e x t u a l evaluations are each a p p r o p r i a t e to d i f f e r e n t p r o g r a m types a n d e v a l u a t i o n purposes. A t y p o l o g y of p r o g r a m characteristics b a s e d o n t h e o r e t i c a l clarity, t r e a t m e n t c o m p l e x i t y , result specificity a n d goal b r e a d t h was develo p e d a n d r e l a t e d to t h e c h o i c e of e v a l u a t i o n alternatives.
It was argued t h a t while n o single evaluative strategy is always a p p r o p r i a t e , c o n t e x t u a l m o d e l s have at least as m u c h validity as m o r e c o m m o n l y used e x p e r i m e n t a l approaches. A l t h o u g h o u r t y p o l o g y o f p r o g r a m characteristics is c o m p r e h e n s i v e , n o t all p r o g r a m t y p e s are equally prevalent. Indeed, n a r r o w l y d e f i n e d p r o g r a m s suitable for p u r e l y e x p e r i m e n t a l e v a l u a t i o n seem to b e the e x c e p t i o n , n o t t h e rule. A c t i o n p r o g r a m s , a f t e r all, are a t t e m p t s to direct social change, a n d p r e s e n t u n d e r s t a n d i n g s of social d y n a m i c s are generally q u i t e imprecise. Most existing p r o g r a m s do have s o m e w h a t f u z z y t h e o r y , fairly b r o a d goals, a n d r a t h e r diffuse results, m a k i n g an u n d e r s t a n d i n g of p r o g r a m c o n t e x t essential. O u r analysis, o f course, is only a first step a n d m o r e specific guidelines are n e e d e d . E x p e r i m e n t a l testing is certainly a w o r t h y l a b o r a t o r y ideal. Yet it is o f t e n o f little help in evaluating a c t i o n p r o g r a m s as t h e y are, a n d as t h e y m u s t be if they are to have a significant i m p a c t o n the social p r o b l e m s we face.
REFERENCES BENNETT, J. W. Anticipation, adaptation, and the concept of culture in anthropology. Science, 1976, 192, 847-853.
BRITAN, G. M. Public policy and innovation: An ethnographic evaluation of the Experimental Technology Incentives Program. Washington: National Academy of Sciences, 1977.
234
GERALD
M. B R I T A N
BRITAN, G. M. The place of anthropology in program evaluation.
KOURILSKY, M. An adversary model for educational evaluation.
Anthropological Quarterly, 1978, 51 (2), 119-128.
Evaluation Comment, 1973, 2, 3-6.
BRITAN, G. M. Evaluating an a t t e m p t to institutionalize change in the federal bureaucracy. In press. CAMPBELL, D. T. From description to experimentation. In C. W. Harris (Ed.), Problems in measuring change. Madison: University of Wisconsin Press, 1963. CAMPBELL, D. T. Quasi-experimental designs. In H. W. Riecken & R. F. Boruch (Eds.), Social experimentation. New York: Academic Press, 1974. CAMPBELL, D. T., & STANLEY, S. C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1966. CENTER FOR NEW SCHOOLS, The use o f ethnography for educational evaluation. Chicago: CNS, 1974. ETZIONI, A. Two approaches to organizational analysis: A critique and a suggestion. Administrative Science Quarterly, 1960, 5, 2 5 7 - 2 5 8 . FOSTER, (;. Applied anthropology. Boston: Little, Brown, 1969. G L A Z E R , E. M., & BACK, T. E. A clinical approach to program evaluation. Evaluation, 1972, 1, 54-60. GLAZER, E. M., & BACK, T. E. A look at participant observation. Evaluation, 1973, 1(3), 46-49. GUTTENTAG, M. Subjectivity and its use in evaluation research.
Evaluation, 1973, 1(2), 60-65.
KRAUSE, M. S., & HOWARD, K. I. Program evaluation in the public interest: A new research methodology. In Readings in program evaluation. Evanston: Northwestern University, 1975. RAPOPORT, R. N. Social anthropology and mental health. In En-
cyclopedia o f mental health. New York: Franklin Watts, 1963. RIECKEN, H. W., & BORUCH, R. F. Social experimentation. New York: Academic Press, 1974. SCHULBERG, H. C., & BAKER, F. Program evaluation models and the implementation of research findings. American Journal o f Public Health, 1968, 58, 1248-1255. SCRIVEN, M. The m e t h o d o l o g y of evaluation. In R. Tyler, R. M. Gagner, & M. Scriven (Eds.), Perspectives o f curriculum development. Chicago: Rand McNally, 1969. SUCHMAN, E. A. Action for what? A critique of evaluative research. In R. O'Toole (Ed.), The organization, management, and tactics o f social research. Cambridge: S c h e n k m a n Publishing Company, 1970. THOMPSON, J. D. Organizations in action. New York: McGrawHill, 1967. WEISS, C. Evaluation research. Englewood Cliffs: Prentice-Hall, 1972. WEISS, R. S. Alternative approaches to the study of complex situations. Human Organization, 1966, 25, 198-206. WEISS, R. S., & REIN, M. The evaluation of broad-aim programs: Difficulties in experimental design and an alternative. In C. Weiss (Ed.), Readings in program evaluation. Englewood Cliffs: PrenticeHall, 1974.
0149-7189/78/0701-0229502.00/0
Pergamon Press. Printed in the U.S.A.
Copyright © 1978 Pergamon Press
EXPERIMENTAL AND C O N T E X T U A L MODELS OF PROGRAM E V A L U A T I O N GERALD M . B ~ T A N Department
of Anthropology
Northwestern
University
ABSTRACT
Evaluation research is one o f the most rapidly evolving fields o f applied behavioralscience. As demand f o r program assessment has increased, the number o f alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines f o r the choice o f appropriate evaluation strategies. The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses o f discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology o f program characteristics (breadth o f goals, scope o f treatment, specificity o f results, and clarity o f theory) is developed and linked to the appropriateness o f experimental and contextual evaluation.
As t h e n u m b e r o f social a c t i o n p r o g r a m s has g r o w n a n d public faith in t h e i r efficacy has declined, t h e d e m a n d for f o r m a l p r o g r a m evaluations has rapidly increased. Yet b e h a v i o r a l scientists w h o are involved in e v a l u a t i o n research face a c o n f u s i n g variety o f c o m p e t i n g assessment strategies a n d have few criteria for c h o o s i n g m e t h o d s a n d m o d e l s a p p r o p r i a t e to t h e p r o g r a m s e t t i n g at h a n d . T h e y sorely n e e d a clearer a n d m o r e u n i f i e d u n d e r s t a n d i n g of alternatives. F r o m the p e r s p e c t i v e o f a n t h r o p o l o g y , h o w e v e r , the a p p a r e n t diversity o f e v a l u a t i o n m o d e l s seems t o r e p r e s e n t a basic t h e o r e t i c a l split -- a d i s t i n c t i o n b e t w e e n " a n a l y t i c a l " a p p r o a c h e s t h a t focus o n discrete causes a n d effects as o p p o s e d to " h o l i s t i c " a p p r o a c h e s t h a t focus o n complex webs o f social a n d c u l t u r a l i n t e r a c t i o n (e.g., B e n n e t t ,
EXPERIMENTAL A c t i o n p r o g r a m s are c r e a t e d t o facilitate desired social change. W h a t e v e r t h e i r s u b s t a n t i v e focus -- h e a l t h , education, law, or a n y t h i n g else -- all a c t i o n p r o g r a m s follow the same f u n d a m e n t a l logic. A f o r m a l o r g a n i z a t i o n , such as a hospital, school, or g o v e r n m e n t agency, perceives a p r o b l e m w i t h i n its j u r i s d i c t i o n . A p r o g r a m is d e v e l o p e d w h i c h a d m i n i s t e r s a t r e a t m e n t i n t e n d e d t o alleviate t h e p r o b l e m . This t r e a t m e n t , in t u r n , affects a c l i e n t population.
1976; Weiss, 1966). Thus, c o m m o n l y used s u m m a t i v e e v a l u a t i o n s are b a s e d o n an e x p e r i m e n t a l m o d e l t h a t simplifies a c t i o n p r o g r a m realities a n d infers cause a n d e f f e c t links t h r o u g h rigorously c o n t r o l l e d c o m p a r i s o n s . O t h e r a p p r o a c h e s , such as systems, process, a n d e t h n o g r a p h i c e v a l u a t i o n , are b a s e d o n a c o n t e x t u a l m o d e l w h i c h seeks to c o m p r e h e n d t h e m u l t i - d i m e n s i o n a l realities of e v e r y d a y program operations. This p a p e r articulates the d i f f e r e n c e s b e t w e e n experim e n t a l a n d c o n t e x t u a l e v a l u a t i o n s a n d e x a m i n e s t h e i r implications. It argues t h a t each a p p r o a c h p r e s e n t s advantages a n d disadvantages, a n d t h a t each is a p p r o p r i a t e to d i f f e r e n t e v a l u a t i o n settings. A p r e l i m i n a r y t y p o l o g y is suggested w h i c h relates t h e use o f e x p e r i m e n t a l a n d cont e x t u a l m o d e l s to p a r t i c u l a r p r o g r a m characteristics.
EVALUATION
MODELS
T h e m o s t c o m m o n way o f assessing such an a c t i o n program is t h r o u g h a s u m m a t i v e e v a l u a t i o n (Scriven, 1969). A s u m m a t i v e e v a l u a t i o n assumes t h a t explicit p r o g r a m goals can b e i s o l a t e d a n d t h e results of t r e a t m e n t c a n b e q u a l i t a t i v e l y m e a s u r e d . These m e a s u r e m e n t s are c o m p a r e d w i t h t h e i n t e n d e d results, t h e results o f a l t e r n a t i v e treatm e n t s , or t h e results o f n o t r e a t m e n t a t all in o r d e r to r e a c h a j u d g e m e n t a b o u t p r o g r a m effectiveness. A l t h o u g h t h e n a t u r e o f t r e a t m e n t is n o t e d , t h e process o f p r o g r a m
Requests for reprints should be sent to Gerald M. Britan, Department of Anthropology, Northwestern University, Evanston, Illinois 60201. 229
230
G E R A L D M. B R I T A N
o p e r a t i o n is n o t a m a j o r c o n c e r n . T r e a t m e n t is viewed as a d i s c r e t e a n d isolatable i n p u t w h i c h can b e i n f e r e n t i a l l y l i n k e d to m e a s u r a b l e o u t c o m e s . T h e m a j o r m e t h o d o l o g i c a l p r o b l e m faced b y such an a p p r o a c h to e v a l u a t i o n is i n s u r i n g i n t e r n a l validity (Campbell, 1 9 6 3 ) ; p r o g r a m results m u s t b e u n e q u i v o c a l l y tied to p r o g r a m t r e a t m e n t a n d n o t to the effects of e x o g e n o u s variables. T h e s o l u t i o n is clear: s u m m a t i v e e v a l u a t i o n s s h o u l d be designed, as m u c h as possible, as e x p e r i m e n t s w h i c h c o n t r a s t s i t u a t i o n s d i f f e r i n g o n l y in the fact o f t r e a t m e n t (e.g., C a m p b e l l & Stanley, 1 9 6 6 ; R i e c k e n & B o r u c h , 1 9 7 4 ; Weiss, 1972). O f t e n , h o w e v e r , " p u r e " exp e r i m e n t s are impossible. Ethical c o n s i d e r a t i o n s m a y restrict r a n d o m a s s i g n m e n t o f subjects to critically n e e d e d p r o g r a m t r e a t m e n t s , legislative m a n d a t e s m a y require t h a t a p r o g r a m b e available to all qualified p a r t i c i p a n t s , or evalu a t i o n m a y b e g i n well a f t e r a p r o g r a m has started, elimin a t i n g any p o s s i b i l i t y o f rigorous c o n t r o l . While t h e r e is n o way to c o m p l e t e l y e l i m i n a t e t h r e a t s to i n t e r n a l validity in such s i t u a t i o n s , q u a s i - e x p e r i m e n t a l designs have b e e n d e v e l o p e d (Campbell, 1974), such as " m a t c h e d p a i r s " or " e x t e n d e d time series," t h a t m i n i m i z e e x o g e n o u s effects a n d p r o v i d e a clear u n d e r s t a n d i n g o f sources o f error. T h e e x p e r i m e n t a l m o d e l , h o w e v e r , r e m a i n s t h e ideal; non-exp e r i m e n t a l " c a s e s t u d i e s " or " a f t e r - o n l y c o m p a r i s o n s , " are seen as last resorts in w h i c h cause a n d e f f e c t is diffic u l t to isolate (Weiss, 1972). Y e t the p r o b l e m s f a c e d b y s u m m a t i v e e v a l u a t i o n trans-
CONTEXTUAL
c e n d issues o f research design a n d h i g h l i g h t the l i m i t a t i o n s of an e x p e r i m e n t a l m o d e l ' s u n d e r l y i n g a s s u m p t i o n s . Summ a t i v e e v a l u a t i o n s t r e a t a c t i o n p r o g r a m s as if t h e y were l a b o r a t o r i e s o f social c h a n g e in w h i c h specific t r e a t m e n t s are always e x p e c t e d to yield p a r t i c u l a r results. B u t social a c t i o n p r o g r a m s are n o t l a b o r a t o r y e x p e r i m e n t s ; goals are n o t always distinct, b u t are o f t e n m u l t i p l e a n d c o n t r a d i c t o r y ; t r e a t m e n t s are n o t single o c c u r r e n c e s , b u t c o n t i n u ing processes; results n e e d n o t be easily m e a s u r a b l e , b u t are o f t e n i n e x a c t a n d m u l t i - d i m e n s i o n a l . D i f f e r e n t program t r e a t m e n t s n o t o n l y p r o d u c e d i f f e r e n t degrees of success, b u t d i f f e r e n t k i n d s o f success, w h i c h m a y b e impossible to c o m p a r e q u a n t i t a t i v e l y . M a n y a c t i o n p r o g r a m s aim at q u a l i t a t i v e changes in i n s t i t u t i o n s and c o m m u n i t i e s ; n o t easily m e a s u r a b l e results, b u t r a t h e r altered c o n t e x t s for social life. T h e e x p e r i m e n t a l m o d e l simplifies a c t i o n p r o g r a m dyn a m i c s i n t o an explicit analysis o f cause a n d effect. Often, howe~er, p r o g r a m o u t c o m e s c a n n o t b e u n d e r s t o o d without considering the procedures through which treatments are applied, the m e a n i n g o f t r e a t m e n t for p a r t i c i p a n t s , a n d the social a n d c u l t u r a l c o n t e x t in w h i c h a c t i o n is embedded. Alternative evaluation approaches which consider the c o m p l e x i t y of such p r o g r a m settings do n o t r e p r e s e n t w a t e r e d - d o w n e x p e r i m e n t a l studies, b u t a t t e m p t s to und e r s t a n d m u l t i - d i m e n s i o n a l causes a n d effects t h r o u g h holistic analyses o f p r o g r a m c o n t e x t .
EVALUATION
C o n t e x t u a l e v a l u a t i o n treats a c t i o n p r o g r a m s as o n g o i n g social realities b y d i r e c t l y s t u d y i n g t h e i r e v e r y d a y activities. Goals, t r e a t m e n t s , a n d results t h e r e f o r e evolve f r o m c o n t i n u t i n g i n t e r a c t i o n s a m o n g p r o g r a m p a r t i c i p a n t s . By u n d e r s t a n d i n g these processes, a c o n t e x t u a l e v a l u a t i o n att e m p t s to explain h o w a p r o g r a m has d e v e l o p e d , w h a t it does, a n d h o w it can b e altered. C o n t e x t u a l e v a l u a t i o n o f t e n p r o v i d e s a basis for later e x p e r i m e n t a l assessment b y explicitly d e l i n e a t i n g the m e a s u r a b l e goals a n d results n e e d e d for logical inference. S o m e p r o g r a m s , h o w e v e r , m a y n e v e r b y suitable for exp e r i m e n t a l s t u d y . T h e i r o p e r a t i o n s involve so m a n y m u t u ally i n t e r a c t i n g variables t h a t causality c a n n o t b e separ a t e d f r o m t h e specific s e t t i n g in w h i c h results occur. In s u c h situations, o n l y c o n t e x t u a l u n d e r s t a n d i n g can provide m e a n i n g f u l p r o g r a m a s s e s s m e n t a n d p l a n n i n g . Several existing e v a l u a t i o n a p p r o a c h e s are implicitly b a s e d on a c o n t e x t u a l m o d e l . S y s t e m s e v a l u 9 t i o n (Etzioni, 1960; S c h u l b e r g & Baker, 1 9 6 8 ; S u c h m a n , 1970), for example, focuses o n t h e o r g a n i z a t i o n a l s e t t i n g m w h i c h program activities d e v e l o p a n d u n f o l d , c o n s i d e r i n g the syst e m i c i n t e r p l a y a m o n g o r g a n i z a t i o n a l e n v i r o n m e n t , program s t r u c t u r e a n d t r e a t m e n t i m p l e m e n t a t i o n . E t h n o graphic e v a l u a t i o n ( C e n t e r for New Schools, 1974) conc e n t r a t e s o n social a n d c u l t u r a l milieu a n d the way in w h i c h p r o g r a m p a r t i c i p a n t s develop d i f f e r e n t p e r c e p t i o n s and understandings of program operations and outcomes. Clinical e v a l u a t i o n (Glazer & Back, 1972, 1973) focuses o n d a y - t o - d a y i m p l e m e n t a t i o n , viewing t r e a t m e n t as a
MODELS
m u l t i - d i m e n s i o n a l process with c h a n g i n g o u t c o m e s over time. Political e v a l u a t i o n , o n the o t h e r h a n d , considers c o m p e t i t i v e r e l a t i o n s h i p s in p r o g r a m a d m i n i s t r a t i o n a n d implementation the process t h r o u g h w h i c h goals, procedures a n d e x p e c t a t i o n s are d e t e r m i n e d (Krause & H o w a r d , 1975). A t t e m p t s have also b e e n m a d e t o f o r m a l i z e particular m e t h o d o l o g i e s , such as adversary e v a l u a t i o n (Kourilsky, 1973) a n d subjective values m e a s u r e m e n t ( G u t t e n tag, 1973). I n s t e a d o f isolating specific causes a n d effects, such c o n t e x t u a l e v a l u a t i o n s try t o c o m p r e h e n d larger systems a n d relationships. T h e y do n o t seek unicausal links a m o n g one d i m e n s i o n a l variables, b u t ask a m o r e basic q u e s t i o n : " W h a t is going on h e r e ? " ( R a p o p o r t , 1963, p. 1900). While an e x p e r i m e n t a l a p p r o a c h assumes t h a t a c t i o n prograins can test generalizable t h e o r e t i c a l p r o p o s i t i o n s , cont e x t u a l e v a l u a t i o n s are, first o f all, e x p l o r a t o r y a n d assume t h a t valid t h e o r e t i c a l q u e s t i o n s t h e m s e l v e s m u s t still be f o r m u l a t e d . A t h e o r e t i c a l a n d m e t h o d o l o g i c a l basis for c o n t e x t u a l e v a l u a t i o n s c a n b e f o u n d in F o s t e r ' s ( 1 9 6 9 ) m o r e general discussion of d i r e c t e d social change. T o u n d e r s t a n d such change, F o s t e r argues, one m u s t c o n s i d e r t h e n a t u r e o f b o t h i n n o v a t i n g o r g a n i z a t i o n s a n d target groups as well as t h e s e t t i n g in w h i c h t h e y i n t e r a c t . T h e f o r m a t for research is a d e t a i l e d case s t u d y w h i c h a n a l y z e s t h e t o t a l s p e c t r u m of events associated w i t h p r o g r a m i m p l e m e n t a t i o n . W h a t is n e e d e d is: • . . an u n u s u a l l y b r o a d a n d flexible field research
E x p e r i m e n t a l and C o n t e x t u a l Models
m e t h o d o l o g y based on a holistic view of society and culture . . . . Where the technical, social, cultural, economic, psychological, and o t h e r p e r t i n e n t factors are nearly infinite, and usually n o t recognized in advance, this exploratory quality is e n o r m o u s l y advantageous. It vastly increases the investigator's chances of hitting u p o n the critical element in any situation. (Foster, 1969, p. 57-58) Foster's goal is to identify the implicit cultural " p r e m ises" of b o t h target groups and innovating organizations and to understand their implications for behavior in action program settings. Such a c o n t e x t u a l evaluation provides a detailed historical account of the factors affecting program operations and results. Research is inductive: issues are defined on the basis o f empirical findings, and answers to preliminary questions are the questions in the n e x t stage o f research. The first step in c o n t e x t u a l evaluation is a description o f w h a t actually happens in program treatment. Analysis then considers w h y such t r e a t m e n t occurs and h o w it relates to formal rules, informal goals, varying participant understandings, external pressures, and multiple program results. Finally, variations in program results are linked to variations in the t r e a t m e n t process and provide a basis for future program improvements. Contextual evaluation uses multiple m e t h o d s and data sources to gain a holistic perspective on program activities. Procedures can vary greatly from case to case and are best illustrated by a concrete example. In 1975, the National A c a d e m y of Sciences asked me to evaluate a federal program which developed coopera-
THE PURPOSES
OF EXPERIMENTAL
Experimental evaluations relate program treatments to program o u t c o m e s w i t h o u t directly examining causal processes. J u d g m e n t s about program effectiveness are made through controlled comparisons of results which assume uni-dimensional program goals and treatments. As a result, findings are clear and precise, but sometimes overly simplified. Contextual evaluations investigate causal relationships b e t w e e n program treatments and o u t c o m e s by directly examining the processes through which results are achieved. Program goals, treatments, and results are seen as multidimensional, and differences in program success are expected. Thus, c o n t e x t u a l evaluations provide in-depth (though sometimes overly subjective) findings. C o n t e x t u a l and experimental approaches not only utilize different kinds of data, but also provide alternative frameworks for understanding which address very different evaluation purposes. The quantitative comparisons of experimental evaluation provide decision makers with a precise basis for choices among program alternatives. Thus, experimental evaluation is especially useful for ex post facto decisions about the c o n t i n u a t i o n and expansion of pilot projects or o t h e r programs of a limited duration. Yet since experimental evaluations tend to ignore the de-
231
tive policy experiments with o t h e r government agencies. Nearly thirty projects had already been developed, which aimed at private sector e c o n o m i c and technological benefits. Their specific results were the subject o f internal review. My research, however, asked a far b r o a d e r question: To what e x t e n t were cooperative policy experiments a feasible strategy for change? I followed a c o n t e x t u a l approach. The first step was to analyze program d o c u m e n t s and identify formal goals and procedures. These were then c o m p a r e d with first-hand observations and discussions of everyday procedures. Differences were n o t e d and these became the subject of intensive interviews. It soon became clear that formal mandates were greatly m o d i f i e d by informal understandings about what kinds o f policy changes were possible and which w o u l d be m o s t beneficial to the program and to individual staff members. A quantitative analysis o f project characteristics d e t e r m i n e d the choice of eleven m o r e intensive case studies. These cases provided additional data on the informal objectives of agency clients, problems of project development, and factors affecting i m p l e m e n t a t i o n success. These materials were c o m b i n e d in an analysis o f program process which yielded specific r e c o m m e n d a t i o n s for changes in policy emphasis, organizational structure, and operating procedures (Britan, 1977, 1978, in press). Many of these r e c o m m e n d a t i o n s have already been implemented. The kinds of i n f o r m a t i o n and analysis provided by such a c o n t e x t u a l study are very different from an experimental appraisal of results. It is not, therefore, surprising that the two evaluation models are associated with rather different evaluation purposes.
AND CONTEXTUAL
EVALUATION
tails o f t r e a t m e n t i m p l e m e n t a t i o n , they provide little formative feedback for the d e v e l o p m e n t o f program improvements (Scriven, 1969). This is precisely the strong p o i n t o f a c o n t e x t u a l study. E x p e r i m e n t a l and c o n t e x t u a l approaches are o f t e n useful c o m p l e m e n t s . C o n t e x t u a l research can define measurable results for later experimental assessment, just as exp e r i m e n t a t i o n can test specific propositions within a broader c o n t e x t u a l framework. The difference b e t w e e n the two approaches, however, is far m o r e than the difference b e t w e e n qualitative and quantitative measurement. Experimental proofs can be inferred as easily on the basis o f qualitative distinctions - - w h e t h e r an effect occurs at all - as quantitative ones. At the same time, c o n t e x t u a l studies measure variables and results whenever appropriate. The approaches differ m o r e p r o f o u n d l y in their assumptions of h o w program dynamics and results can best be understood. The purpose of an evaluation, and the appropriateness of these alternative evaluation models, depends greatly on the nature of the program being considered. Next, a typology of program characteristics is developed which provides preliminary guidelines for the choice of evaluation strategies.
232
G E R A L D M. B R I T A N
EVALUATION
MODELS
AND
A l t h o u g h all a c t i o n p r o g r a m s have a similar logic in facilit a t i n g social change, t h e i r p a r t i c u l a r c h a r a c t e r i s t i c s vary greatly. A n e w e d u c a t i o n a l p r o g r a m , for e x a m p l e , c o u l d involve a single change in an o n g o i n g c u r r i c u l u m or the e s t a b l i s h m e n t of an entire a l t e r n a t i v e school. It is n o t very surprising t h a t e v a l u a t i o n s o f these t w o p r o g r a m s w o u l d b e very different. The choice of contextual and experimental evaluation strategies c a n be l i n k e d to an u n d e r s t a n d i n g o f such program differences. A few basic p r o g r a m c h a r a c t e r i s t i c s seem especially crucial: specificity of p r o g r a m goals, b r e a d t h o f p r o g r a m t r e a t m e n t , clarity of u n d e r l y i n g t h e o r y , a n d scope of p r o g r a m results ( S u c h m a n , 1 9 7 0 ; T h o m p s o n , 1967; Weiss, 1972; Weiss & Rein, 1974).
Program Goals Program goals are the i n t e n d e d results of p r o g r a m treatm e n t . T h e y m a y b e e x t r e m e l y specific, as in a medical ino c u l a t i o n p r o g r a m , or t h e y m a y b e diffuse, as in a h e a l t h p r o g r a m t h a t aims at i m p r o v i n g the general well-being o f a target c o m m u n i t y . In a d d i t i o n , p r o g r a m goals m a y be singular or m u l t i p l e , f o r m a l l y s t a t e d or i n f o r m a l l y u n d e r s t o o d , agreed u p o n or controversial.
Program Treatment and Program T h e o r y P r o g r a m t r e a t m e n t s can be single i n p u t s (requiring reckless drivers to a t t e n d a special l e c t u r e ) or m u l t i p l e i n p u t s (a m a n y - f a c e t e d driver safety p r o g r a m ) ; t r e a t m e n t s can o c c u r at o n e p o i n t in time or t h e y can involve an o n g o i n g p r o g r a m ; t r e a t m e n t s can address a single p r o b l e m or a n u m b e r o f d i f f e r e n t ones. In principle, p r o g r a m t r e a t m e n t is b a s e d on u n d e r l y i n g theory an u n d e r s t a n d i n g o f the cause a n d e f f e c t relat i o n s h i p b e t w e e n p r o g r a m t r e a t m e n t a n d the a c h i e v e m e n t of p r o g r a m goals, t"ew p r o g r a m s , h o w e v e r , have a clear a n d precise t h e o r e t i c a l basis a n d t h e r e f o r e p r o g r a m treatm e n t s o f t e n involve m u l t i p l e , fuzzy, or even c o n t r a d i c t o r y u n d e r s t a n d i n g s of t h e p r o b l e m to b e solved. Program Results A l t h o u g h p r o g r a m results are i n t e n d e d to fulfill p r o g r a m goals, actual results m a y vary f r o m t h e e x p e c t e d a n d una n t i c i p a t e d results m a y b e far m o r e i m p o r t a n t t h a n int e n d e d ones. While s o m e p r o g r a m results, such as the t e a c h i n g of m a t h e m a t i c a l skills, m a y be specific a n d m e a s u r a b l e , others, such as i m p r o v e m e n t s in the quality of life, are m o r e diffuse.
Other Program Characteristics O t h e r i m p o r t a n t p r o g r a m c h a r a c t e r i s t i c s m i g h t also be identified. A p r o g r a m ' s o r g a n i z a t i o n a l locus as p a r t o f an e s t a b l i s h e d agency or an i n d e p e n d e n t e n t i t y is o f t e n an i m p o r t a n t c o n s i d e r a t i o n . Program scale w h e t h e r activities o c c u r at an i n s t i t u t i o n a l , c o m m u n i t y , or n a t i o n a l level c o u l d also b e c o n s i d e r e d . R a t h e r t h a n c o m p l i c a t e the issue at this stage, h o w e v e r , we will simply assume t h a t such factors are s u b s u m e d b y p r o g r a m goala, p r o g r a m results, a n d p r o g r a m t h e o r y a n d t r e a t m e n t .
PROGRAM
CHARACTERISTICS
Program T y p e s T h e s e p r o g r a m c h a r a c t e r i s t i c s can b e d i c h o t o m i z e d in t e r m s o f n o m i n a l d i s t i n c t i o n s a l o n g three m a j o r d i m e n sions. Thus, p r o g r a m goals can be n a r r o w or b r o a d , program results can b e specific or diffuse, and p r o g r a m t h e o r y can b e clear or fuzzy (simple or c o m p l i c a t e d treatm e n t s are seen to reflect t h e o r y ) . A total o f eight differe n t p r o g r a m t y p e s are possible. First we will c o n s i d e r the t w o e x t r e m e cases, w h e r e t h e c h o i c e b e t w e e n e x p e r i m e n tal a n d c o n t e x t u a l e v a l u a t i o n is clear. 1. Narrow Goals, Clear Theory, Specific Results. This t y p e o f p r o g r a m nearly a p p r o x i m a t e s t h e l a b o r a t o r y ideal a n d is, t h e r e f o r e , the m o s t a p p r o p r i a t e for e x p e r i m e n t a l assessment. For e x a m p l e , w h e n a p u b l i c h e a l t h p r o g r a m is testing the efficacy of a n e w vaccine, t r e a t m e n t can be una m b i g u o u s l y l i n k e d to o u t c o m e a n d a precise j u d g m e n t of effectiveness can b e made. Such t r e a t m e n t s are b a s e d o n u n i - d i m e n s i o n a l cause a n d e f f e c t a s s u m p t i o n s . While social p r o g r a m s are rarely this clear, w h e n t r e a t m e n t is simple a n d results precise, e x p e r i m e n t a l e v a l u a t i o n is a p p r o p r i a t e . S u p p l e m e n t a r y c o n t e x t u a l research m i g h t also b e used to verify t h r e a t m e n t o c c u r r e n c e , elicit a l t e r n a t i v e goals, or assess u n i n t e n d e d results. 2. Broad Goals, Fuzzy Theory, Diffuse Results. Focusing o n discrete causes a n d effects m a k e s little sense w h e n program o p e r a t i o n is so p o o r l y u n d e r s t o o d . W h e n evaluating a c o m m u n i t y m e n t a l h e a l t h c e n t e r , for e x a m p l e , a res e a r c h e r faces m u t l i p l e goals, diverse u n d e r s t a n d i n g s , varied t r e a t m e n t s , and results t h a t range f r o m changes in individual p s y c h o l o g y to overall effects on e n t i r e neighb o r h o o d s . A d e t a i l e d e x a m i n a t i o n of p r o g r a m c o n t e x t explains hove these a n d o t h e r factors i n t e r r e l a t e in an ongoing p r o g r a m process. This w o u l d not, o f course, preclude the use of later e x p e r i m e n t a l validation, if the clear d e f i n i t i o n s n e e d e d for logical i n f e r e n c e e v e n t u a l l y emerge. Most a c t i o n p r o g r a m s fall s o m e w h e r e b e t w e e n these e x t r e m e s o f specificity a n d diffuseness, m a k i n g the c h o i c e o f an assessment strategy m o r e c o m p l i c a t e d . E i t h e r an exp e r i m e n t a l or c o n t e x t u a l e v a l u a t i o n can o f t e n be applied, b u t each involves d i s t i n c t a d v a n t a g e s and disadvantages.
3. Broad Goals, Clear Theory, Specific Results. S c h o o l b r e a k f a s t p r o g r a m s are p r o v i d e d to p r o m o t e b r o a d changes in h e a l t h , e m o t i o n a l a d j u s t m e n t , a n d a c a d e m i c p e r f o r m a n c e o f s t u d e n t s . Before e x p e r i m e n t a l research can p r o c e e d , these b r o a d f o r m a l a n d i n f o r m a l goals m u s t be b e t t e r specified. Once this is d o n e , actual results, such as d e c l i n i n g illness rates, can b e e x p e r i m e n t a l l y verified. T h e reasons w h y some e x p e c t e d effects do n o t o c c u r w o u l d , h o w e v e r , b e a s u b j e c t for c o n t e x t u a l study.
4. Narrow Goals, Clear Theory, Diffuse Results. A comp e n s a t o r y m a t h e m a t i c s p r o g r a m is designed to i m p r o v e s t u d e n t skills t h r o u g h r e m e d i a l training. T h e effectiveness of this t r a i n i n g can be easily t e s t e d t h r o u g h an experim e n t a l c o m p a r i s o n of s t a n d a r d i z e d test scores. Such an a p p r o a c h , h o w e v e r , m a y ignore equally i m p o r t a n t supple-
E x p e r i m e n t a l a n d C o n t e x t u a l Models
233
m e n t a r y results -- i m p r o v e m e n t s in o t h e r subject areas, e n h a n c e d s t u d e n t a d j u s t m e n t , or altered career interests. Moreover, if the results of c o m p e n s a t o r y t r a i n i n g v a r y s u b s t a n t i a l l y b e t w e e n classrooms or schools (as is t h e case in at least o n e m a j o r M i d w e s t e r n city), c o n t e x t u a l research can find reasons in t h e v a r i a t i o n in p r o g r a m implementation.
p e r i m e n t a l e v a l u a t i o n w o u l d tell us little a b o u t w h y t h e y were achieved or h o w t h e y c o u l d b e i m p r o v e d . Such an e v a l u a t i o n w o u l d also ignore o t h e r p o t e n t i a l l y i m p o r t a n t p r o g r a m effects, such as i m p r o v e m e n t s in p r e - r e t i r e m e n t career s a t i s f a c t i o n or p o s t - r e t i r e m e n t a d j u s t m e n t . A cont e x t u a l s t u d y seems w a r r a n t e d to d e t e r m i n e w h a t the club actually does a n d w h a t effects it has.
5. Narrow Goals, Fuzzy Theory, Specific Results. If goals
7o Broad Goals, Fuzzy Theory, Specific Results. A comm u n i t y d e v e l o p m e n t p r o g r a m can fulfill a m u l t i t u d e o f purposes. W i t h o u t a c o n t e x t u a l s t u d y o f these goals a n d t h e m e t h o d s t h r o u g h w h i c h t h e y are achieved, t h e r e are few guidelines for e x p e r i m e n t a l assessment, even if actual results are quite specific. C o n t e x t u a l e v a l u a t i o n m a y provide a basis for l a t e r e x p e r i m e n t a l analysis, b u t it also adds d a t a t h a t is d i r e c t l y r e l e v a n t to f u t u r e p r o g r a m p l a n n i n g .
are n a r r o w a n d results specific, e x p e r i m e n t a l i n f e r e n c e can p r o v i d e a clear basis for j u d g i n g p r o g r a m effectiveness. It c a n n o t , h o w e v e r , clarify the reasons for p r o g r a m failure or success. This k n o w l e d g e is especially i m p o r t a n t if p r o g r a m results are i n c o n s i s t e n t . F o r e x a m p l e , a m a j o r A m e r i c a n c o m p a n y r e c e n t l y i n t r o d u c e d n e w w o r d - p r o c e s s i n g techn o l o g y a i m e d at e n h a n c i n g business office efficiency. In a b o u t 80% o f the cases, office efficiency did in fact imp r o v e ; b u t in 20% it actually declined. T h e c o m p a n y , t h e r e f o r e , c o m m i s s i o n e d a c o n t e x t u a l s t u d y to e x a m i n e t h e i m p a c t o f its e q u i p m e n t o n office r o u t i n e s a n d to learn w h y efficiency s o m e t i m e s d r o p p e d . These findings will b e i n c o r p o r a t e d in i m p r o v e d t e c h n o l o g y in the future.
6. Narrow Goals, Fuzzy Theory, Diffuse Results. A comp a n y m a y i n a u g u r a t e a p r e - r e t i r e m e n t club for the relatively n a r r o w p u r p o s e o f i m p r o v i n g the financial p l a n n i n g o f e m p l o y e e s . Such a p r o g r a m m i g h t b e e x p e r i m e n t a l l y e v a l u a t e d simply t h r o u g h c o m p a r i s o n s o f the financial s t a t u s o f retirees. Yet, if results were positive, such an ex-
8. Broad Goals, Clear Theory, Diffuse Results. It seems u n l i k e l y t h a t m a n y p r o g r a m s w i t h clearly d e f i n e d t h e o r y a n d t r e a t m e n t will also have b r o a d goals and diffuse results. Such situations, however, w o u l d call for c o n t e x t u a l appraisal a n d a clearer u n d e r s t a n d i n g o f goals a n d results before experimental study could proceed. In general, e x p e r i m e n t a l e v a l u a t i o n clearly judges p r o g r a m results b u t adds l i t d e u n d e r s t a n d i n g o f causal process, while c o n t e x t u a l research elucidates causal processes b u t provides little basis for explicit j u d g m e n t . Dep e n d i n g o n e v a l u a t i o n p u r p o s e s , the t w o a p p r o a c h e s can o f t e n b e usefully c o m b i n e d .
DISCUSSION A l t h o u g h applied social scientists are faced w i t h a growing n u m b e r of c o m p e t i n g a p p r o a c h e s to p r o g r a m evaluation, few c h o i c e guidelines have b e e n developed. This p a p e r has argued t h a t t h e a p p a r e n t variety o f e v a l u a t i o n strategies r e p r e s e n t s t w o u n d e r l y i n g m o d e l s t h a t are b a s e d o n d i f f e r e n t epistemological a s s u m p t i o n s . E x p e r i m e n t a l evalu a t i o n s isolate discrete causes a n d effects t h r o u g h controlled c o m p a r i s o n s of specific p r o g r a m results. T h e goal is an explicit j u d g m e n t of p r o g r a m success. C o n t e x t u a l evaluation, on the o t h e r h a n d , tries t o u n d e r s t a n d the c o m p l e x i t y o f a c t i o n p r o g r a m goals, t r e a t m e n t s , a n d results as it u n f o l d s in actual p r o g r a m processes. J u d g m e n t s o f success c a n n o t be e x p r e s s e d along a single dim e n s i o n a n d t h e view of d i f f e r e n t individuals a n d institut i o n s can b e e x p e c t e d to vary. In a practical sense, e x p e r i m e n t a l and c o n t e x t u a l evaluations are each a p p r o p r i a t e to d i f f e r e n t p r o g r a m types a n d e v a l u a t i o n purposes. A t y p o l o g y of p r o g r a m characteristics b a s e d o n t h e o r e t i c a l clarity, t r e a t m e n t c o m p l e x i t y , result specificity a n d goal b r e a d t h was develo p e d a n d r e l a t e d to t h e c h o i c e of e v a l u a t i o n alternatives.
It was argued t h a t while n o single evaluative strategy is always a p p r o p r i a t e , c o n t e x t u a l m o d e l s have at least as m u c h validity as m o r e c o m m o n l y used e x p e r i m e n t a l approaches. A l t h o u g h o u r t y p o l o g y o f p r o g r a m characteristics is c o m p r e h e n s i v e , n o t all p r o g r a m t y p e s are equally prevalent. Indeed, n a r r o w l y d e f i n e d p r o g r a m s suitable for p u r e l y e x p e r i m e n t a l e v a l u a t i o n seem to b e the e x c e p t i o n , n o t t h e rule. A c t i o n p r o g r a m s , a f t e r all, are a t t e m p t s to direct social change, a n d p r e s e n t u n d e r s t a n d i n g s of social d y n a m i c s are generally q u i t e imprecise. Most existing p r o g r a m s do have s o m e w h a t f u z z y t h e o r y , fairly b r o a d goals, a n d r a t h e r diffuse results, m a k i n g an u n d e r s t a n d i n g of p r o g r a m c o n t e x t essential. O u r analysis, o f course, is only a first step a n d m o r e specific guidelines are n e e d e d . E x p e r i m e n t a l testing is certainly a w o r t h y l a b o r a t o r y ideal. Yet it is o f t e n o f little help in evaluating a c t i o n p r o g r a m s as t h e y are, a n d as t h e y m u s t be if they are to have a significant i m p a c t o n the social p r o b l e m s we face.
REFERENCES BENNETT, J. W. Anticipation, adaptation, and the concept of culture in anthropology. Science, 1976, 192, 847-853.
BRITAN, G. M. Public policy and innovation: An ethnographic evaluation of the Experimental Technology Incentives Program. Washington: National Academy of Sciences, 1977.
234
GERALD
M. B R I T A N
BRITAN, G. M. The place of anthropology in program evaluation.
KOURILSKY, M. An adversary model for educational evaluation.
Anthropological Quarterly, 1978, 51 (2), 119-128.
Evaluation Comment, 1973, 2, 3-6.
BRITAN, G. M. Evaluating an a t t e m p t to institutionalize change in the federal bureaucracy. In press. CAMPBELL, D. T. From description to experimentation. In C. W. Harris (Ed.), Problems in measuring change. Madison: University of Wisconsin Press, 1963. CAMPBELL, D. T. Quasi-experimental designs. In H. W. Riecken & R. F. Boruch (Eds.), Social experimentation. New York: Academic Press, 1974. CAMPBELL, D. T., & STANLEY, S. C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1966. CENTER FOR NEW SCHOOLS, The use o f ethnography for educational evaluation. Chicago: CNS, 1974. ETZIONI, A. Two approaches to organizational analysis: A critique and a suggestion. Administrative Science Quarterly, 1960, 5, 2 5 7 - 2 5 8 . FOSTER, (;. Applied anthropology. Boston: Little, Brown, 1969. G L A Z E R , E. M., & BACK, T. E. A clinical approach to program evaluation. Evaluation, 1972, 1, 54-60. GLAZER, E. M., & BACK, T. E. A look at participant observation. Evaluation, 1973, 1(3), 46-49. GUTTENTAG, M. Subjectivity and its use in evaluation research.
Evaluation, 1973, 1(2), 60-65.
KRAUSE, M. S., & HOWARD, K. I. Program evaluation in the public interest: A new research methodology. In Readings in program evaluation. Evanston: Northwestern University, 1975. RAPOPORT, R. N. Social anthropology and mental health. In En-
cyclopedia o f mental health. New York: Franklin Watts, 1963. RIECKEN, H. W., & BORUCH, R. F. Social experimentation. New York: Academic Press, 1974. SCHULBERG, H. C., & BAKER, F. Program evaluation models and the implementation of research findings. American Journal o f Public Health, 1968, 58, 1248-1255. SCRIVEN, M. The m e t h o d o l o g y of evaluation. In R. Tyler, R. M. Gagner, & M. Scriven (Eds.), Perspectives o f curriculum development. Chicago: Rand McNally, 1969. SUCHMAN, E. A. Action for what? A critique of evaluative research. In R. O'Toole (Ed.), The organization, management, and tactics o f social research. Cambridge: S c h e n k m a n Publishing Company, 1970. THOMPSON, J. D. Organizations in action. New York: McGrawHill, 1967. WEISS, C. Evaluation research. Englewood Cliffs: Prentice-Hall, 1972. WEISS, R. S. Alternative approaches to the study of complex situations. Human Organization, 1966, 25, 198-206. WEISS, R. S., & REIN, M. The evaluation of broad-aim programs: Difficulties in experimental design and an alternative. In C. Weiss (Ed.), Readings in program evaluation. Englewood Cliffs: PrenticeHall, 1974.