- Email: [email protected]
Studying the local use of evaluation: A discussion of theoretical issues and an empirical study
Studios in Educational Evaluation. Vol. 8, pp. 175-183
0191-491X/82/020175-09504.5010 Copyright©1982PergamonPressLtd.
Printedin GreatBritain.All rights reserved
S T U D Y I N G THE LOCAL USE OF EVALUATION: A DISCUSSION THEORETICAL ISSUES A N D A N EMPIRICAL S T U D Y
OF
Jean A. King 7-ulane University, New Orleans, Louisiana, USA
IntJrOd~ L i k e t h e Eloi a n d t h e M o r l o c k s in H. G. Wells' T i m e M a c h i n e , e v a l u a t o r s a n d t h e i r c l i e n t s f o r t h e m o s t p a r t l i v e in s e p a r a t e w o r l d s a n d m a k e o n l y o c c a s i o n a l contacts. Though the comparison with Wells' future obviously exaggerates reality, t h e r e is e n o u g h t r u t h in i t t o m a k e an e v a l u a t o r u n c o m f o r t a b l e . Wells presents a distressing f o r e c a s t in his v i s i o n o f t h e EIoi, d e s c e n d a n t s o f t h e British aristocracy who share a world with the Morlocks, the ever hungry descendants of the working class. In t h a t f a r d i s t a n t t i m e , t h e t w o c o m m u n i t i e s live harmoniously. T h e f r a i l EIoi l i v e a b o v e t h e g r o u n d a n d h a v e no u n m e t needs, either felt or unfelt, while the hirsute Morlocks live below the ground and k e e p t h e h u g e m a c h i n e r y t h a t is t h e i r l i v e l i h o o d g r e a s e d a n d t u r n i n g . T h e r e is a catch, of course, and a deadly one: at n i g h t t h e M o r l o c k s c l i m b f r o m t h e i r subterranean l a i r s t o f e e d o n t h e c h i l d l i k e a n d d e f e n s e l e s s EIoi. What a t f i r s t appears to the Time Traveller a p a r a d i s e r e v e a l s i t s e l f f i n a l l y to be a c o n t i n u a l nightmare. To administrators and other practitioners, e v a l u a t o r s m a y seem l i k e t h e E I o i - naive, spoiled, out of shape, and used to getting their way-while they, like the M o r l o c k s , t o i l in t h e s t i f l i n g r e a l i t y a n d o v e r p o w e r i n g c o n s t r a i n t s o f t h e i r school settings. If i t is r e m e m b e r e d t h a t t h e M o r l o c k s d e v o u r t h e EIoi o n e a t a t i m e , t h e i m a g e o f t h e h a p l e s s e v a l u a t o r in t h e t h r o e s o f a c o n t r o v e r s i a l s t u d y can s e r v e as a r e m i n d e r t h a t t h e j o b o f e v a l u a t o r is n o t w i t h o u t its d a n g e r s . To e v a l u a t o r s , on t h e o t h e r h a n d , d e c i s i o n - m a k e r s a n d o t h e r c l i e n t s m a y seem a b i t l i k e t h e EIoi. U n a p p r e c i a t i v e o r e v e n f r i g h t e n e d o f t h e i m p r e s s i v e social s c i e n c e m a c h i n e r y t h e e v a l u a t o r c o n t r o l s , t h e y l i v e in a w o r l d d i s t i n c t f r o m t h e r i g o r s o f multiple regression and factor analysis. Unless dragged, the rarely enter the evaluator's world, preferring the bliss of ignorance to so-called "objective" realities, a n d y e t t h e y d o n o t h e s i t a t e t o use t h e e m p i r i c i s t M o r l o c k p r o d u c t s when these suit their needs. Typically, evaluators and their clients do inhabit separate worlds, seeing life through different conceptual glasses. Caplan's "two communities" theory (see, f o r e x a m p l e , C a p l a n e t a l . , 1975) s u g g e s t s t h a t an e v a l u a t o r ' s c o n c e r n f o r t e c h nically correct procedures may conflict with a program staff's implementation plans o r w i t h a local d e c i s i o n m a k e r ' s i n f o r m a t i o n n e e d s . While t h e e v a l u a t o r sees a local p r o g r a m as a p i e c e o f an e d u c a t i o n a l e x p e r i m e n t , an a d m i n i s t r a t o r may i n s t e a d see i t as p r o v i d i n g e s s e n t i a l s e r v i c e s t o s t u d e n t s . Years ago, Wilensky
175
176
J. A. King
and L e b e a u x n o t e d t h a t " w h a t t h e social s c i e n t i s t t h i n k s of as ' o b j e c t i v e i n v e s t i g a t i o n ' t h e p r a c t i t i o n e r o f t e n t a k e s as ' h o s t i l e a t t a c k ' " (1958, p. 2 0 ) . C e r t a i n e m p i r i c a l e v i d e n c e d o c u m e n t s t h e s e p a r a t i o n of e v a l u a t o r and u s e r . In a s u r v e y of school a d m i n i s t r a t o r s , K i n g and T h o m p s o n f o u n d o n l y 2& who had d a i l y c o n t a c t w i t h e v a l u a t o r s , w h i l e t h e m a j o r i t y r e p o r t e d s p e a k i n g to e v a l u a t o r s less t h a n once e v e r y t w o w e e k s (1981, p. 9 ) . L y o n et a l . ' s r e c e n t s u r v e y of LEA e v a l u a t i o n u n i t heads f o u n d t h a t m a n y h a v e had l i t t l e e x p e r i e n c e at t h e school l e v e l , p o t e n t i a l l y m a k i n g t h e i r a w a r e n e s s of t h a t s e t t i n g less t h a n d e s i r a b l e ; " n o t o n l y h a v e 42& of them n o t t a u g h t , b u t 70& h a v e not r u n a s c h o o l " ( L y o n et a l . , 1978, p. 6 6 ) . In s t u d y i n g t h e local uses of T i t l e I e v a l u a t i o n s , D a v i d f o u n d t h a t f o r m a n y t e a c h e r s e v a l u a t i o n is s y n o n y m o u s w i t h a c h i e v e m e n t t e s t i n g , s u g g e s t i n g t h a t u s e r s ' p e r c e p t i o n s of e v a l u a t i o n c l e a r l y d i f f e r f r o m e v a l u a t o r s ' ( D a v i d , 1981). A further s t u d y s u g g e s t i n g t h e d i f f e r i n g p e r s p e c t i v e s of e v a l u a t o r s and t h e i r c l i e n t s is t h a t of K e n n e d y , A p t i n g , and N e u m a n n , who f o u n d t h a t w h i l e c l i e n t s r a r e l y q u e s t i o n e d t h e t e c h n i c a l q u a l i t y of e v a l u a t o r s ' w o r k , they frequently q u e s t i o n its " e d u c a t i o n a l " q u a l i t y ( 1 9 8 0 ) . I have purposely exaggerated the separation between e v a l u a t o r s and users b e c a u s e - - l i k e it o r n o t - - w e m u s t acknowledge t h e f a c t t h a t t h e o n l y place o u r w o r k can be a success is in t h e u s e r s ~ w o r l d . If o u r r e s u l t s a r e n o t used t h e r e , t h e y a r e n o t u s e d at a l l ; and if t h e y a r e n o t used at a l l , s o m e o n e - - p r o b a b l y u s - - h a s w a s t e d an e n o r m o u s a m o u n t of e n e r g y in an era of s e v e r e e n e r g y s h o r t ages. My message r e d u c e s at last to t h i s simple t h o u g h t : T h e w o r l d of t h e e v a l u a t o r m u s t become n o t l i k e t h e u s e r s ' w o r l d , b u t m u s t be t h e u s e r s ' w o r l d . We m u s t n o t o n l y e n t e r t h e w o r l d of o u r local c l i e n t s , we need to r e n t o f f i c e space. T h i s r e a s o n i n g p r o v i d e s t h e t h e o r e t i c a l r a t i o n a l e f o r t h e s t u d y I will d e s c r i b e in t h i s p a p e r . T h e p u r p o s e of t h e p r o j e c t is to s t u d y t h e p r o c e s s of e v a l u a t i o n use at t h e local l e v e l , h e l p i n g us to u n d e r s t a n d b e t t e r how i n f o r m a t i o n is c r e a t e d at t h e local level and to d i s c o v e r t h o s e f a c t o r s a f f e c t i n g w h a t h a p p e n s to r e s u l t s in t h e s e t t i n g s w h e r e t h e y c o u n t . T h e p a p e r has t h r e e s e c t i o n s : first, a d i s c u s s i o n of t h e c o n c e p t of e v a l u a t i o n u t i l i z a t i o n / u s e and t h e r e l a t e c o n c e p t of u s e r r a t i o n a l i t y ; a n d s e c o n d , a d e s c r i p t i o n of a n a t u r a l i s t i c s t u d y currently underway. B e f o r e c o n t i n u i n g w i t h t h e t e x t , h o w e v e r , I must p r e s e n t a c r i t i c a l d i g r e s s i o n . Several theorists--notably Weiss (1979) and C a p l a n ( 1 9 8 0 ) - - h a v e p o i n t e d to t h e d a n g e r of a s s u m i n g a u t o m a t i c a l l y t h a t t h e use of e v a l u a t i o n r e s u l t s is g o o d a n d t h e n o n - u s e n e c e s s a r i l y b a d , an a s s u m p t i o n f a i r l y common in t h e u t i l i z a t i o n l i t e r a ture. If, h o w e v e r , the results are flawed for methodological reasons or have been d o c t o r e d to s u i t an i n s t i t u t i o n ' s p o l i t i c s , a d e c i s i o n m a k e r may well be wise n o t to use t h e m . T h i s is n o t to s a y t h a t we m u s t w a i t u n t i l we a t t a i n e v a l u a t i o n n i r v a n a w h e r e e v a l u a t o r s a c r o s s t h e c o u n t r y c o l l e c t ideal d a t a , a n a l y z e them i m p e c c a b l y , a n d p r e s e n t them in h i g h l y a p p r o p r i a t e f o r m s b e f o r e we b e g i n to s t u d y t h e f a c t o r s t h a t a f f e c t use. It is c e r t a i n l y i m p o r t a n t t h a t t h e e v a l u a t i o n c o m m u n i t y c o n t i n u e to make its S t a n d a r d s e x p l i c i t , to b e g i n to p o l i c e its r a n k s , and to i n s i s t t h a t p o o r q u a l i t y e v a l u a t i o n s n o t be t o l e r a t e d , a l t h o u g h d o i n g so may not i n c r e a s e t h e use of r e s u l t s ( s e e , f o r e x a m p l e , P a t t o n et a l . , 1977, p. 151). B u t w h i l e we a r e d o i n g t h i s on one f r o n t , we must w o r k s i m u l t a n e o u s l y on a n o t h e r , i . e . , t r y i n g to u n d e r s t a n d w h a t we can do to i n c r e a s e t h e a p p r o p r i a t e use of e v a l u a t i o n r e s u l t s . We h a v e n ' t t h e l u x u r y of time to p u t o u r s h o p in order before our customers arrive. T h e y a r e h e r e , and some are e v e n e a g e r to see w h a t we h a v e to o f f e r . B y s t u d y i n g t h e c u r r e n t use of i n f o r m a t i o n at t h e local l e v e l , we may l e a r n how to i n c r e a s e o u r c h a n c e s t h a t r e s u l t s w o r t h u s i n g get used. T o p u t i t a n o t h e r w a y , some of us w i s e r , n o t e n t i r e l y d e f e n s e l e s s Eloi
Studying the Local Use of Evaluation
must recognize that share our breathing
the key to our survival space and to learn the
177
is t o w a l k a m o n g t h e b e i n g s w h o key to their spirit and ingenuity.
The Concepts of Utilization ( U s e ) and Rationality T h e c o n c e p t o f e v a l u a t i o n u t i l i z a t i o n has c h a n g e d s i g n i f i c a n t l y in t h e p a s t t e n years. Once upon a time, the evaluation community subscribed to a "big bang" theory of utilization (Andrews, 1979), assuming somewhat naively that merely p r o v i d i n g d e c i s i o n m a k e r s t h e r e s u l t s o f e v a l u a t i o n s t u d i e s w o u l d l e a d t o t h e use of those results. From this perspective, evaluators were frequently chagrined and even embarrassed to discover that the results they rigorously generated more o f t e n e n d e d u p c o l l e c t i n g d u s t on s o m e o n e ' s s h e l f t h a n a f f e c t i n g f u t u r e d e c i s i o n s . The literature abounds with gloomly perceptions of the "failure of educational evaluation" (see King, Thompson, & Pechman, t981, pp. 1-2, for a collection of such quotations). A t o n e p o i n t , R o b e r t S t a k e w e n t so f a r as t o w o n d e r w h e t h e r " e v a l u a t i o n is g o i n g t o c o n t r i b u t e m o r e t o t h e p r o b l e m s o f e d u c a t i o n o r m o r e to t h e s o l u t i o n s " ( 1 9 7 6 , p. 1 ) . T h i s p e r s p e c t i v e is w h a t A l k i n , D a i l l a k , a n d White call t h e " m a i n s t r e a m " v i e w p o i n t o f e v a l u a t i o n u t i l i z a t i o n , i . e . r t h e v i e w p o i n t t h a t "contends that evaluations seldom influence program decision makers and holds out l i t t l e h o p e t h a t e v a l u a t i o n w i l l e v e r b r e a k t h r o u g h t h e b a r r i e r s to r e a l i m p a c t on programs" (1979, p. 17). The results of empirical studies of utilization, however, have challenged the w o r s t - c a s e p e r c e p t i o n s o f t h i s m a i n s t r e a m p e r s p e c t i v e , a n d an " a l t e r n a t i v e " v i e w p o i n t has e m e r g e d in r e c e n t y e a r s ( A l k i n , Daillak, & White, 1979). Studies by Patton et al. (1977), Alkin, Daillak, a n d White ( 1 9 7 9 ) , Dickey (1979), and Kennedy, Apling, and Neumann (1980) suggest that evaluation results are used b y d e c i s i o n m a k e r s , " b u t n o t in t h e c l e a r - c u t a n d o r g a n i z a t i o n - s h a k i n g ways that social s c i e n t i s t s s o m e t i m e s b e l i e v e r e s e a r c h s h o u l d be u s e d " ( P a t t o n et a l . , 1977, p. 144). The current v i e w o f e v a l u a t i o n use as s u m m a r i z e d b y B r o w n a n d B r a s k a m p is t h a t effective utilization does n o t n e c e s s a r i l y mean t h a t a n y o f t h e recommendations are implemented or that there are any immedia t e l y a p p a r e n t d e c i s i o n s b a s e d on t h e i n f o r m a t i o n . ( 1 9 8 0 , p. 91) Rather, results frequently i n f l u e n c e u s e r s in i n d i r e c t o r g r a d u a l w a y s . The d i s c o u r a g i n g m a i n s t r e a m p i c t u r e o f e v a l u a t i o n use m a y h a v e s t e m m e d m o r e f r o m a m i s t a k e n e x p e c t a t i o n r e g a r d i n g t h e n a t u r e o f use t h a n f r o m t h e a c t u a l l e v e l o f use in t h e real w o r l d . Wise ( 1 9 7 8 ) has w r i t t e n t h a t If t h e r e is an e v a l u a t i o n utilization problem, i t is n o t t h a t decision-makers do n o t use t h e i n f o r m a t i o n t h e y r e c e i v e , i t is t h a t e v a l u a t o r s c a n n o t e a s i l y see t h e i r i n f o r m a t i o n b e i n g u s e d in the incrementalism of real-world decision-making. ( p . 24) O r , as Pogo so e l o q u e n t l y o n c e s a i d , "We h a v e met t h e e n e m y , a n d t h e y a r e u s . " No l o n g e r , t h e n , is i t s u f f i c i e n t t o l o o k f o r d i r e c t a n d r e a d i l y a p p a r e n t i n s t a n c e s of utilization. S u p p o r t e d b y e m p i r i c a l d a t a a n d common s e n s e , t h e a l t e r n a t i v e v i e w p o i n t has t a k e n t h e f i e l d . Given this viewpoint, e v e n t h e c o n c e p t u a l label n e e d s t o be c h a n g e d . To speak of utilization is t o s u g g e s t t h e m a i n s t r e a m v i e w a n d t o l i n g u i s t i c a l l y bias the search for cases. U t i l i z a t i o n was an a p p r o p r i a t e term for researchers with the mainstream view, but, as Weiss ( 1 9 7 9 ) w r i t e s , i t " e m b o d i e s an i n a p p r o p r i a t e imagery . . . b e c a u s e o f its o v e r t o n e s of instrumental episodic application. People don't utilize research the way that they utilize a hammer" (p. 2). For this reason, the alternative viewpoint suggests the need for substituting the term use, a word with few inappropriate connotations, for the term utilization.
178
J. A. King
Like the concept of utilization, t h e c o n c e p t o f local u s e r s ' r a t i o n a l i t y has also c h a n g e d in r e c e n t y e a r s . T h e m a i n s t r e a m v i e w p o i n t a s s u m e d a classic, b u t l i m i t e d n o t i o n o f r a t i o n a l b e h a v i o r , a n o t i o n i m p l i c i t in w h a t t call t h e r a t i o n a l m y t h of e v a l u a t i o n u s e . B r i e f l y s t a t e d , t h e m y t h goes l i k e t h i s : Harried decision-makers, over-worked b u t r a t i o n a l at h e a r t , a r e presented evaluative information t h a t is r e l e v a n t to d e c i s i o n s t h e y a r e a b o u t to m a k e . S i t t i n g c a l m l y at t h e i r d e s k s , t h e y consider the data, weigh [heir options, then make the "correct" decision, i.e., the choice supported by the data. The evaluator, b e a m i n g h a p p i l y b e h i n d t h e s c e n e s , t a k e s p r i d e in n u r turing such rational action. (King, Thompson, & Pechman, 1981, p. 4) E v a l u a t o r s no l o n g e r p r e t e n d that this scenario will always--or even ever--be played out. Weiss ( 1 9 7 9 ) p r e s e n t s e x a m p l e s w h e r e t h e " r a t i o n a l " c h o i c e f o r a d e c i s i o n m a k e r is u n c l e a r , f o r e x a m p l e , w h e n e v a l u a t i o n r e s u l t s a r e i n c o m p l e t e o r conflicting; t h e f r e q u e n t p r e s e n c e of such s i t u a t i o n s in t h e real w o r l d is u n q u e s t i o n a b l e a n d m a k e s t h e c a u s e - e f f e c t n o t i o n of e v a l u a t i o n use u n r e a l i s t i c . A l s o , as K i n g , T h o m p s o n , a n d Pechman n o t e , t h e m y t h i c a l v i e w d e n i e s t h e p r o c e s s a n d c o n t e x t o f real w o r l d d e c i s i o n m a k i n g ( 1 9 8 1 , p. 4 ) . C r o n b a c h e t al. w r i t e t h a t d e c i s i o n s as " f o r m a l c h o i c e ( s ) at a p a r t i c u l a r " t i m e b e t w e e n d i s c r e t e a l t e r n a t i v e s " rarely occur in e v a l u a t i o n settings (1980, p. 8 4 ) . Rather, in c o n t e x t s of pluralistic accommodation, decision makers "satisfice," i.e., find a course of a c t i o n t h a t is " g o o d e n o u g h " ( S i m o n , 1975, p. 2 0 4 ) ; a n d , o v e r a p e r i o d o f t i m e , decisions evolve. T h e " r a t i o n a l " d e c i s i o n in such e n v i r o n m e n t s is w h a t e v e r is likely to work, and "rationality" is a r e l a t i v e n o t i o n . If, for example, analysis s u g g e s t s t h a t n o t u s i n g an e v a l u a t i o n r e s u l t is a p p r o p r i a t e , then non-use may represent a highly rational "use" of evaluation (see King, Thompson, & Pechman, 1981, p p . 6 - t 2 ) . In d i s c u s s i n g w h a t h a p p e n s to e v a l u a t i o n r e s u l t s , S m i t h d i s t i n g u i s h e s between use, the conscious consideration and e m p l o y m e n t o f r e s u l t s , a n d i m p a c t , " a n y d i s c e r n a b l e a c t i o n s , e v e n t s , o r c h a n g e s in c o n d i t i o n s t h a t a r e d i r e c t l y i n f l u e n c e d b y t h e e v a l u a t i o n , its p r o c e s s e s , p r o d u c t s , o r f i n d i n g s " ( 1 9 8 0 , p. 2 5 ) . I t is n o t e n o u g h to o b s e r v e c h a n g e s r e s u l t i n g f r o m e v a l u a t i o n s . F o r an e x a m p l e to be l a b e l e d u s e , t h e b l a c k b o x m u s t be o p e n e d to r e v e a l s o m e o n e p u r p o s e l y d o i n g something with the results. T h i s d e f i n i t i o n of u s e , t h e n , has a p e r s o n a t t a c h e d . Smith's definition--the " c o n s c i o u s e m p l o y m e n t o f an e v a l u a t i o n ( i t s p r o c e s s e s , p r o d u c t s , o r r e s u l t s ) to a c h i e v e some d e s i r e d e n d o r i m p a c t " - - r e f l e c t s t h e p u r p o s e f u l n a t u r e of use ( 1 9 8 0 , p. 2 5 ) , a n d t h e c o n s c i o u s e l e m e n t is b o t h a n e c e s s a r y a n d s u f f i c i e n t c o n d i t i o n of use. T h e s e r i o u s c o n s i d e r a t i o n of e v a l u a t i o n r e s u l t s is also n e c e s s a r y to u s e , t h o u g h n o t s u f f i c i e n t ; o n e can i m a g i n e , f o r e x a m p l e , a d e c i s i o n m a k e r w h o r e a d s an e v a l u a t i o n r e p o r t , b u t is f o r c e d to m o v e on t o o t h e r c o n c e r n s b e f o r e b e i n g a b l e to a c t on t h e r e s u l t s in a n y w a y ( e v e n i f j u s t to i g n o r e t h e m c o n s c i o u s l y ) . Table 1 presents conceptual dimensions of evaluation use, integrating the earlier f r a m e w o r k s of A I k i n , D a i t t a k , a n d White ( 1 9 7 9 ) , Weiss ( 1 9 7 9 ) , a n d B r o w n ( 1 9 8 1 ) . T h e t a b l e is s t r u c t u r e d on t h e j o u r n a l i s t ' s "SW's a n d an H" ( w h o , w h y , w h a t , w h e r e , w h e n , a n d h o w ) a n d s u g g e s t s t w o i m p o r t a n t p o i n t s f o r s t u d y i n g local u s e . F i r s t , to see w h a t f a c t o r s a f f e c t u s e , we m u s t f o c u s on t h e u s e r and h i s / h e r response to the information presented. As P a t t o n et a l . ' s p e r s o n a l f a c t o r d e m o n strates (1977), the user's characteristics and purposes will in large part d e t e r m i n e to w h a t e x t e n t r e s u l t s w i l l be u s e d .
."How i m m e d i a t e the use?" ."How m u c h e f f e c t is required?"
Outcome
is
."How d i r e c t is derivation from the s t u d y ? "
."What is u s e d ? "
."By h o w m a n y p e o p l e ?
."By w h o m is it used?"
Weiss (1979, pp. 11-12) 3)
Instrumental, S y m b o l i c (e.g. Rich , 1978)
(Patton et
in
• "To w h a t e x t e n t is the information required?"
• Political context (Patton et al.,
•"What is the e v a l u a tion c o n t e x t ? "
•C h a n g e s in v a l u e s , u n d e r s t a n d ing, roles, o r g a n i z a t i o n , m a t e r i a l s (Fullan, 1979)
factor 1977)
• R e p o r t style (e.g. N e w m a n , B r o w n & B r a s k a m p , 1980; B r a s k a m p , Brown, & N e w m a n , 1981)
• Issues Addressed: Districtwide, p r o g r a m b u i l d i n g level, c l i n i c a l (Kennedy, A p l i n g , & N e w m a n , 1980)
• M o d e s of Use: Conceptual, Peiz, 1977;
• Personal factor al., 1977)
Variables Suggested Other Literature
• "What k i n d s of i n f o r m a t i o n are used?"
."For what purpose is the i n f o r m a t i o n u~ed?"
."Who uses the information?
Brown (1981, p.
Frameworks
Timing
factors
.Extraorganizational
•P r e e x i s t i n g e v a l u a t i o n bounds .Evaluator's approach .Evaluator credibility .Organizational factors
Context
style
• Information content and r e p o r t i n g
• Administrator
of users
Daillak, & White (1979, p. 235)
• Orientation
Alkin,
Information Used
Purpose
User
Dimensions
Earlier Conceptual
N
m
0
(,1
m
U
g
e
0
g
o
C
O
~o
180
J. A. King
B u t , as t h e o t h e r d i m e n s i o n s o f T a b l e 1 s u g g e s t , t h i s is o n l y p a r t o f t h e use picture. T h e s e c o n d p o i n t t h a t m u s t be s t r e s s e d is t h a t use c a n n o t be d i s c u s s e d outside of the context of that use. T h e c o n t e x t s h a p e s a n d l i m i t s w h a t can be done with information, and without a thick description of the evaluation context, i t is m e a n i n g l e s s to d i s c u s s u s e . T h e i m p o r t a n c e o f c o n t e x t is e v i d e n t in t h e f a c t t h a t f i v e o f t h e e i g h t f a c t o r s in A l k i n , Daillak, and White's analytic framework o f local e v a l u a t i o n use, derived from naturalistic case s t u d i e s , r e l a t e to t h e evaluation setting. It s h o u l d also be n o t e d t h a t t h e t w o e v a l u a t o r f a c t o r s - - a p p r o a c h and credibility-h a v e b e e n i n c l u d e d u n d e r c o n t e x t ; in t h i n k i n g a b o u t u s e , we m u s t r e c o g n i z e t h a t t h e e v a l u a t o r becomes s i m p l y a p a r t o f t h e u s e r ' s e n v i r o n m e n t . This appropria t e l y r e d u c e s t h e e v a l u a t o r f r o m t h e k e y c h a r a c t e r in e v a l u a t i o n use to m e r e l y an i m p o r t a n t c o m p o n e n t o f t h e use s e t t i n g . B y a c t i n g in c e r t a i n w a y s , e v a l u a t o r s m a y p r o m o t e o r i n h i b i t u s e ; b u t , as n o t e d a b o v e a n d in K e n n e d y , A p l i n g , a n d N e u m a n n ( 1 9 8 0 ) , e v a l u a t i o n use is to a l a r g e e x t e n t o u t o f e v a l u a t o r s ' h a n d s . G i v e n t h a t t h i s is o f t e n t h e c a s e , t h e q u e s t i o n t h e n b e c o m e s , w h a t can we do about this? Studying local e v a l u a t i o n use m a y lead to a n e w a w a r e n e s s o f t h e dimensions of use. B y o p e n i n g o u r e y e s to t h e w a y s i n f o r m a t i o n is c r e a t e d a n d p r o c e s s e d in c o n t e x t , q u e s t i o n s w i l l e m e r g e e n a b l i n g us to f o c u s on t h e n a t u r e o f use. T h e s t u d y I w i l l d e s c r i b e h e r e is a t t e m p t i n g b o t h t o r a i s e s u c h q u e s t i o n s a n d to b e g i n p r o v i d i n g a n s w e r s b y l o o k i n g at use as a p r o c e s s , n o t j u s t as a product.
The C u r r e n t Study O u r s t u d y is a d d r e s s i n g t w o q u e s t i o n s : ( 1 ) What p r o c e s s e s a r e i n v o l v e d in t h e g e n e r a t i o n a n d use o f i n f o r m a t i o n in t h e r e s e a r c h a n d e v a l u a t i o n u n i t o f a l a r g e p u b l i c school s y s t e m ? a n d ( 2 ) What is t h e v a l i d i t y of t h e f a c t o r s i d e n t i f i e d in previous r e s e a r c h f o r a n a l y z i n g t h e a c t i o n s o f e v a l u a t i o n u s e r s w i t h i n a local educational agency? The study will have two phases. The phase currently u n d e r w a y is an i n t e n s i v e case s t u d y o f t h e r e s e a r c h a n d e v a l u a t i o n u n i t o f a l a r g e c i t y s c h o o l d i s t r i c t , using Guba's wave conceptualization of inquiry (Guba, 1978). Moving between discovery and verification modes, this phase will ultimately include three "waves." T h e p u r p o s e o f t h e f i r s t w a v e is t o c r e a t e a p o r t r a i t o f t h e w o r k i n g s o f t h e r e s e a r c h a n d e v a l u a t i o n u n i t as seen b y its s t a f f , i . e . , b y t h o s e d i r e c t l y i n v o l v e d in d a t a c o l l e c t i o n a n d p o t e n t i a l u s e . B e c a u s e t h e d e s i g n is e m e r g i n g as t h e s t u d y p r o g r e s s e s , i t is d i f f i c u l t t o p r e s e n t i t in d e t a i l at t h i s t i m e . H o w e v e r , we h a v e b e g u n b y c o n d u c t i n g e x t e n s i v e i n t e r v i e w s w i t h t h e R&E s t a f f a n d k e y a g e n t s w h o interact with the evaluation personnel and process, and by observing day-to-day l i f e in t h e o f f i c e ( a t t e n d i n g m e e t i n g s , c o n d u c t i n g o n - s i t e v i s i t s , c o m p l e t i n g d a t a forms, interacting with the system's administrative structure, etc.). In a d d i t i o n , we w i l l e x a m i n e all a v a i l a b l e d o c u m e n t s a n d r e p o r t s u n t i l t h e s t r u c t u r e of t h i s d y n a m i c c o n t e x t is r e v e a l e d . The write-up of this description and the ensuing member check will conclude the first wave and will present a picture of evaluation use in process. T h e second w a v e of the case study will m o v e further from the internal perceptions of the R & E unit into the context in which that use occurs. T w o books-Patton et al. (1977) a n d Alkin, Daillak, a n d White (1979)--suggest the importance of this move. B a s e d on our findings in the first wave, w e will administer a questionnaire to instructional supervisors a n d coordinators, to all local principals, a n d to a r a n d o m sample of 500 teachers. W e will also interview key central administration decision m a k e r s along with selected supervisors, principals, teachers, and students to determine the w a y s they think about and use information
181
Studying the Local Use of Evaluation
ge n e ra t e d by the e v a l u a t i o n process in both f e d e r a l l y and locally f u n d e d p r o grams. At the completion of this wave, the e a r l i e r d e s c r i p t i o n will be elaborated into a draft of our case study. M o v i n g even further into the context surrounding the use of local evaluation results, the third w a v e will include interviews with m e m b e r s of the local Board of Education and with appropriate representatives of the State D e p a r t m e n t of Education. T h e first phase of the project will then conclude with the fully developed case study. T h e second phase of the project will seek to integrate the theory of evaluation use developed from practice with the extant, empirically based "theory" of use in the literature. Recognizing the limits of studying a single case, w e will test the theory derived from the case study against the empirical and real world theory derived from the literature and from L E A practitioners. This will clarify and elaborate on our findings about evaluation use in public schools. A copy of the case study and the derived theoretical f r a m e w o r k will be sent to these experts along with a structured set of interview questions. Telephone interviews will be conducted using these questions as guides, but allowing the interviewees to discuss in general the question of evaluation use in public schools. T h e project combines several approaches in studying the process of evaluation information use and in beginning to validate a theory of such use. T h e case study methodology of the first phase combines ethnographic techniques with interviews and s u r v e y methods to both reveal and test the actual processes involved in the use of evaluation information in terms of the interplay a m o n g key factors, actors, and agents. Rather than tracing the impact of a single study or data set, w e will examine the use and effects of studies done in the past, those ongoing, and those projected for the future. T h e validation procedures of the second phase will address questions of the applicability of our results and, w e hope, will promote discussion of critical issues in evaluation use by practitioners and scholars in the field.
REFERENCES Alkin,
M.C.
using
naturalistic
research
for
the
study
of
evaluation.
Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, 1979. (ERIC_, No. 170 378) Alkin,
M.C.,
Daillak,
make a difference?
R.H., & White, P. Using e v a l u a t i o n s : Beverly Hills, Calif. : Sage Publications,
Does e v a l u a t i o n
1979.
Andrews, J.V. Reactions to program e v a l u a t i o n : A q u a l i t a t i v e analysis (Doctoral d i s s e r t a t i o n , Cornell U n i v e r s i t y , 1979). Dissertation Abstracts International, 1979, 39, 6717A. ( U n i v e r s i t y Microfilms No. 79-10, 731) B r o w n , R.D. Evaluation utilization: A critique. Unpublished manuscript,
A literature
r e v i e w and r e s e a r c h
agenda,
1981.
In B r o w n , R . D . , & Braskamp, L . A . Summary: C o m m o n themes and a checklist. L . A . Braskamp & R.D. Brown ( E d s . ) , U t i l i z a t i o n of e v a l u a t i o n information. San Francisco: Jossey-Bass, 1980.
Caplan, N.
What do we know about knowledge u t i l i z a t i o n ?
R.D. B r o w n (Eds.), Jossey-Bass, 1980.
Utilization
of e v a l u a t i o n
In L . A .
information.
San
Braskamp & Francisco:
182
J. A. King
Caplan,
N.,
Morrison,
ledge in policy
A.,
decisions
& Stambaugh,
R.J.
at the national
The use o f s o c i a l level: A report to
s c i e n c e knowrespondents.
Ann A r b o r : C e n t e r f o r Research on U t i l i z a t i o n of S c i e n t i f i c K n o w l e d g e , Institute for Social Research, University of Michigan, 1975. (ERIC No. 111 729) Cronbach, L.J., Ambron, S.R., Dornbusch, S.M., P h i l l i p s , D . C . , Walker, D . F . , & Weiner, S.S. evaluation. San Francisco: J o s s e y - B a s s , 1980. David,
J.
Analysis,
Hess,
R.D.,
Toward
Local uses of T i t l e I e v a l u a t i o n s . E d u c a t i o n a l 1981, 3, 27-39.
Hornik,
reform
of
Evaluation
R.C., program
and P o l i c y
D i c k e y , B. U t i l i z a t i o n of e v a l u a t i o n s of small scale educational p r o j e c t s ( D o c t o r a l d i s s e r t a t i o n , U n i v e r s i t y of Minnesota, 1979). Dissertation Abstracts International, 1979, 40, 3245A. ( U n i v e r s i t y Microfilms No. 79-26, 115) Fullan, M. C o n c e p t u a l i z i n g p r o g r a m s of curriculum implementation. Paper p r e sented f o r the Symposium on C u r r i c u l u m I n q u i r y in Canada, V i c t o r i a , 1979. Glaser,
B.G.,
qualitative
& Strauss, A.L. D i s c o v e r y of grounded research. Chicago: Aldine, 1967.
theory:
Strategies
Guba, E.G. Toward a methodology of n a t u r a l i s t i c inquiry in educatinal Los Angeles: C e n t e r f o r the S t u d y of E v a l u a t i o n , 1978. Guba, E . G . , 1981. Kennedy,
& Lincoln, Y . S .
M.M.,
information
Apling,
R.,
Effective
evaluation.
& Neumann, W.F. Cambridge,
in p u b l i c schools.
San Francisco:
The r o l e
Mass. :
of
The
for
evaluation.
Jossey-Bass,
evaluation
Huron
and t e s t
Institute,
1980. King,
J.A.,
& T h o m p s o n , B . A n a t i o n w i d e survey of administrators' perceptions Paper p r e s e n t e d at the Annual Meeting of the American Educational Research Association, Los A n g e l e s , 1981.
of
King,
evaluation.
J.A., school
Use). Lyon,
T h o m p s o n , B . , & Pechman, E.M. I m p r o v i n g e v a l u a t i o n use i n l o c a l (Final r e p o r t of NIE G r a n t 80-0082, O p t i m i z i n g Evaluation New O r l e a n s : New Orleans Public Schools, June 1981.
settings
C.D.,
districts.
Doscher, L . , McGranahan, P., & Williams, R. E v a l u a t i o n and s c h o o l Los A n g e l e s : C e n t e r f o r the S t u d y of E v a l u a t i o n , 1978.
Newman, D . L . , Brown, R.D., & the u t i l i z a t i o n of e v a l u a t i o n . Utilization
Patton, M.Q.
of evaluative
Qualitative
Braskamp. L.A. Communication t h e o r y and In L . A . B r a s k a m p & R . D . Brown ( E d s . ) , information. San Francisco: Jossey-Bass. 1980.
evaluation
methods.
Beverly Hills: Sage, 1980.
Patton, M . Q . , Grimes, P . S . , G u t h r i e , K . M . , B r e n n a n , N . J . , Grench B . D . , & Blyth, D.A. In search of impact: An analysis of u t i l i z a t i o n of federal health e v a l u a t i o n r e s e a r c h . In C . H . Weiss ( E d . ) , U s i n g s o c i a l r e s e a r c h i n public policymaking. L e x i n g t o n , Mass.: D . C . Heath, 1977. Pelz,
D.C. Some e x p a n d e d p e r s p e c t i v e s on use of social science in p u b l i c p o l i c y . In J.M. Yinger & S.J. Cutler (Eds.), Major social issues, a m u l t i d i s c i p l i n a r y view. N e w York: The Free Press, 1978.
Studying the Local Use of Evaluation
Rich,
183
R.F. Uses of social science i n f o r m a t i o n b y f e d e r a l b u r e a u c r a t s : Knowledge f o r action v e r s u s k n o w l e d g e f o r u n d e r s t a n d i n g . In C . H . Weiss ( E d . ) , Using social research in public p o l i c y m a k i n g . Lexington, Mass. : D.C. Heath, 1977.
Simon, H.A. Smith,
New
Administrative behavior.
N.L.
The
president's
corner.
York:
MacMillan, 1957.
Evaluation News,
Summer,
1980,
24-25.
Stake, R. Evaluation design, instrumentation, data collection and analysis of d a t a Urbana, III.: Center for Instructional Research and Curriculum Evaluation, 1976. Thompson,
B.,
&
King,
J.A.
A critique
of e v a l u a t i o n - u s e research
methods.
CEDR Quarterly, in press.
Weiss, C . H .
Conceptual issues in measuring the u t i l i z a t i o n evaluation. Paper p r e s e n t e d at the Annual Meeting
of r e s e a r c h
of
the
and
Evaluation
Research Society, Minneapolis, 1979. Wilensky, H.L., & Lebeaux, C.N. N e w York: Russell Sage, 1958. Wise,
Industrial
society
and social welfare.
R.I. W h a t w e k n o w a b o u t the d e c i s i o n - m a k i n g and decision settings. Paper p r e s e n t e d at t h e Annual Meeting of the American Educational Research A s s o c i a t i o n , T o r o n t o , 1978.
THE JEAN KING Louisiana.
is
Professor
of
AUTHOR
Education
at
Tulane
University,
New
Orleans,
0191-491X/82/020175-09504.5010 Copyright©1982PergamonPressLtd.
Printedin GreatBritain.All rights reserved
S T U D Y I N G THE LOCAL USE OF EVALUATION: A DISCUSSION THEORETICAL ISSUES A N D A N EMPIRICAL S T U D Y
OF
Jean A. King 7-ulane University, New Orleans, Louisiana, USA
IntJrOd~ L i k e t h e Eloi a n d t h e M o r l o c k s in H. G. Wells' T i m e M a c h i n e , e v a l u a t o r s a n d t h e i r c l i e n t s f o r t h e m o s t p a r t l i v e in s e p a r a t e w o r l d s a n d m a k e o n l y o c c a s i o n a l contacts. Though the comparison with Wells' future obviously exaggerates reality, t h e r e is e n o u g h t r u t h in i t t o m a k e an e v a l u a t o r u n c o m f o r t a b l e . Wells presents a distressing f o r e c a s t in his v i s i o n o f t h e EIoi, d e s c e n d a n t s o f t h e British aristocracy who share a world with the Morlocks, the ever hungry descendants of the working class. In t h a t f a r d i s t a n t t i m e , t h e t w o c o m m u n i t i e s live harmoniously. T h e f r a i l EIoi l i v e a b o v e t h e g r o u n d a n d h a v e no u n m e t needs, either felt or unfelt, while the hirsute Morlocks live below the ground and k e e p t h e h u g e m a c h i n e r y t h a t is t h e i r l i v e l i h o o d g r e a s e d a n d t u r n i n g . T h e r e is a catch, of course, and a deadly one: at n i g h t t h e M o r l o c k s c l i m b f r o m t h e i r subterranean l a i r s t o f e e d o n t h e c h i l d l i k e a n d d e f e n s e l e s s EIoi. What a t f i r s t appears to the Time Traveller a p a r a d i s e r e v e a l s i t s e l f f i n a l l y to be a c o n t i n u a l nightmare. To administrators and other practitioners, e v a l u a t o r s m a y seem l i k e t h e E I o i - naive, spoiled, out of shape, and used to getting their way-while they, like the M o r l o c k s , t o i l in t h e s t i f l i n g r e a l i t y a n d o v e r p o w e r i n g c o n s t r a i n t s o f t h e i r school settings. If i t is r e m e m b e r e d t h a t t h e M o r l o c k s d e v o u r t h e EIoi o n e a t a t i m e , t h e i m a g e o f t h e h a p l e s s e v a l u a t o r in t h e t h r o e s o f a c o n t r o v e r s i a l s t u d y can s e r v e as a r e m i n d e r t h a t t h e j o b o f e v a l u a t o r is n o t w i t h o u t its d a n g e r s . To e v a l u a t o r s , on t h e o t h e r h a n d , d e c i s i o n - m a k e r s a n d o t h e r c l i e n t s m a y seem a b i t l i k e t h e EIoi. U n a p p r e c i a t i v e o r e v e n f r i g h t e n e d o f t h e i m p r e s s i v e social s c i e n c e m a c h i n e r y t h e e v a l u a t o r c o n t r o l s , t h e y l i v e in a w o r l d d i s t i n c t f r o m t h e r i g o r s o f multiple regression and factor analysis. Unless dragged, the rarely enter the evaluator's world, preferring the bliss of ignorance to so-called "objective" realities, a n d y e t t h e y d o n o t h e s i t a t e t o use t h e e m p i r i c i s t M o r l o c k p r o d u c t s when these suit their needs. Typically, evaluators and their clients do inhabit separate worlds, seeing life through different conceptual glasses. Caplan's "two communities" theory (see, f o r e x a m p l e , C a p l a n e t a l . , 1975) s u g g e s t s t h a t an e v a l u a t o r ' s c o n c e r n f o r t e c h nically correct procedures may conflict with a program staff's implementation plans o r w i t h a local d e c i s i o n m a k e r ' s i n f o r m a t i o n n e e d s . While t h e e v a l u a t o r sees a local p r o g r a m as a p i e c e o f an e d u c a t i o n a l e x p e r i m e n t , an a d m i n i s t r a t o r may i n s t e a d see i t as p r o v i d i n g e s s e n t i a l s e r v i c e s t o s t u d e n t s . Years ago, Wilensky
175
176
J. A. King
and L e b e a u x n o t e d t h a t " w h a t t h e social s c i e n t i s t t h i n k s of as ' o b j e c t i v e i n v e s t i g a t i o n ' t h e p r a c t i t i o n e r o f t e n t a k e s as ' h o s t i l e a t t a c k ' " (1958, p. 2 0 ) . C e r t a i n e m p i r i c a l e v i d e n c e d o c u m e n t s t h e s e p a r a t i o n of e v a l u a t o r and u s e r . In a s u r v e y of school a d m i n i s t r a t o r s , K i n g and T h o m p s o n f o u n d o n l y 2& who had d a i l y c o n t a c t w i t h e v a l u a t o r s , w h i l e t h e m a j o r i t y r e p o r t e d s p e a k i n g to e v a l u a t o r s less t h a n once e v e r y t w o w e e k s (1981, p. 9 ) . L y o n et a l . ' s r e c e n t s u r v e y of LEA e v a l u a t i o n u n i t heads f o u n d t h a t m a n y h a v e had l i t t l e e x p e r i e n c e at t h e school l e v e l , p o t e n t i a l l y m a k i n g t h e i r a w a r e n e s s of t h a t s e t t i n g less t h a n d e s i r a b l e ; " n o t o n l y h a v e 42& of them n o t t a u g h t , b u t 70& h a v e not r u n a s c h o o l " ( L y o n et a l . , 1978, p. 6 6 ) . In s t u d y i n g t h e local uses of T i t l e I e v a l u a t i o n s , D a v i d f o u n d t h a t f o r m a n y t e a c h e r s e v a l u a t i o n is s y n o n y m o u s w i t h a c h i e v e m e n t t e s t i n g , s u g g e s t i n g t h a t u s e r s ' p e r c e p t i o n s of e v a l u a t i o n c l e a r l y d i f f e r f r o m e v a l u a t o r s ' ( D a v i d , 1981). A further s t u d y s u g g e s t i n g t h e d i f f e r i n g p e r s p e c t i v e s of e v a l u a t o r s and t h e i r c l i e n t s is t h a t of K e n n e d y , A p t i n g , and N e u m a n n , who f o u n d t h a t w h i l e c l i e n t s r a r e l y q u e s t i o n e d t h e t e c h n i c a l q u a l i t y of e v a l u a t o r s ' w o r k , they frequently q u e s t i o n its " e d u c a t i o n a l " q u a l i t y ( 1 9 8 0 ) . I have purposely exaggerated the separation between e v a l u a t o r s and users b e c a u s e - - l i k e it o r n o t - - w e m u s t acknowledge t h e f a c t t h a t t h e o n l y place o u r w o r k can be a success is in t h e u s e r s ~ w o r l d . If o u r r e s u l t s a r e n o t used t h e r e , t h e y a r e n o t u s e d at a l l ; and if t h e y a r e n o t used at a l l , s o m e o n e - - p r o b a b l y u s - - h a s w a s t e d an e n o r m o u s a m o u n t of e n e r g y in an era of s e v e r e e n e r g y s h o r t ages. My message r e d u c e s at last to t h i s simple t h o u g h t : T h e w o r l d of t h e e v a l u a t o r m u s t become n o t l i k e t h e u s e r s ' w o r l d , b u t m u s t be t h e u s e r s ' w o r l d . We m u s t n o t o n l y e n t e r t h e w o r l d of o u r local c l i e n t s , we need to r e n t o f f i c e space. T h i s r e a s o n i n g p r o v i d e s t h e t h e o r e t i c a l r a t i o n a l e f o r t h e s t u d y I will d e s c r i b e in t h i s p a p e r . T h e p u r p o s e of t h e p r o j e c t is to s t u d y t h e p r o c e s s of e v a l u a t i o n use at t h e local l e v e l , h e l p i n g us to u n d e r s t a n d b e t t e r how i n f o r m a t i o n is c r e a t e d at t h e local level and to d i s c o v e r t h o s e f a c t o r s a f f e c t i n g w h a t h a p p e n s to r e s u l t s in t h e s e t t i n g s w h e r e t h e y c o u n t . T h e p a p e r has t h r e e s e c t i o n s : first, a d i s c u s s i o n of t h e c o n c e p t of e v a l u a t i o n u t i l i z a t i o n / u s e and t h e r e l a t e c o n c e p t of u s e r r a t i o n a l i t y ; a n d s e c o n d , a d e s c r i p t i o n of a n a t u r a l i s t i c s t u d y currently underway. B e f o r e c o n t i n u i n g w i t h t h e t e x t , h o w e v e r , I must p r e s e n t a c r i t i c a l d i g r e s s i o n . Several theorists--notably Weiss (1979) and C a p l a n ( 1 9 8 0 ) - - h a v e p o i n t e d to t h e d a n g e r of a s s u m i n g a u t o m a t i c a l l y t h a t t h e use of e v a l u a t i o n r e s u l t s is g o o d a n d t h e n o n - u s e n e c e s s a r i l y b a d , an a s s u m p t i o n f a i r l y common in t h e u t i l i z a t i o n l i t e r a ture. If, h o w e v e r , the results are flawed for methodological reasons or have been d o c t o r e d to s u i t an i n s t i t u t i o n ' s p o l i t i c s , a d e c i s i o n m a k e r may well be wise n o t to use t h e m . T h i s is n o t to s a y t h a t we m u s t w a i t u n t i l we a t t a i n e v a l u a t i o n n i r v a n a w h e r e e v a l u a t o r s a c r o s s t h e c o u n t r y c o l l e c t ideal d a t a , a n a l y z e them i m p e c c a b l y , a n d p r e s e n t them in h i g h l y a p p r o p r i a t e f o r m s b e f o r e we b e g i n to s t u d y t h e f a c t o r s t h a t a f f e c t use. It is c e r t a i n l y i m p o r t a n t t h a t t h e e v a l u a t i o n c o m m u n i t y c o n t i n u e to make its S t a n d a r d s e x p l i c i t , to b e g i n to p o l i c e its r a n k s , and to i n s i s t t h a t p o o r q u a l i t y e v a l u a t i o n s n o t be t o l e r a t e d , a l t h o u g h d o i n g so may not i n c r e a s e t h e use of r e s u l t s ( s e e , f o r e x a m p l e , P a t t o n et a l . , 1977, p. 151). B u t w h i l e we a r e d o i n g t h i s on one f r o n t , we must w o r k s i m u l t a n e o u s l y on a n o t h e r , i . e . , t r y i n g to u n d e r s t a n d w h a t we can do to i n c r e a s e t h e a p p r o p r i a t e use of e v a l u a t i o n r e s u l t s . We h a v e n ' t t h e l u x u r y of time to p u t o u r s h o p in order before our customers arrive. T h e y a r e h e r e , and some are e v e n e a g e r to see w h a t we h a v e to o f f e r . B y s t u d y i n g t h e c u r r e n t use of i n f o r m a t i o n at t h e local l e v e l , we may l e a r n how to i n c r e a s e o u r c h a n c e s t h a t r e s u l t s w o r t h u s i n g get used. T o p u t i t a n o t h e r w a y , some of us w i s e r , n o t e n t i r e l y d e f e n s e l e s s Eloi
Studying the Local Use of Evaluation
must recognize that share our breathing
the key to our survival space and to learn the
177
is t o w a l k a m o n g t h e b e i n g s w h o key to their spirit and ingenuity.
The Concepts of Utilization ( U s e ) and Rationality T h e c o n c e p t o f e v a l u a t i o n u t i l i z a t i o n has c h a n g e d s i g n i f i c a n t l y in t h e p a s t t e n years. Once upon a time, the evaluation community subscribed to a "big bang" theory of utilization (Andrews, 1979), assuming somewhat naively that merely p r o v i d i n g d e c i s i o n m a k e r s t h e r e s u l t s o f e v a l u a t i o n s t u d i e s w o u l d l e a d t o t h e use of those results. From this perspective, evaluators were frequently chagrined and even embarrassed to discover that the results they rigorously generated more o f t e n e n d e d u p c o l l e c t i n g d u s t on s o m e o n e ' s s h e l f t h a n a f f e c t i n g f u t u r e d e c i s i o n s . The literature abounds with gloomly perceptions of the "failure of educational evaluation" (see King, Thompson, & Pechman, t981, pp. 1-2, for a collection of such quotations). A t o n e p o i n t , R o b e r t S t a k e w e n t so f a r as t o w o n d e r w h e t h e r " e v a l u a t i o n is g o i n g t o c o n t r i b u t e m o r e t o t h e p r o b l e m s o f e d u c a t i o n o r m o r e to t h e s o l u t i o n s " ( 1 9 7 6 , p. 1 ) . T h i s p e r s p e c t i v e is w h a t A l k i n , D a i l l a k , a n d White call t h e " m a i n s t r e a m " v i e w p o i n t o f e v a l u a t i o n u t i l i z a t i o n , i . e . r t h e v i e w p o i n t t h a t "contends that evaluations seldom influence program decision makers and holds out l i t t l e h o p e t h a t e v a l u a t i o n w i l l e v e r b r e a k t h r o u g h t h e b a r r i e r s to r e a l i m p a c t on programs" (1979, p. 17). The results of empirical studies of utilization, however, have challenged the w o r s t - c a s e p e r c e p t i o n s o f t h i s m a i n s t r e a m p e r s p e c t i v e , a n d an " a l t e r n a t i v e " v i e w p o i n t has e m e r g e d in r e c e n t y e a r s ( A l k i n , Daillak, & White, 1979). Studies by Patton et al. (1977), Alkin, Daillak, a n d White ( 1 9 7 9 ) , Dickey (1979), and Kennedy, Apling, and Neumann (1980) suggest that evaluation results are used b y d e c i s i o n m a k e r s , " b u t n o t in t h e c l e a r - c u t a n d o r g a n i z a t i o n - s h a k i n g ways that social s c i e n t i s t s s o m e t i m e s b e l i e v e r e s e a r c h s h o u l d be u s e d " ( P a t t o n et a l . , 1977, p. 144). The current v i e w o f e v a l u a t i o n use as s u m m a r i z e d b y B r o w n a n d B r a s k a m p is t h a t effective utilization does n o t n e c e s s a r i l y mean t h a t a n y o f t h e recommendations are implemented or that there are any immedia t e l y a p p a r e n t d e c i s i o n s b a s e d on t h e i n f o r m a t i o n . ( 1 9 8 0 , p. 91) Rather, results frequently i n f l u e n c e u s e r s in i n d i r e c t o r g r a d u a l w a y s . The d i s c o u r a g i n g m a i n s t r e a m p i c t u r e o f e v a l u a t i o n use m a y h a v e s t e m m e d m o r e f r o m a m i s t a k e n e x p e c t a t i o n r e g a r d i n g t h e n a t u r e o f use t h a n f r o m t h e a c t u a l l e v e l o f use in t h e real w o r l d . Wise ( 1 9 7 8 ) has w r i t t e n t h a t If t h e r e is an e v a l u a t i o n utilization problem, i t is n o t t h a t decision-makers do n o t use t h e i n f o r m a t i o n t h e y r e c e i v e , i t is t h a t e v a l u a t o r s c a n n o t e a s i l y see t h e i r i n f o r m a t i o n b e i n g u s e d in the incrementalism of real-world decision-making. ( p . 24) O r , as Pogo so e l o q u e n t l y o n c e s a i d , "We h a v e met t h e e n e m y , a n d t h e y a r e u s . " No l o n g e r , t h e n , is i t s u f f i c i e n t t o l o o k f o r d i r e c t a n d r e a d i l y a p p a r e n t i n s t a n c e s of utilization. S u p p o r t e d b y e m p i r i c a l d a t a a n d common s e n s e , t h e a l t e r n a t i v e v i e w p o i n t has t a k e n t h e f i e l d . Given this viewpoint, e v e n t h e c o n c e p t u a l label n e e d s t o be c h a n g e d . To speak of utilization is t o s u g g e s t t h e m a i n s t r e a m v i e w a n d t o l i n g u i s t i c a l l y bias the search for cases. U t i l i z a t i o n was an a p p r o p r i a t e term for researchers with the mainstream view, but, as Weiss ( 1 9 7 9 ) w r i t e s , i t " e m b o d i e s an i n a p p r o p r i a t e imagery . . . b e c a u s e o f its o v e r t o n e s of instrumental episodic application. People don't utilize research the way that they utilize a hammer" (p. 2). For this reason, the alternative viewpoint suggests the need for substituting the term use, a word with few inappropriate connotations, for the term utilization.
178
J. A. King
Like the concept of utilization, t h e c o n c e p t o f local u s e r s ' r a t i o n a l i t y has also c h a n g e d in r e c e n t y e a r s . T h e m a i n s t r e a m v i e w p o i n t a s s u m e d a classic, b u t l i m i t e d n o t i o n o f r a t i o n a l b e h a v i o r , a n o t i o n i m p l i c i t in w h a t t call t h e r a t i o n a l m y t h of e v a l u a t i o n u s e . B r i e f l y s t a t e d , t h e m y t h goes l i k e t h i s : Harried decision-makers, over-worked b u t r a t i o n a l at h e a r t , a r e presented evaluative information t h a t is r e l e v a n t to d e c i s i o n s t h e y a r e a b o u t to m a k e . S i t t i n g c a l m l y at t h e i r d e s k s , t h e y consider the data, weigh [heir options, then make the "correct" decision, i.e., the choice supported by the data. The evaluator, b e a m i n g h a p p i l y b e h i n d t h e s c e n e s , t a k e s p r i d e in n u r turing such rational action. (King, Thompson, & Pechman, 1981, p. 4) E v a l u a t o r s no l o n g e r p r e t e n d that this scenario will always--or even ever--be played out. Weiss ( 1 9 7 9 ) p r e s e n t s e x a m p l e s w h e r e t h e " r a t i o n a l " c h o i c e f o r a d e c i s i o n m a k e r is u n c l e a r , f o r e x a m p l e , w h e n e v a l u a t i o n r e s u l t s a r e i n c o m p l e t e o r conflicting; t h e f r e q u e n t p r e s e n c e of such s i t u a t i o n s in t h e real w o r l d is u n q u e s t i o n a b l e a n d m a k e s t h e c a u s e - e f f e c t n o t i o n of e v a l u a t i o n use u n r e a l i s t i c . A l s o , as K i n g , T h o m p s o n , a n d Pechman n o t e , t h e m y t h i c a l v i e w d e n i e s t h e p r o c e s s a n d c o n t e x t o f real w o r l d d e c i s i o n m a k i n g ( 1 9 8 1 , p. 4 ) . C r o n b a c h e t al. w r i t e t h a t d e c i s i o n s as " f o r m a l c h o i c e ( s ) at a p a r t i c u l a r " t i m e b e t w e e n d i s c r e t e a l t e r n a t i v e s " rarely occur in e v a l u a t i o n settings (1980, p. 8 4 ) . Rather, in c o n t e x t s of pluralistic accommodation, decision makers "satisfice," i.e., find a course of a c t i o n t h a t is " g o o d e n o u g h " ( S i m o n , 1975, p. 2 0 4 ) ; a n d , o v e r a p e r i o d o f t i m e , decisions evolve. T h e " r a t i o n a l " d e c i s i o n in such e n v i r o n m e n t s is w h a t e v e r is likely to work, and "rationality" is a r e l a t i v e n o t i o n . If, for example, analysis s u g g e s t s t h a t n o t u s i n g an e v a l u a t i o n r e s u l t is a p p r o p r i a t e , then non-use may represent a highly rational "use" of evaluation (see King, Thompson, & Pechman, 1981, p p . 6 - t 2 ) . In d i s c u s s i n g w h a t h a p p e n s to e v a l u a t i o n r e s u l t s , S m i t h d i s t i n g u i s h e s between use, the conscious consideration and e m p l o y m e n t o f r e s u l t s , a n d i m p a c t , " a n y d i s c e r n a b l e a c t i o n s , e v e n t s , o r c h a n g e s in c o n d i t i o n s t h a t a r e d i r e c t l y i n f l u e n c e d b y t h e e v a l u a t i o n , its p r o c e s s e s , p r o d u c t s , o r f i n d i n g s " ( 1 9 8 0 , p. 2 5 ) . I t is n o t e n o u g h to o b s e r v e c h a n g e s r e s u l t i n g f r o m e v a l u a t i o n s . F o r an e x a m p l e to be l a b e l e d u s e , t h e b l a c k b o x m u s t be o p e n e d to r e v e a l s o m e o n e p u r p o s e l y d o i n g something with the results. T h i s d e f i n i t i o n of u s e , t h e n , has a p e r s o n a t t a c h e d . Smith's definition--the " c o n s c i o u s e m p l o y m e n t o f an e v a l u a t i o n ( i t s p r o c e s s e s , p r o d u c t s , o r r e s u l t s ) to a c h i e v e some d e s i r e d e n d o r i m p a c t " - - r e f l e c t s t h e p u r p o s e f u l n a t u r e of use ( 1 9 8 0 , p. 2 5 ) , a n d t h e c o n s c i o u s e l e m e n t is b o t h a n e c e s s a r y a n d s u f f i c i e n t c o n d i t i o n of use. T h e s e r i o u s c o n s i d e r a t i o n of e v a l u a t i o n r e s u l t s is also n e c e s s a r y to u s e , t h o u g h n o t s u f f i c i e n t ; o n e can i m a g i n e , f o r e x a m p l e , a d e c i s i o n m a k e r w h o r e a d s an e v a l u a t i o n r e p o r t , b u t is f o r c e d to m o v e on t o o t h e r c o n c e r n s b e f o r e b e i n g a b l e to a c t on t h e r e s u l t s in a n y w a y ( e v e n i f j u s t to i g n o r e t h e m c o n s c i o u s l y ) . Table 1 presents conceptual dimensions of evaluation use, integrating the earlier f r a m e w o r k s of A I k i n , D a i t t a k , a n d White ( 1 9 7 9 ) , Weiss ( 1 9 7 9 ) , a n d B r o w n ( 1 9 8 1 ) . T h e t a b l e is s t r u c t u r e d on t h e j o u r n a l i s t ' s "SW's a n d an H" ( w h o , w h y , w h a t , w h e r e , w h e n , a n d h o w ) a n d s u g g e s t s t w o i m p o r t a n t p o i n t s f o r s t u d y i n g local u s e . F i r s t , to see w h a t f a c t o r s a f f e c t u s e , we m u s t f o c u s on t h e u s e r and h i s / h e r response to the information presented. As P a t t o n et a l . ' s p e r s o n a l f a c t o r d e m o n strates (1977), the user's characteristics and purposes will in large part d e t e r m i n e to w h a t e x t e n t r e s u l t s w i l l be u s e d .
."How i m m e d i a t e the use?" ."How m u c h e f f e c t is required?"
Outcome
is
."How d i r e c t is derivation from the s t u d y ? "
."What is u s e d ? "
."By h o w m a n y p e o p l e ?
."By w h o m is it used?"
Weiss (1979, pp. 11-12) 3)
Instrumental, S y m b o l i c (e.g. Rich , 1978)
(Patton et
in
• "To w h a t e x t e n t is the information required?"
• Political context (Patton et al.,
•"What is the e v a l u a tion c o n t e x t ? "
•C h a n g e s in v a l u e s , u n d e r s t a n d ing, roles, o r g a n i z a t i o n , m a t e r i a l s (Fullan, 1979)
factor 1977)
• R e p o r t style (e.g. N e w m a n , B r o w n & B r a s k a m p , 1980; B r a s k a m p , Brown, & N e w m a n , 1981)
• Issues Addressed: Districtwide, p r o g r a m b u i l d i n g level, c l i n i c a l (Kennedy, A p l i n g , & N e w m a n , 1980)
• M o d e s of Use: Conceptual, Peiz, 1977;
• Personal factor al., 1977)
Variables Suggested Other Literature
• "What k i n d s of i n f o r m a t i o n are used?"
."For what purpose is the i n f o r m a t i o n u~ed?"
."Who uses the information?
Brown (1981, p.
Frameworks
Timing
factors
.Extraorganizational
•P r e e x i s t i n g e v a l u a t i o n bounds .Evaluator's approach .Evaluator credibility .Organizational factors
Context
style
• Information content and r e p o r t i n g
• Administrator
of users
Daillak, & White (1979, p. 235)
• Orientation
Alkin,
Information Used
Purpose
User
Dimensions
Earlier Conceptual
N
m
0
(,1
m
U
g
e
0
g
o
C
O
~o
180
J. A. King
B u t , as t h e o t h e r d i m e n s i o n s o f T a b l e 1 s u g g e s t , t h i s is o n l y p a r t o f t h e use picture. T h e s e c o n d p o i n t t h a t m u s t be s t r e s s e d is t h a t use c a n n o t be d i s c u s s e d outside of the context of that use. T h e c o n t e x t s h a p e s a n d l i m i t s w h a t can be done with information, and without a thick description of the evaluation context, i t is m e a n i n g l e s s to d i s c u s s u s e . T h e i m p o r t a n c e o f c o n t e x t is e v i d e n t in t h e f a c t t h a t f i v e o f t h e e i g h t f a c t o r s in A l k i n , Daillak, and White's analytic framework o f local e v a l u a t i o n use, derived from naturalistic case s t u d i e s , r e l a t e to t h e evaluation setting. It s h o u l d also be n o t e d t h a t t h e t w o e v a l u a t o r f a c t o r s - - a p p r o a c h and credibility-h a v e b e e n i n c l u d e d u n d e r c o n t e x t ; in t h i n k i n g a b o u t u s e , we m u s t r e c o g n i z e t h a t t h e e v a l u a t o r becomes s i m p l y a p a r t o f t h e u s e r ' s e n v i r o n m e n t . This appropria t e l y r e d u c e s t h e e v a l u a t o r f r o m t h e k e y c h a r a c t e r in e v a l u a t i o n use to m e r e l y an i m p o r t a n t c o m p o n e n t o f t h e use s e t t i n g . B y a c t i n g in c e r t a i n w a y s , e v a l u a t o r s m a y p r o m o t e o r i n h i b i t u s e ; b u t , as n o t e d a b o v e a n d in K e n n e d y , A p l i n g , a n d N e u m a n n ( 1 9 8 0 ) , e v a l u a t i o n use is to a l a r g e e x t e n t o u t o f e v a l u a t o r s ' h a n d s . G i v e n t h a t t h i s is o f t e n t h e c a s e , t h e q u e s t i o n t h e n b e c o m e s , w h a t can we do about this? Studying local e v a l u a t i o n use m a y lead to a n e w a w a r e n e s s o f t h e dimensions of use. B y o p e n i n g o u r e y e s to t h e w a y s i n f o r m a t i o n is c r e a t e d a n d p r o c e s s e d in c o n t e x t , q u e s t i o n s w i l l e m e r g e e n a b l i n g us to f o c u s on t h e n a t u r e o f use. T h e s t u d y I w i l l d e s c r i b e h e r e is a t t e m p t i n g b o t h t o r a i s e s u c h q u e s t i o n s a n d to b e g i n p r o v i d i n g a n s w e r s b y l o o k i n g at use as a p r o c e s s , n o t j u s t as a product.
The C u r r e n t Study O u r s t u d y is a d d r e s s i n g t w o q u e s t i o n s : ( 1 ) What p r o c e s s e s a r e i n v o l v e d in t h e g e n e r a t i o n a n d use o f i n f o r m a t i o n in t h e r e s e a r c h a n d e v a l u a t i o n u n i t o f a l a r g e p u b l i c school s y s t e m ? a n d ( 2 ) What is t h e v a l i d i t y of t h e f a c t o r s i d e n t i f i e d in previous r e s e a r c h f o r a n a l y z i n g t h e a c t i o n s o f e v a l u a t i o n u s e r s w i t h i n a local educational agency? The study will have two phases. The phase currently u n d e r w a y is an i n t e n s i v e case s t u d y o f t h e r e s e a r c h a n d e v a l u a t i o n u n i t o f a l a r g e c i t y s c h o o l d i s t r i c t , using Guba's wave conceptualization of inquiry (Guba, 1978). Moving between discovery and verification modes, this phase will ultimately include three "waves." T h e p u r p o s e o f t h e f i r s t w a v e is t o c r e a t e a p o r t r a i t o f t h e w o r k i n g s o f t h e r e s e a r c h a n d e v a l u a t i o n u n i t as seen b y its s t a f f , i . e . , b y t h o s e d i r e c t l y i n v o l v e d in d a t a c o l l e c t i o n a n d p o t e n t i a l u s e . B e c a u s e t h e d e s i g n is e m e r g i n g as t h e s t u d y p r o g r e s s e s , i t is d i f f i c u l t t o p r e s e n t i t in d e t a i l at t h i s t i m e . H o w e v e r , we h a v e b e g u n b y c o n d u c t i n g e x t e n s i v e i n t e r v i e w s w i t h t h e R&E s t a f f a n d k e y a g e n t s w h o interact with the evaluation personnel and process, and by observing day-to-day l i f e in t h e o f f i c e ( a t t e n d i n g m e e t i n g s , c o n d u c t i n g o n - s i t e v i s i t s , c o m p l e t i n g d a t a forms, interacting with the system's administrative structure, etc.). In a d d i t i o n , we w i l l e x a m i n e all a v a i l a b l e d o c u m e n t s a n d r e p o r t s u n t i l t h e s t r u c t u r e of t h i s d y n a m i c c o n t e x t is r e v e a l e d . The write-up of this description and the ensuing member check will conclude the first wave and will present a picture of evaluation use in process. T h e second w a v e of the case study will m o v e further from the internal perceptions of the R & E unit into the context in which that use occurs. T w o books-Patton et al. (1977) a n d Alkin, Daillak, a n d White (1979)--suggest the importance of this move. B a s e d on our findings in the first wave, w e will administer a questionnaire to instructional supervisors a n d coordinators, to all local principals, a n d to a r a n d o m sample of 500 teachers. W e will also interview key central administration decision m a k e r s along with selected supervisors, principals, teachers, and students to determine the w a y s they think about and use information
181
Studying the Local Use of Evaluation
ge n e ra t e d by the e v a l u a t i o n process in both f e d e r a l l y and locally f u n d e d p r o grams. At the completion of this wave, the e a r l i e r d e s c r i p t i o n will be elaborated into a draft of our case study. M o v i n g even further into the context surrounding the use of local evaluation results, the third w a v e will include interviews with m e m b e r s of the local Board of Education and with appropriate representatives of the State D e p a r t m e n t of Education. T h e first phase of the project will then conclude with the fully developed case study. T h e second phase of the project will seek to integrate the theory of evaluation use developed from practice with the extant, empirically based "theory" of use in the literature. Recognizing the limits of studying a single case, w e will test the theory derived from the case study against the empirical and real world theory derived from the literature and from L E A practitioners. This will clarify and elaborate on our findings about evaluation use in public schools. A copy of the case study and the derived theoretical f r a m e w o r k will be sent to these experts along with a structured set of interview questions. Telephone interviews will be conducted using these questions as guides, but allowing the interviewees to discuss in general the question of evaluation use in public schools. T h e project combines several approaches in studying the process of evaluation information use and in beginning to validate a theory of such use. T h e case study methodology of the first phase combines ethnographic techniques with interviews and s u r v e y methods to both reveal and test the actual processes involved in the use of evaluation information in terms of the interplay a m o n g key factors, actors, and agents. Rather than tracing the impact of a single study or data set, w e will examine the use and effects of studies done in the past, those ongoing, and those projected for the future. T h e validation procedures of the second phase will address questions of the applicability of our results and, w e hope, will promote discussion of critical issues in evaluation use by practitioners and scholars in the field.
REFERENCES Alkin,
M.C.
using
naturalistic
research
for
the
study
of
evaluation.
Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, 1979. (ERIC_, No. 170 378) Alkin,
M.C.,
Daillak,
make a difference?
R.H., & White, P. Using e v a l u a t i o n s : Beverly Hills, Calif. : Sage Publications,
Does e v a l u a t i o n
1979.
Andrews, J.V. Reactions to program e v a l u a t i o n : A q u a l i t a t i v e analysis (Doctoral d i s s e r t a t i o n , Cornell U n i v e r s i t y , 1979). Dissertation Abstracts International, 1979, 39, 6717A. ( U n i v e r s i t y Microfilms No. 79-10, 731) B r o w n , R.D. Evaluation utilization: A critique. Unpublished manuscript,
A literature
r e v i e w and r e s e a r c h
agenda,
1981.
In B r o w n , R . D . , & Braskamp, L . A . Summary: C o m m o n themes and a checklist. L . A . Braskamp & R.D. Brown ( E d s . ) , U t i l i z a t i o n of e v a l u a t i o n information. San Francisco: Jossey-Bass, 1980.
Caplan, N.
What do we know about knowledge u t i l i z a t i o n ?
R.D. B r o w n (Eds.), Jossey-Bass, 1980.
Utilization
of e v a l u a t i o n
In L . A .
information.
San
Braskamp & Francisco:
182
J. A. King
Caplan,
N.,
Morrison,
ledge in policy
A.,
decisions
& Stambaugh,
R.J.
at the national
The use o f s o c i a l level: A report to
s c i e n c e knowrespondents.
Ann A r b o r : C e n t e r f o r Research on U t i l i z a t i o n of S c i e n t i f i c K n o w l e d g e , Institute for Social Research, University of Michigan, 1975. (ERIC No. 111 729) Cronbach, L.J., Ambron, S.R., Dornbusch, S.M., P h i l l i p s , D . C . , Walker, D . F . , & Weiner, S.S. evaluation. San Francisco: J o s s e y - B a s s , 1980. David,
J.
Analysis,
Hess,
R.D.,
Toward
Local uses of T i t l e I e v a l u a t i o n s . E d u c a t i o n a l 1981, 3, 27-39.
Hornik,
reform
of
Evaluation
R.C., program
and P o l i c y
D i c k e y , B. U t i l i z a t i o n of e v a l u a t i o n s of small scale educational p r o j e c t s ( D o c t o r a l d i s s e r t a t i o n , U n i v e r s i t y of Minnesota, 1979). Dissertation Abstracts International, 1979, 40, 3245A. ( U n i v e r s i t y Microfilms No. 79-26, 115) Fullan, M. C o n c e p t u a l i z i n g p r o g r a m s of curriculum implementation. Paper p r e sented f o r the Symposium on C u r r i c u l u m I n q u i r y in Canada, V i c t o r i a , 1979. Glaser,
B.G.,
qualitative
& Strauss, A.L. D i s c o v e r y of grounded research. Chicago: Aldine, 1967.
theory:
Strategies
Guba, E.G. Toward a methodology of n a t u r a l i s t i c inquiry in educatinal Los Angeles: C e n t e r f o r the S t u d y of E v a l u a t i o n , 1978. Guba, E . G . , 1981. Kennedy,
& Lincoln, Y . S .
M.M.,
information
Apling,
R.,
Effective
evaluation.
& Neumann, W.F. Cambridge,
in p u b l i c schools.
San Francisco:
The r o l e
Mass. :
of
The
for
evaluation.
Jossey-Bass,
evaluation
Huron
and t e s t
Institute,
1980. King,
J.A.,
& T h o m p s o n , B . A n a t i o n w i d e survey of administrators' perceptions Paper p r e s e n t e d at the Annual Meeting of the American Educational Research Association, Los A n g e l e s , 1981.
of
King,
evaluation.
J.A., school
Use). Lyon,
T h o m p s o n , B . , & Pechman, E.M. I m p r o v i n g e v a l u a t i o n use i n l o c a l (Final r e p o r t of NIE G r a n t 80-0082, O p t i m i z i n g Evaluation New O r l e a n s : New Orleans Public Schools, June 1981.
settings
C.D.,
districts.
Doscher, L . , McGranahan, P., & Williams, R. E v a l u a t i o n and s c h o o l Los A n g e l e s : C e n t e r f o r the S t u d y of E v a l u a t i o n , 1978.
Newman, D . L . , Brown, R.D., & the u t i l i z a t i o n of e v a l u a t i o n . Utilization
Patton, M.Q.
of evaluative
Qualitative
Braskamp. L.A. Communication t h e o r y and In L . A . B r a s k a m p & R . D . Brown ( E d s . ) , information. San Francisco: Jossey-Bass. 1980.
evaluation
methods.
Beverly Hills: Sage, 1980.
Patton, M . Q . , Grimes, P . S . , G u t h r i e , K . M . , B r e n n a n , N . J . , Grench B . D . , & Blyth, D.A. In search of impact: An analysis of u t i l i z a t i o n of federal health e v a l u a t i o n r e s e a r c h . In C . H . Weiss ( E d . ) , U s i n g s o c i a l r e s e a r c h i n public policymaking. L e x i n g t o n , Mass.: D . C . Heath, 1977. Pelz,
D.C. Some e x p a n d e d p e r s p e c t i v e s on use of social science in p u b l i c p o l i c y . In J.M. Yinger & S.J. Cutler (Eds.), Major social issues, a m u l t i d i s c i p l i n a r y view. N e w York: The Free Press, 1978.
Studying the Local Use of Evaluation
Rich,
183
R.F. Uses of social science i n f o r m a t i o n b y f e d e r a l b u r e a u c r a t s : Knowledge f o r action v e r s u s k n o w l e d g e f o r u n d e r s t a n d i n g . In C . H . Weiss ( E d . ) , Using social research in public p o l i c y m a k i n g . Lexington, Mass. : D.C. Heath, 1977.
Simon, H.A. Smith,
New
Administrative behavior.
N.L.
The
president's
corner.
York:
MacMillan, 1957.
Evaluation News,
Summer,
1980,
24-25.
Stake, R. Evaluation design, instrumentation, data collection and analysis of d a t a Urbana, III.: Center for Instructional Research and Curriculum Evaluation, 1976. Thompson,
B.,
&
King,
J.A.
A critique
of e v a l u a t i o n - u s e research
methods.
CEDR Quarterly, in press.
Weiss, C . H .
Conceptual issues in measuring the u t i l i z a t i o n evaluation. Paper p r e s e n t e d at the Annual Meeting
of r e s e a r c h
of
the
and
Evaluation
Research Society, Minneapolis, 1979. Wilensky, H.L., & Lebeaux, C.N. N e w York: Russell Sage, 1958. Wise,
Industrial
society
and social welfare.
R.I. W h a t w e k n o w a b o u t the d e c i s i o n - m a k i n g and decision settings. Paper p r e s e n t e d at t h e Annual Meeting of the American Educational Research A s s o c i a t i o n , T o r o n t o , 1978.
THE JEAN KING Louisiana.
is
Professor
of
AUTHOR
Education
at
Tulane
University,
New
Orleans,