- Email: [email protected]
Fostering utilization through multiple data gathering methods
Studies in Educational Evaluation, Vol. 14, pp. 113-133, 1988 Printed in Great Britain. All rights reserved.
0191-491X/88 $0.00 + .50 Copyright © 1988 Pergamon Press plc
FOSTERING UTILIZATION T H R O U G H MULTIPLE DATA GAT HERI N G M E T H O D S Susan D. Turner, Jayne Hartman, Lore A. Nielsen and Judith Lombana Hi//sborough County Public Schools, Tampa, Florida
Introduction To b o r r o w a p h r a s e from the i n c o m p a r a b l e Casey Stengel, the s t u d y of e v a l u a t i o n utilization h a s "...come along slow, b u t fast." Casey's r e m a r k referred, of course, to baseball's Miracle Mets w hose u s u a l plodding s e a s o n s in th e cellar t u r n e d a r o u n d s u d d e n l y in t he s u m m e r of 1969. While our s u b j e c t pales beside the "old professor's" in t e r m s of d r a m a a n d excitement, his m e s s a g e c a n be a p p r o p r i a t e l y applied to the p a s t d e c a d e ' s r e s e a r c h into a n d d is cu s s io n of evaluation utilization. Who first s p o t t e d t h e u n u s e d e v a l u a t i o n r e p o r t s in social service, e d u c a t i o n , a n d g o v e r n m e n t a g e n c y b a s e m e n t s , or w ho first l a m e n t e d their idleness is u n k n o w n . In the p a s t t e n years, however, dozens of articles have b e e n w r i t t e n , p a p e r s p r e s e n t e d , a n d s t u d i e s c o n d u c t e d to d i s c o v e r and d i s c u s s factors w hi ch influence utilization. Most recently, t he C e n t e r for the S t u d y of E v a l u a t i o n at UCLA h a s developed a tri-category analytic framework with t h e p u r p o s e of pr ovi di ng a formal s t r u c t u r e for u n d e r s t a n d i n g and c o n d u c t i n g r e s e a r c h on ut i l i z a t i on (Alkin, 1984). W i t h i n t h r e e general c ateg o r ies (context factors, h u m a n factors and e v a l u a t i o n factors) multiple d e s c r i p t o r s comprise the framework which is b a s e d on a review and analysis of t h e u t i l i z a t i o n l i t e r a t u r e , s e v e r a l c a s e s t u d i e s a n d t h e c o n t r i b u t o r s ' e x p e r i e n c e s in c o n d u c t i n g e v a l u a t i o n s , B e c a u s e it is i n t e n d e d to guide practitioners', theoreticians' and r e s e a r c h e r s ' efforts in s u b s e q u e n t y e a r s and h a s b e e n d i s t r i b u t e d for t h a t p u r p o s e , t he fram ew ork, s h o w n in figure 1, 113
114
S. D. Turner et aL
s h o u l d be a s inclusive as possible. A. S e t t i n g t h e S t a g e Pre-existing evaluation b o u n d s User identity Project characteristics Organizational features B.
Identifying/Organizing the Participants User characteristics: * i n t e r e s t in e v a l u a t i o n * c o m m i t m e n t to u s e * p r o f e s s i o n a l style Evaluator characteristics: * background and identity *
credibility
* c h o i c e of role * w i l l i n g n e s s to involve u s e r s * rapport with users * c o m m i t m e n t to u s e * political s e n s i t i v i t y E v a l u a t i o n p r o c e d u r e s -- p l a n C.
Operationalizing the Interactive Process E v a l u a t i o n p r o c e d u r e s -- e x e c u t i o n S u b s t a n c e of e v a l u a t i o n i n f o r m a t i o n E v a l u a t o r c o m m i t m e n t to u s e I n f o r m a t i o n d i a l o g u e -- f o r m a t i v e User information processing preferences
D. A d d i n g t h e F i n i s h i n g T o u c h e s Evaluation reporting Evaluation characteristics: * c o m m i t m e n t to u s e * political s e n s i t i v i t y I n f o r m a t i o n d i a l o g u e -- s u m m a t i v e U s e r c o m m i t m e n t to u s e S o u r c e : Alkin, 1 9 8 4 : 9 1 F i g u r e I: F a c t o r P a t t e r n for E v a l u a t i o n Use T h e p u r p o s e of t h i s p a p e r is to p r o p o s e t h e a d d i t i o n of o n e f a c t o r to the framework:
m u l t i p l e x i t y (variety) of d a t a g a t h e r i n g m e t h o d s .
Toward
Fostering Utilization
115
this end, two d i s t i n c t d i s c u s s i o n s comprise the paper. First, supportive a r g u m e n t s for the a s s u m p t i o n t h a t this factor s u b s t a n t i a l l y c o n t r i b u t e s to utilization are p r e s e n t e d and relevant literature is cited. Then, to provide preliminary evidence of the factor's legitimacy, a n exploratory s t u d y of four highly-utilized educational program evaluations is described. Before beginning a d i s c u s s i o n of the declared topic, some prefatory r e m a r k s are n e c e s s a r y to familiarize r e a d e r s with o u r experiences a n d p o s i t i o n s (i.e., biases) r e g a r d i n g the s t u d y of utilization. As i n t e r n a l evaluators in a large school district, we know first-hand the t r u t h of Alkin's, Daillak's a n d White's (2979: 233) statement, "...the degree of utilization of the e v a l u a t i o n a p p a r e n t l y is not d e t e r m i n e d by mere c h a n c e b u t is, to a considerable extent, associated with certain characteristics of the evaluation situation." Plainly, we recognize the decisive interpersonal factors frequently associated with utilization (Patton et al., 1977; Patton, 1978, Gurel, 1975). We are h e s i t a n t , however, to e n d o r s e " o r i e n t a t i o n of t h e evaluator" as "perhaps the m o s t influential [factor] in determining w h e t h e r utilization will occur" (Alkin a n d Law, 1980: 79). E n h a n c i n g utilization is an accepted p a r t of our jobs. While we occasionally (albeit infrequently) are t e m p t e d by Harry T r u m a n ' s notion, "I have found the best way to give advice.., is to find o u t w h a t t h e y w a n t and advise t h e m to do it," the professional literature a n d daily occurrences have proved the m o s t valuable resources in moving toward t h a t goal. As t h e l i t e r a t u r e h a s t e n d e d toward e n d l e s s lists of h y p o t h e s i z e d utilization factors (Oman a n d Chitwood, 1984; Leviton a n d Hughes, 1981; Dickey, 1980; H a n s e n , Martin a n d Oxford, 1980; Alkin, DaiUak a n d White, 1979; Young, 1978; Cox, 1977), we have become increasingly skeptical of the direction a n d value of utilization s t u d y . In the end, we are inclined toward the conviction of Patton and his associates (1977:142). The i s s u e at this time is n o t the s e a r c h for a single f o r m u l a of utilization success, nor the generation of ever-longer lists of possible factors affecting Utilization. The t a s k for the p r e s e n t is tO identify and define a few key variables t h a t m a y m a k e a m a j o r difference in a significant n u m b e r of evaluation cases. Realizing, however, the need for a n d potential value of factor frameworks in guiding, c o n d u c t i n g a n d applying research, we e m b r a c e Alkin's c u r r e n t s t r u c t u r e a n d choose to suggest minor revisions a n d modifications as it is implemented.
116
S. D. Turner et aL
Supportive A r g u m e n t s We w h o l e h e a r t e d l y agree with Connolly a n d Port er (1980) who noted t h a t c h a r a c t e r i s t i c s a s s o c i a t e d with ut i l i zat i on are i n t i m a t e l y i nt ert w i ned. Not one, b u t c o n s t e l l a t i o n s of f a c t o r s c o n t r i b u t e in di fferent s i t u a t i o n s to utilization (Alkin, Daillak, & White, 1979). In a r g u i n g for the i ncl usi on of m u l t i p l e x i t y of d a t a g a t h e r i n g m e t h o d s , we are aware t h a t t he p r e s e n c e of this factor could be inferred to exist a l r e a d y within t he f r a m e w o r k b e c a u s e of its r e l a t i o n s h i p , d i r e c t l y or s u b t l y , to o t h e r e s t a b l i s h e d f a c t o r s . T he t r a d i t i o n a l f u n c t i o n s of multiple d a t a g a t h e r i n g m e t h o d s have b e e n t hose of v alid atio n a n d believability of findings (Patton, 1978; B u n k e r , 1978). Aside from t h e s e p e d e s t r i a n motives, we m a i n t a i n t h a t t he u s e of m u l t i p l e d a t a g a t h e r i n g m e t h o d s c o n t r i b u t e s u n i q u e l y to utilization, u n r e l a t e d to t h r e a t s to validity. Th e following t h e s e s explicate this belief. Multiplexity of d a t a g a t h e r i n g m e t h o d s c a n i n f l u e n c e i n v o l v e m e n t of p r o g r a m d e v e l o p e r s , p a r t i c i p a n t s a n d deci si on m a k e r s in t h e e v a l u a t i o n p ro ces s . Waller (1979) f o u n d t h a t i nvol vem ent of t he u s e r w as the p r i m a r y f acto r a s s o c i a t e d with utilization. Active involvement i n c r e a s e s c o m m i t m e n t to utilization of findings b e c a u s e u s e r s are a n integral p a r t of the evaluation's i m p l e m e n t a t i o n . As r e p o r t e d by O m a n and Chitwood (1984), m a n a g e m e n t s t u d i e s w h i c h i n c l u d e d an interactive p r o c e s s b e t w e e n a n a l y s t and involved u n i t p e r s o n n e l s h o w e d a b o v e - a v e r a g e levels of a c c e p t a n c e . Previously, Leviton a n d H u g h e s (1981) h a d r e p o r t e d positive r e l a t i o n s h i p s b e t w e e n the level of u s e r i n v o l v e m e n t a n d ut i l i z a t ion. In a d d i t i o n , Cooley's (1983) " c l i e n t - o r i e n t a t i o n " a n d P a t t o n ' s (1982) " p r o c e s s - o r i e n t e d " a p p r o a c h e s to e valu atio n in clude this active involvement element. A v a r i e t y of d a t a g a t h e r i n g m e t h o d s c a n i m p a c t c o m m u n i c a t i o n b e t w e e n p r o g r a m decision m a k e r a n d e v a l u a t o r a n d b e t w e e n e v a l u a t o r and p r o g r a m p a r t i c i p a n t s . One i nve s t i gat i on involving m e n t a l h e a l t h agencies found that more useful studies featured more frequent communication b e t w e e n e v a l u a t o r a n d staff (Windle and Bates, 1974). Similarly, Glaser and T ay lo r (1973) c o n c l u d e d t h a t s u c c e s s f u l r e s e a r c h s t u d i e s were distinguished from u n s u c c e s s f u l ones b y close c o m m u n i c a t i o n b e t w e e n r e s e a r c h e r s and consumers. F r e q u e n t c o n t a c t b e t w e e n e v a l u a t i o n p r i n c i p a l s c a n alter d e c i s i o n m a k e r a n d e v a l u a t o r p e r c e p t i o n s of one a n o t h e r . C o n n o l l y a n d P o r t e r (1980) a d v a n c e d t h e i d e a t h a t p r e v i o u s i n t e r a c t i o n s b e t w e e n e v a l u a t o r a n d m a n a g e r i nf l ue nc e t h e t r u s t place in the e v a l u a t i o n findings, a n d t h a t i n c r e a s e d t r u s t l eads to utilization. F u r t h e r , t h e y a r g u e d t h a t
Fostering Utilization
117
t w o - w a y c o m m u n i c a t i o n b e t w e e n m a n a g e r a n d e v a l u a t o r is c r u c i a l to evaluation utilization. Multiplexity of d a t a g a t h e r i n g m e t h o d s can affect credibility of the evaluator a n d evaluation findings. Leviton and Hughes (1981) suggested t h a t a l t h o u g h evidence of its effect on utiliztion is mixed, credibility of the producer of a n evaluation is likely to be important. Use of multiple methods a n d m e a s u r e s allows t h e e v a l u a t o r i n c r e a s e d o p p o r t u n i t i e s in w h i c h to d e m o n s t r a t e h i s / h e r abilities and skills. As a result, the p r o g r a m decision m a k e r comes to rely on the evaluator as a primary source of information. Multiple d a t a gathering m e t h o d s c a n be associated with the visiblity the project a t t a i n s with a n u m b e r of different audiences. B u n k e r (1978:132) c o n t e n d e d t h a t "both a c c o u n t a b i l i t y a n d p r o b a b i l i t y of u t i l i z a t i o n are increased by wider distribution of findings." Similarly, in a s t u d y for the General Accounting Office, Young (1978) noted d i s s e m i n a t i o n a m o n g issues t h a t are i m p o r t a n t to utilization. S o m e w h a t d i s c o u n t i n g the relationship between d i s s e m i n a t i o n (i.e., visibility) a n d utilization, Dickey (1980) pointed out the general agreement in the literature on the value of dissemination b u t she reported finding no research d o c u m e n t i n g this relationship. Evaluator u n d e r s t a n d i n g of a project can be influenced by the n u m b e r a n d variety of data-gathering methods. If a decision m a k e r is removed from the evaluation process, h i s / h e r information needs m a y be unclear. In this case, the u s e of multiple d a t a collection m e t h o d s could act to circumvent u n c e r t a i n t y or misconceptions about questions to be investigated. Oman and Chitwood (1984) reported t h a t studies revolving a r o u n d elaborate large scale q u e s t i o n s h a d a lower level of a c c e p t a n c e t h a n s t u d i e s of more focused i s s u e s . Likewise, Connolly a n d Porter (1980) c o m m e n t e d t h a t decision m a k e r u s e of e v a l u a t i o n i n f o r m a t i o n is more likely if the findings are produced in response to a specific information need. Dickey (1980) a n d Cox (1977) b o t h reported i n s t a n c e s in w h i c h decision m a k e r s were u n a b l e to utilize findings b e c a u s e their questions were not a d d r e s s e d by the studies' designs and procedures. The u s e of m u l t i p l e m e a s u r e s improves the evaluator's ability to u n d e r s t a n d b o t h major and minor project issues, talk more fluently a b o u t the project a n d r e p r e s e n t it more accurately. Also. it allows the evaluator to anticipate questions t h a t m a y arise during the course of the project a n d have readily available data for discussion. Multiplexity of d a t a gathering m e t h o d s can influence the probability t h a t a methodological mix of collection strategies a n d reporting techniques
118
S. D. T u r n e r et a/.
will occur. Th e u s e of a variety of m e t h o d s p r o m o t e s wide appeal to assorted d e c i s i o n - m a k e r styles. O m a n a n d Chitwood (1984) r e p o r t e d s t u d i e s t h a t s h o w differences in a c c e p t a n c e of m a n a g e m e n t r e p o r t s b a s e d on m e t h o d s of i n f o r m a t i o n collection. Along s i m i l ar lines, V an de Vall et al (1976:170) c o n c l u d e d t h a t m o r e utilization will o c c u r for m a n a g e m e n t r e s e a r c h st udi es w h i c h e m p l o y "a hi gh r a t e of e c l e c t i c i s m in t h e e x p l o r a t o r y st age of a project." According to C ohen (1977), h e a l t h care a d m i n i s t r a t o r s v a r y in the d e g r e e to w h i c h t h e y v a l u e a n d u n d e r s t a n d scientific i n f o r m a t i o n . He implied t h a t s u n d r y m e t h o d s of collection a n d r e p o r t i n g m i g h t a s s i s t in i n cr eas in g the i m p a c t of evaluation information. Based on a s t u d y of a group of e d u c a t i o n a l s u b j e c t s , Dickey (1980:76) c o n c l u d e d "If the decision m a k e r d oes n o t u n d e r s t a n d or see as a p p r o p r i a t e t he w a y in w h i c h d a t a are collected, t h e n belief in the d a t a is likely to be d i m i n i s h e d a n d utilization of the ev alu atio n low." Her c o m m e n t s regarding t he face value of validity in the c o n d u c t of e v a l u a t i o n s give f u r t h e r s u p p o r t to t he n o t i o n t h a t multiplexity of d a t a gathering m e t h o d s m a y be an i m p o r t a n t variable in utilization. A v a r i e t y of d a t a g a t h e r i n g m e t h o d s m a y i m p a c t t h e s c o p e of an evaluation. In a n article regarding this topic, B u n k e r (1978) pointed out t h a t e v a l u a t i o n utility a n d c o m p r e h e n s i v e n e s s are directly related. Specifically, he c o m m e n t e d t h a t m u l t i p l e d a t a s o u r c e s , m u l t i p l e t y p e s of dat a, a n d d a t a collection over time i n c r e a s e p r o s p e c t s for s u c c e s s in utilizing e v a l u a t i o n information.
Program Evaluation Descriptions To t e s t s u p p o r t for proposing the multiplexity factor for t he utilization framework, four ongoing highly-utilized educational program evaluations were e x a m i n e d .
Brief d e s c r i p t i o n s of e a c h follow.
S u m m e r I n s t i t u t e Pr ogr am As a r e s u l t of t h e F l o r i d a legislative a c t i o n , s u m m e r i n - s e r v i c e i n s t i t u t e s in science and m a t h e m a t i c s were held a c r o s s t he st at e d u r i n g the s u m m e r of 1984. In H i l l s bor ough C o u n t y a p p r o x i m a t e l y 4 0 0 e l e m e n t a r y , j u n i o r high, a n d s e n i o r high t e a c h e r s part i ci pat ed in one of eighteen training p r o g r a m s r e l a t e d to t h e i r 1 9 8 4 - 8 5 t e a c h i n g a s s i g n m e n t s . Institutes i n c l u d e d i n s t r u c t i o n in t he following areas: c u r r e n t c o n t e n t i n f o r m a t i o n , c o n t e n t p r e p a r a t i o n c o n s i s t e n t with t h e Fl ori da S t a n d a r d s of Excellence, u p d a t i n g in q u a n t i t a t i v e l a b o r a t o r y a n d field d a t a - g a t h e r i n g t e c h n i q u e s ,
Fostering Utilization
119
i n f o r m a t i o n on related careers, a n d clinical practice in t e a c h i n g c o n t e n t learned in the institute. S t r u c t u r e a n d c o n t e n t of the s i x t y - h o u r institutes were o u t l i n e d by d i s t r i c t i n s t r u c t i o n a l s u p e r v i s o r s c o l l a b o r a t i n g with university faculty. Teachers who a t t e n d e d the i n s t i t u t e s were paid for their p a r t i c i p a t i o n a n d were eligible to e a r n credit t o w a r d s r e n e w a l of their t e a c h i n g certificates. Overall, more t h a n n i n e t y percent of the participants completed their program successfully. Multiple d a t a sources were u s e d in the program evaluation to provide p r o g r a m d e s i g n e r s w i t h i n f o r m a t i o n to modify t h e ongoing p r o g r a m in s u b s e q u e n t s u m m e r s and to gather outcome information a b o u t the program effects at three levels: immediate knowledge gain of participants, impact of t h e i n s t i t u t e s on i n - s c h o o l i n s t r u c t i o n , a n d a n y effects on s t u d e n t achievement. T e s t r e s u l t s a n d s u c c e s s f u l p r o g r a m c o m p l e t i o n were s u m m a r i z e d by the institutes as well as t h r o u g h use of extensive participant opinonnaires. The impact on instruction was a s s e s s e d by m a i l - o u t surveys a n d Concerns-Based Adoption Model (CBAM) Levels of Use interviews with a s t r a t i f i e d r a n d o m s a m p l e of a p p r o x i m a t e l y twenty-five p e r c e n t of the p a r t i c i p a n t s . For one of the i n s t i t u t e s in which there was a n identifiable m a t c h between institute c o n t e n t and course content, s t u d e n t s were given a subject area achievement test and their scores c o m p a r e d to a control group from the previous year to gauge effect on s t u d e n t learning. Data gathering activities a n d reporting f u n c t i o n s were s c h e d u l e d over a year's time and a s u b s e q u e n t utilization survey of program designers indicated t h a t not only did evaluation activities fulfill state reporting r e q u i r e m e n t s , b u t t h e y were u s e d to improve the 1985 s u m m e r institute as well. C h a p t e r 1 Reading Microcu0mputer C o m o o n e n t In operation for the third year, the C h a p t e r 1 reading microcomputer c o m p o n e n t is b e i n g i n t e g r a t e d into 116 s e p a r a t e c l a s s r o o m s w i t h i n s e v e n t y - f o u r H i l l s b o r o u g h C o u n t y e l e m e n t a r y s c h o o l s covering g r a d e s three-six. Over 3,500 Chapter 1 s t u d e n t s are given additional reinforcement of reading skills t h r o u g h use of c o m p u t e r - a s s i s t e d instruction. Additionally, these s t u d e n t s are exposed to c o m p u t i n g skills which previously h a d been reserved for s t u d e n t s in above average groupings. The d i s t r i c t - w i d e i m p l e m e n t a t i o n of t h e r e a d i n g m i c r o c o m p u t e r c o m p o n e n t is d e s i g n e d in two p h a s e s : planning and training. Pilot e v a l u a t i o n d a t a provided p r o g r a m developers w i t h t h e f r a m e w o r k for a modified district plan for y e a r two. The t r a i n i n g of t e a c h e r s a n d aides is
120
S. D, T u r n e r et a/.
divided in to six levels with i n s t a l l a t i o n of h a r d w a r e at t h e s c h o o l sites following Level II. Following a carefully c o n s t r u c t e d e v a l u a t i o n model which i n c o r p o r a t e s c o m p o n e n t s of t h e C o n c e r n s - B a s e d A d o p t i o n Model (Hall, 1979), the district p h a s e was initiated in y e a r two and c o n t i n u e s to date. Th e P r o c e d u r e s for Adopting E d u c a t i o n a l I n n o v a t i o n s Project at the R & D C e n t e r for T e a c h e r E d u c a t i o n developed a m odel e m p h a s i z i n g i n n o v a t i o n a d o p t i o n , t h e C o n c e r n s - B a s e d A d o p t i o n Model (CBAM), t h a t a s s u m e s c h a n g e to be a highly p e r s o n a l and l e n g t h y process, one t h a t a f f e c t s i n d i v i d u a l s differently. T he m o d e l h y p o t h e s i z e s two d i m e n s i o n s a l o n g w h i c h i n d i v i d u a l s grow as t h e y b e c o m e m o r e f a m i l i a r w i t h a n d s o p h i s t i c a t e d in u s i n g i n n o v a t i o n s : S t a g e s of C o n c e r n A b o u t t h e I n n o v a t i o n (SoC) a n d Levels of Use of t h e I n n o v a t i o n (I~U). T h e S t a g e s of C o n c e r n d i m e n s i o n f o c u s e s on t he feelings, t h o u g h t s a n d i n f o r m a t i o n n e e d s of t he i n n o v a t i o n "user." Indi vi dual s a p p e a r to be mo r e s e l f - c o n c e r n e d w h e n t h e y first find out a b o u t an innovation, wondering w h a t it will r e q u i r e of t h e m , h o w t h e i r roles m u s t c h a n g e . c o n c e r n s ar e resolved, m o r e " t a s k " - o r i e n t e d c o n c e r n s emerge.
As t h e s e Questions
a b o u t w h a t m a t e r i a l s are n e e d e d daily a n d h o w to s c h e d u l e time m ore effectively are typical. Later c o n c e r n s focus on how t he i n n o v a t i o n affects s t u d e n t s , h o w to c o m b i n e efforts or even replace t he i n n o v a t i o n to e n h a n c e s t u d e n t learning. this p r o g r e s s i o n .
The CBAM des c r i bes seven Stages of C o n c e r n t h a t reflect
T h e o t h e r d i m e n s i o n of t he CBAM, Levels of Use of t he I n n o v a t i o n (LoU), d e s c r i b e s t he b e h a v i o r of i ndi vi d ual s as t h e y b e c o m e m o r e familiar with a n d m o r e skilled in u s i n g an innovation. E a c h of t h e eight identified Levels of Use focuses on behavior t h a t is c h a r a c t e r i s t i c of the innovation u s e r at a p a r t i c u l a r stage of development. For example, at LoU II (Preparation), t he individual is p r e p a r i n g for first use. This involves g a t h e r i n g r e s o u r c e s , finding out detailed requirements, arranging the physical setting and s c h e d u l i n g initial steps. At e a c h of t he eight levels, specific b e h a v i o r s have b e e n defined t h a t de s cr i be c h a r a c t e r i s t i c s of the i n n o v a t i o n user. (Loucks, Newlove, & Hall, 1975, pp. 1-2). Evaluation is i n t e g r a t e d i n t o all p h a s e s of t h e p r o j e c t ' s i m p l e m e n t a t i o n a n d e m b o d i e s t h e o n g o i n g a p p l i c a t i o n of t h e CBAM. Multiple d a t a s o u r c e s are u s e d c o n t i n u o u s l y on a p l a n n e d review cycle to provide f e e d b a c k to t he p r o g r a m developers. Prior to t he s t a r t of in-service
Fostering Utilization
121
training, a configuration was developed w h i c h defined critical c o m p o n e n t s for y e a r two. From this configuration, a checklist was generated and is used on visits by t h e m i c r o c o m p u t e r t r a i n i n g specialist. In addition, all in-service p r o g r a m s are e v a l u a t e d u s i n g a W o r k s h o p e v a l u a t i o n Form c o n s t r u c t e d by t h e federal evaluation office to certify delivery a n d a s s e s s u s e f u l n e s s of workshop information. Via written a n d verbal reports, results of these evaluation forms are provided to supervisors within two days of each w o r k s h o p . To date, the s c h e d u l e d periodic u s e of O p e n - e n d e d Stages of C o n c e r n Q u e s t i o n n a i r e (SoCQ) a n d C o n f i g u r a t i o n C h e c k l i s t d a t a have provided the m o s t useful information for program planners. Academically Gifted Program The E l e m e n t a r y Academically Gifted Program (AGP) is designed to offer a c a d e m i c a l l y challenging c o n t e n t in the a r e a s of m a t h e m a t i c s a n d science to gifted s t u d e n t s in grades three t h r o u g h six. S t u d e n t s receive two h o u r s per d a y of a c c e l e r a t e d a n d e x t e n d e d i n s t r u c t i o n delivered by a specially trained t e a c h e r certified in gifted education. The AGP's p h i l o s o p h y is a significant d e p a r t u r e from typical gifted c u r r i c u l u m m o d e l s a n d the previous local model in t h a t it e m p h a s i z e s specialized academic i n s t r u c t i o n in two highly visible disciplines a n d singles out a particular group of s t u d e n t s from their peers at the school building. For t h e s e r e a s o n s , as well as program improvement, a p r o g r a m evaluation is r e g a r d e d as a n essential e l e m e n t in p r o g r a m i m p l e m e n t a t i o n p r o c e d u r e s and activities. The evaluation model resembles the traditional formative- summative pattern. Initially, descriptive q u e s t i o n s were g e n e r a t e d a n d i n f o r m a t i o n g a t h e r i n g o c c u r r e d for t h e p u r p o s e s of h i s t o r i c a l d o c u m e n t a t i o n , c o m m u n i c a t i o n with interested b u t s o m e w h a t skeptical c o m m u n i t y groups, and program decision-making. The s e c o n d p r o g r a m y e a r ' s e m p h a s i s i n c l u d e d t h e f o r m a t i v e c o m p o n e n t w i t h modified q u e s t i o n s a n d d a t a collection s t r a t e g i e s , a n d a d d e d s u m m a t i v e q u e s t i o n s a n d p r o c e d u r e s . Questions a n d focus for the third program year reflect program changes and m o d i f i c a t i o n s , c o l l a b o r a t i o n with a n i n t e r n a t i o n a l l y - recognized gifted educator, a n d continued summative assessment. D a t a collection p r o c e d u r e s have involved s a m p l e s of every relevant d a t a source from school a n d district a d m i n i s t r a t o r s to s t u d e n t s , parents, t e a c h e r s , u n i v e r s i t y personnel a n d the professional literature. Strategies
122
S. D. Turner et aL
u s e d for i n f o r m a t i o n g a t h e r i n g have i n c l u d e d m a i l - o u t s u r v e y s , t e l e p h o n e and face-to-face interviews, classroom observations, town meetings, p e n c i l - p a p e r tests, a n d r e c or d reviews of s u c h t hi ngs as science fair entries, class rosters, a n d staff d e v e l o p m e n t offerings. The L a b o r a t o r y for Individualized F o u n d a t i o n T e a c h i n g {LIFT} T h e L a b o r a t o r y for I n d i v i d u a l i z e d F o u n d a t i o n T e a c h i n g (LIFT), an a l t e r n a t i v e e d u c a t i o n p r o g r a m , w a s i nit i at ed in one H i l l s b o r o u g h C o u n t y e l e m e n t a r y school d u r i n g the s e c o n d s e m e s t e r of t he 1984-85 school year. T h e p u r p o s e of LIFT is to p r o v i d e a h i g h l y m o t i v a t i n g , e d u c a t i o n a l e n v i r o n m e n t d e s i g n e d to i n c r e a s e t h e r a t e of a c a d e m i c p r o g r e s s a n d d e c r e a s e i n a p p r o p r i a t e s t u d e n t b e h a v i o r s w h i c h i nt erfere with the learning p r o c e s s . S t u d e n t s c o m p r i s i n g t he t a r g e t g r o u p are b o t h male a n d female from e t h n i c g r o u p s r e p r e s e n t e d in t he c o u n t r y , a n d a t t e n d g r a d e s four and five. T h e t e a c h e r , a s s i s t e d by a n i n s t r u c t i o n a l aide, i n s t r u c t s e i g h t e e n s t u d e n t s following m a i n l y t he general e l e m e n t a r y c u r r i c u l u m . Multiple d a t a g a t h e r i n g m e t h o d s w e r e e m p l o y e d in t h e p r o g r a m e v a l u a t i o n providing descriptive i nf or m a ti on to d o c u m e n t , modify, as well as i m p l e m e n t the p r o g r a m at twenty-five additional e l e m e n t a r y school s duri ng the 1 9 8 5 - 8 6 school year. To m o n i t o r LIFT's m o s t critical c h a r a c t e r i s t i c s as t h e y evolved at t he initial site, w e e k l y o b s e r v a t i o n s of LIFT w ere m ade. D u r in g t h e s e weekly visits, t he principal, s t u d e n t s , t eacher, aide and s u p p o r t s t a f f wer e i n t e r v i e w e d a n d r e c o r d s were reviewed. Additional interviews were c o n d u c t e d with district a d m i n i s t r a t o r s a n d p a r e n t s of LIFT s t u d e n t s .
Methodology Th e f o u r p r o g r a m evaluations described above formed the basis of this e x p l o r a t o r y s t u d y . T h u s far, r e s e a r c h on ut i l i zat i on h a s b e e n c o n d u c t e d p r i m a r i l y t h r o u g h r e t r o s p e c t i v e c a s e s t u d i e s a n d s i m u l a t i o n investigations, Hence, t h e s e ongoing s t u d i e s provided a v a r i a n t m o d e for utilization study. I n f o r m a t i o n w a s collected f r om d e c i s i o n m a k e r s a n d e v a l u a t o r s via obs e r v a t i o n s a n d surveys. The following similarities exist a m o n g the p r o g r a m evaluations examined. 1) All ar e p r i m a r i l y formative s t u d i e s w h o s e p u r p o s e is to a s s i s t in decision making. 2) All are c o n d u c t e d by an internal evaluator. 3} All h a v e available r e c e n t w r i t t e n r e p o r t s w h i c h explicitly s t a t e
Fostering Utilization
123
conclusions a n d / o r recommendations. 4) All involve centralized a d m i n i s t r a t o r s or c u r r i c u l u m supervi sors as p r i m a r y decision m a k e r s . 5) All em pl oy multiple m e a s u r e s and multiple d a t a sources. 6) All are ongoing a n d c u r r e n t l y being u s e d in t he decision m aki ng process. To a s s u r e t h a t each evaluation h a d achieved a high level of utilization, decision m a k e r s were observed a n d queried regardi ng t hei r utilization of the evaluation's recommendations.
A r a t h e r n a r r o w definition of u s e - t h a t of
actions b a s e d on r e c o m m e n d a t i o n s a n d / o r c o n c l u s i o n s - was employed. Lack of specification of t he type of utilization h a s b e e n cited as a m e t h o d o l o g i c a l p r o b l e m in t h e s t u d y of ut i l i zat i on (Leviton a n d H ughes, 1981).
To alleviate this w e a k n e s s , Rich's (1977) " i n s t r u m e n t a l use" was
selected as th e utilization definition. Table 1 shows, for e a c h evaluation, the n u m b e r a n d p e r c e n t a g e of c o n c l u s i o n s a n d / o r r e c o m m e n d a t i n s w h i c h were acted u p o n b y at least one decision maker. Table 1:
F r e q u e n c i e s a n d P e r c e n t a g e s b y E v a l u a t i o n Projects of Utilization of C o n c l u s i o n s / R e c o m m e n d a t i o n s N um ber of C o n c l u s i o n s /
Number &
Recommendations
Acted Upon
Evaluation Project
S u m m e r Institute
27
26 (96%)
C h a p t e r 1 Reading Microcomputer Component Academically Gifted Program
6
6 (100%)
32
23 (72%)
13
10 (77%)
L ab o r ato r y for Individualized F o u n d a t i o n Teaching
After e s t a b l i s h i n g t h a t t he s t u d y m e t t he high-utilization criterion (i.e., a t l e a s t t w o - t h i r d s of c o n c l u s i o n s / r e c o m m e n d a t i o n s w ere followed), e a c h d ecis io n m a k e r w a s t h e n a s k e d to com p l et e two s u r v e y s developed by the a u t h o r s . C o n t e n t of t he s u r v e y s was derived by u s i n g c o m p o n e n t s of two Alkin categories ( h u m a n factors a n d e v a l u a t i o n factors), a d d i n g a descri pt or for th e mu ltip le xi t y factor, and including several o t h e r e l e m e n t s occasionally
124 S. D. Turner eta/.
mentioned
in t h e u t i l i z a t i o n l i t e r a t u r e .
The first survey, which presented
seventeen factors hypothesized and/or utilization, required respondents influenced
their
recommendations.
utilization
s h o w n to h a v e s o m e r e l a t i o n s h i p to
to i n d i c a t e t h e e x t e n t to w h i c h e a c h f a c t o r
of w r i t t e n
evaluation
conclusions
and/or
The second survey showed the same seventeen factors
a n d r e q u i r e d t h e r e s p o n d e n t to r a n k o r d e r t h e five f a c t o r s w h i c h e x e r t e d t h e m o s t i n f l u e n c e in h i s / h e r
d e c i s i o n to utilize t h e e v a l u a t i o n i n f o r m a t i o n .
(See
A p p e n d i c e s A a n d B for s u r v e y s . ) O n t h e f i r s t s u r v e y we h y p o t h e s i z e d t h a t t h e m u l t i p l e x i t y f a c t o r w o u l d b e c o n s i d e r e d a m a j o r i n f l u e n c e in u t i l i z a t i o n a s f r e q u e n t l y a s a t l e a s t h a l f t h e other factors.
O n t h e r a n k o r d e r s u r v e y , we did n o t e x p e c t t h e m u l t i p l e x i t y
f a c t o r to o u t r a n k
o t h e r f a c t o r s , b u t to b e a m o n g t h e five m o s t i n f l u e n t i a l
factors as frequently as at least half the other factors.
Results Table indicated
2
shows
the
number
and
percentage
of r e s p o n d e n t s
on the first survey the factors they considered
m o d e r a t e a n d little or n o i n f l u e n c e in e v a l u a t i o n u t i l i z a t i o n . in d e s c e n d i n g
who
to b e of m a j o r , F a c t o r s a r e listed
o r d e r a c c o r d i n g to t h e f r e q u e n c y w i t h w h i c h t h e f a c t o r w a s
r e p o r t e d a s a m a j o r i n f l u e n c e (Table 2). P e r c e n t s in t h e first n u m e r i c a l c o l u m n s h o w t h a t e i g h t of tile s e v e n t e e n factors were considered by more than half the respondents
to b e of m a j o r
i n f l u e n c e in t h e i r d e c i s i o n to utilize t h e e v a l u a t i o n d a t a .
A m o n g these eight
factors
of d a t a
is t h e
methods.
multiplexity
factor descriptor:
variety
collection
Thus, as hypothesized, the multiplexity variable was marked at least
as frequently as half the other factors.
I n d e e d , v a r i e t y of d a t a
collection
m e t h o d s w a s r a t e d a s a m a j o r i n f l u e n c e m o r e f r e q u e n t l y t h a n t e n f a c t o r s , less f r e q u e n t l y t h a n five f a c t o r s a n d w i t h t h e s a m e f r e q u e n c y a s o n e factor. As n o t e d Alkin,
by other utilization researchers
Daillak & White
utilization.
1979), m a n y
(Connally & Porter
factors simultaneously
1980;
contribute
to
Not s u r p r i s i n g l y , t h e n , t h e m u l t i p l e x i t y f a c t o r w a s n o t s e l e c t e d a s a
m a j o r i n f l u e n c e in i s o l a t i o n .
As we r e v i e w e d t h e d a t a , it a p p e a r e d a s if o t h e r
u t i l i z a t i o n f a c t o r s m i g h t h a v e s o m e r e l a t i o n s h i p to t h e m u l t i p l e x i t y factor.
To
investigate further, surveys were separated into two groups: subgroup I c o n s i s t e d of t h e e l e v e n r e s p o n d e n t s w h o s e l e c t e d t h e m u l t i p l e x i t y f a c t o r a s a m a j o r i n f l u e n c e ; s u b g r o u p II i n c l u d e d t h e e i g h t r e s p o n d e n t s w h o did n o t
Fostering Utilization 125
T a b l e 2: F r e q u e n c i e s
and Percentages
b y E x t e n t of I n f l u e n c e for F a c t o r s
(N=I9)
Major Factors
Influence a N
Moderate
Little or No
Influence
Influence
%
N
%
N
16
84
2
11
0
planning S u b s t a n c e of r e p o r t
15 14
79 74
2 3
11 16
0 2
0 11
Evaluator's credentials
13
68
2
11
2
11
T i m e l i n e s s of r e p o r t i n g
%
Y o u r i n v o l v e m e n t in e v a l u a t i o n
M e t h o d s of r e p o r t i n g
12
63
1
5
2
11
V a r i e t y of d a t a collection m e t h o d s b
11
58
5
26
1
5
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
11
58
4
21
1
5
E v a l u a t o r ' s i n t e r e s t in t h e p r o g r a m
10
53
2
11
6
32
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
9
47
6
32
1
5
T y p e of e v a l u a t i o n m o d e l
8
42
4
21
3
16
8
42
5
26
3
16
7
37
4
21
6
32
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a collection activities a n d r e s u l t s Reporting format Rapport with evaluator
7
37
3
16
7
37
I n s t r u m e n t s u s e d to g a t h e r d a t a
6
32
7
37
3
16
F r e q u e n c y of d a t a collection
6
32
5
26
3
16
E x t e n t of l i t e r a t u r e r e v i e w
1
5
5
26
8
42
Evaluation cost
0
0
1
5
16
84
a T h e m e d i a n n u m b e r c h e c k e d a s " m a j o r influence" w a s 7. b H y p o t h e s i z e d m u l t i p l e x i t y factor.
Note : S o m e r e s p o n d e n t s did n o t r e s p o n d to all factors; t h u s , e a c h r o w d o e s n o t s u m to 19 or 100%.
126 S. D, Turner eta/.
T a b l e 3:
Frequencies
a n d P e r c e n t a g e s for F a c t o r s of M a j o r I n f l u e n c e W i t h
and Without Multiplexity Factor
Factors
SubKrouD 1
Subgroup 2
Major Influence
Major Influence
with Multiplexity
w i t h o u t Multiplexity
F a c t o r (N = 11) N
T i m e l i n e s s of r e p o r t i n g
%
F a c t o r (N = 8) N
%
10
91
6
75
10
91
5
63
8
73
6
75
Y o u r i n v o l v e m e n t in e v a l u a t i o n planning S u b s t a n c e of r e p o r t Evaluator's credentials
7
64
6
75
M e t h o d s of r e p o r t i n g
8
73
4
50
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
6
55
5
63
E v a l u a t o r ' s i n t e r e s t in t h e p r o g r a m
7
64
3
38
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
7
64
2
25
T y p e of e v a l u a t i o n m o d e l
6
55
2
25
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a 6
55
2
25
Reporting format
collection activities a n d r e s u l t s
5
45
2
25
Rapport with evaluator
5
45
2
25
I n s t r u m e n t s u s e d to g a t h e r d a t a F r e q u e n c y of d a t a collection
5 5
45 45
1
13
1
13
E x t e n t of l i t e r a t u r e r e v i e w
1
9
0
0
Evaluation cost
0
0
0
0
Note:
T h e m e d i a n n u m b e r c h e c k e d a s a m a j o r i n f l u e n c e for s u b g r o u p I w a s 10. T h e m e d i a n n u m b e r c h e c k e d for s u b g r o u p II w a s 7.
Fostering Utilization
127
i d e n t i f y t h e m u l t i p l e x i t y f a c t o r as a m a j o r i n f l u e n c e . Then, differences b e t w e e n t h e two g r o u p s were noted. Table 3 (with f a c t o r s a r r a n g e d in the s a m e order as Table 2) displays frequencies for b o t h s u b g r o u p s . Due to t he small n u m b e r of r e s p o n d e n t s in b o t h groups, we employed a c a u t i o u s definition of difference (i.e., a "difference" exists w h e n a factor was selected b y a majority of r e s p o n d e n t s in one s u b g r o u p and less t h a n a majority of r e s p o n d e n t s in the ot her subgroup). Based on this rule of t h u m b , evidence of a d i s c r e p a n c y exists for four factors: e v a l u a t o r ' s i n t e r e s t in t he program, a s s o r t m e n t of information sources, type of evaluation model, and f r e q u e n c y of f e e d b a c k c o n c e r n i n g dat ~ e v a l u a t i o n activities a n d r e s u l t s . In e a c h case, r e s p o n d e n t s who identified t he multiplexity factor as a m a j o r i nfl uence were m or e likely to select t h e s e four factors as well. Conversely, r e s p o n d e n t s who did n o t c h o o s e t he multiplexity factor were less likely to identify t hese factors as m a j o r influences. Although v a r i o u s i n t e r p r e t a t i o n s of this o c c u r r e n c e can be posed, th e n a t u r e of t h e s e four factors s u g g e s t an i n t e r a c t i o n a m o n g t h e m a n d th e mu ltip l exi t y factor. Additional s t u d y of t h e s e a s s o c i a t i o n s might be fruitful. Tab le 4 s h o w s two pi e c e s of i n f o r m a t i o n r e g a r d i n g t h e r a n k o r d e r survey. Listed in t he first n u m e r i c a l c o l u m n is t h e n u m b e r of r e s p o n d e n t s w h o r a n k e d th e factor a m o n g t he five m o s t influential factors. To exam i ne e a c h factor's relative i m p o r t a n c e , r a n k i n g s were a s s i g n e d inverse weights as s h o w n at the b o t t o m of t he table. Then. weights were s u m m e d for each factor a n d are s h o w n in the second n u m e r i c a l column. R e s u l t s from t h e r a n k or de r s u r v e y m i r r o r t he earlier r e s u l t s in t h a t ~ u b s t a n c e of r e p o r t a n d t i m e l i n e s s of r e p o r t i n g are f a c t o r s w h i c h m o s t i n f l u e n c e d u s e of t h e e v a l u a t i o n i n f o r m a t i o n . H o w e v e r , y o u r (user) i n v o l v e m e n t in 1;he e v a l u a t i o n p l a n n i n g , t h e s e c o n d h i g h e s t p e r c e n t in the initial survey, does not a p p e a r as frequently a m o n g t he m o s t influential factors w h e n r e s p o n d e n t s r a n k e d only five of t h e s e v e n t e e n factors. B a s e d on the weighted s u m scores, two o t h e r factors, f r e q u e n c y of f e e d b a c k and r e p o r t i n g f o r m a t , edge o u t t he u s e r i n v o l v e m e n t factor for t h e t hi rd a n d f o u r t h place spots. As h y p o t h e s i z e d , t h e m u l t i p l e x l t y f a c t o r w a s m a r k e d a m o n g t he five m o s t influential factors as f r e q u e n t l y as hal f t he o t h e r factors. Its weighted s u m score of s i xt een was obt ai ned b e c a u s e t he multiplexity factor was r a n k e d first, s e c o n d a n d third one time each a n d f o u r t h by two r e s p o n d e n t s .
128
S. D. Turner eta/.
Table 4:
F r e q u e n c i e s a n d W e i g h t s for F a c t o r s R a n k e d A m o n g t h e Five Most I n f l u e n t i a l (N = 19) S u m m a t i o n of N
Weighted R a n k s a
12
29
5
18
16
62
Evaluator's credentials
6
13
M e t h o d s of r e p o r t i n g
7
Factors T i m e l i n e s s of r e p o r t i n g Your i n v o l v e m e n t in e v a l u a t i o n p l a n n i n g S u b s t a n c e of r e p o r t
Variety of d a t a collection m e t h o d s
5
18 16
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
6
17
E v a l u a t o r ' s i n t e r e s t in the p r o g r a m
3
15
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
3
11
Type of e v a l u a t i o n m o d e l
3
9
8
20
Reporting f o r m a t
7
20
R a p p o r t with e v a l u a t o r
2
6
I n s t r u m e n t s u s e d to g a t h e r d a t a F r e q u e n c y of d a t a collection
7 1
18 2
E x t e n t of literature review
0
0
E v a l u a t i o n cost
0
0
a R a n k s w e r e a s s i g n e d w e i g h t s a s follows:
r a n k 1=5, r a n k 2=4, r a n k 3=3,
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a collection activities a n d r e s u l t s
r a n k 4=2, r a n k 5= 1. Then, for e a c h f a c t o r t h e w e i g h t s were a d d e d together.
Discussion T h e p r i m a r y p u r p o s e of t h i s p a p e r w a s to p r o p o s e a n d e x p l o r e the a d d i t i o n of o n e e v a l u a t i o n f a c t o r to A l k i n ' s f r a m e w o r k .
Our preliminary
r e s e a r c h s u p p o r t s t h e s o u n d n e s s of this n o t i o n a n d also i n d i c a t e s revision a n d / o r r e t h i n k i n g of two f r a m e w o r k categories: e v a l u a t i o n f a c t o r s a n d h u m a n factors.
Fostering Utilization
129
R e a s s u r i n g l y , r e s u l t s of t hi s i n v e s t i g a t i o n are c o n g r u e n t with earlier s t u d i e s w h i c h f o u n d t i m e l i n e s s of r e p o r t i n g a n d s u b s t a n c e of r e o o r t to be m a j o r f acto r s in e v a l u a t i o n utilization. Within his framework, Alkin considers timing of i n f o r m a t i o n along with f r e o u e n c v of i n f o r m a t i o n provided a n d f o r m a t p r e s e n t a t i o n as e l e m e n t s of e v a l u a t i o n r e p o r t i n g . Bot h e v a l u a t i o n r e p o r t i n g a n d s u b s t a n c e of e va l ua t i on i nf or m a t i on are designated b y Alkin as evaluation factors. I n c l u d e d within s.ub~tance of e v a l u a t i o n i n f o r m a t i o n are i n f o r m a t i o n r e l e v a n c e a n d i n f o r m a t i o n s p e c i f i c i t y . Indeed, o u r fi ndi ngs s u p p o r t the i n c l u s i o n of t h e s e f a c t o r s w i t h i n t he f r a m e w o r k a n d are f u r t h e r u p h e l d by o t h e r r e s e a r c h e r s , i n c l u d i n g P a u l e y a n d C o h e n (1984) who c o n t e n d "reports m u s t be timely a n d relevant to potential decision makers." In a d d i t i o n to t i m e l i n e s s a n d s u b s t a n c e , Alkin's f r a m e w o r k c o n t a i n s o t h e r factors w h i c h r e s p o n d e n t s in this s t u d y c o n s i d e r e d as m a j o r influences in t h e i r e v a l u a t i o n utilization. T he p r o p o s e d fact or -- m u l t i p l e x i t y of d a t a g a t h e r i n g m e t h o d s -- while n o t i n c l u d e d in t h e exi st i ng f r a m e w o r k , was c o n s i d e r e d as influential to utilization as some w hi ch are al ready present. O u r r e s u l t s lend c r e d e n c e to t h e a s s u m p t i o n t h a t a v a r i e t y of d a t a g a t h e r i n g m e t h o d s n o t only affects u n i q u e l y t he u s e of evaluation information b u t c o n t r i b u t e s to related utilization factors, s u c h as evaluator's willingness to involve u s e r s , e v a l u a t o r ' s r a P P o r t with u s e r s , e v a l u a t o r ' s credibilitv, u s e r s ' c o m m i t m e n t to us e , s u b s t a n c e of e v a l u a t i o n i n f o r m a t i o n , a n d e v a l u a t i o n reDortinff. B e c a u s e t hi s s t u d y h a s s e r i o u s m e t h o d o l o g i c a l l i m i t a t i o n s (several of w h i c h were d i s c u s s e d earlier) a n d b e c a u s e , as C o h e n (1977) notes, "formative e v a l u a t i o n findings are p r o b a b l y m or e easily u s e d t h a n t h o s e obt ai ned from a s u m m a t i v e p r o j ect , " l e g i t i m a c y of t h e m u l t i p l e x i t y f a c t o r for s u m m a t i v e e v a l u a t i o n s m u s t . be i n v e s t i g a t e d . F u r t h e r s t u d i e s also m i g h t i n c l u d e evaluations c o n d u c t e d by external evaluators. Incidentally, r e s u l t s of t hi s s t u d y p r o v o k e d s o m e t h o u g h t s a b o u t t he h u m a n f a c t o r s w i t h i n t he f r am e w or k. In reviewing t he r e s u l t s of t he r a n k o rd er survey, we were s o m e w h a t a s t o n i s h e d to find t he h i g h e s t scoring factors to be r e l a t e d to m e t h o d o l o g y r a t h e r t h a n h u m a n factors. Having c u t o u r e v a l u a t o r s ' t e e t h on t h e i m p o r t a n c e of "evaluator c h a r a c t e r i s t i c s , " this finding s t i m u l a t e d m u c h di s c us s i on a b o u t the clarity of o u r survey i n s t r u m e n t , veracity of th e r e s p o n d e n t group a n d quality of d a t a collection p r o c e d u r e s . Finally, we a g r e e d t h a t h u m a n f a c t o r s in u t i l i z a t i o n lose t h e i r p e d e s t a l p o s i t i o n for p r o j e c t s w h i c h involve i n t e m a l e v a l u a t o r s who h a v e proved t h e i r w o r t h and
130
S. D. Turner et aL
e s t a b l i s h e d t h e i r roles. Also r e l e v a n t to t hi s s t u d y is t h e f a c t t h a t all r e s p o n d e n t s h a ve b e e n e d u c a t e d as to p r o g r a m e v a l u a t i o n ' s p u r p o s e s and p r o c e d u r e s a n d pr evi ous l y have utilized e v al uat i on results. In t h e s e instances, d e c i s i o n m a k e r s m a y c o n c e n t r a t e on m e t h o d o l o g i c a l c o n s i d e r a t i o n s in d e c i d i n g w h e t h e r to u s e e v a l u a t i o n i n f o r m a t i o n . Most i n t e r n a l e v a l u a t o r s o p e r a t e from a d v a n t a g e o u s p o s i t i o n s in this r e g a r d in t h a t t h e y are able to a s s i s t d e c i s i o n m a k e r s in clarifying q u e s t i o n s , a r t i c u l a t i n g t h e e v a l u a t i o n d e sig n a n d u n d e r s t a n d i n g r e s u l t s within t he c o n t e x t of t h e i r organi zat i ons. Bickel (1984) r e c e n t l y o b s e r v e d (and r i g h t l y so) t h a t a r e s i d e n t e v a l u a t o r develops a b e t t e r u n d e r s t a n d i n g of the c o n t e x t in w hi ch d a t a are collected and interpreted. C o n s e q u e n c e s of t h i s e v a l u a t o r role are s u r e l y r e l a t e d to utilization. A s e c o n d i s s u e w o r t h y of f u r t h e r s t u d y is t h e "interest in the evalution" factor. Within the existing framework, this variable is r e g a r d e d as solely a user c h a r a c t e r i s t i c . O u r findings s u g g e s t t h a t i n t e r e s t in t he proj ect also m a y be r e l a t e d to e v a l u a t o r c h a r a c t e r i s t i c s . R e p l i c a t i o n s of t h i s s t u d y s h o u l d i n v e s t i g a t e this di s t i nc t i on. In s u m , we have provi ded p r e l i m i n a r y s u p p o r t t h a t e v a l u a t i o n utilization can be i n f l u e n c e d by the p r e s e n c e of multiple dat a g a t h e r i n g m e t h o d s . We feel strongly, b o t h as a r e s u l t of o u r daily w ork and this exploration, t h a t t h e multiplexity factor is an i m p o r t a n t one in utilization. N e v e r t h e l e s s , we a r e t h e first to a d m i t t h a t a d d i t i o n a l q u a n t i t a t i v e and r i g o r o u s s t u d y of t hi s v a r i a b l e is n e c e s s a r y , a n d we i n t e n d to p u r s u e its c o n t r i b u t i o n s with o t h e r e va l ua t i on projects.
References Alkin, M.C. (1984). O r g a n i z i n g for e v a l u a t i o n u s e : A h a n d b o o k a d m i n i s t r a t o r s . Los Angeles: C e nt e r for the S t u d y of Evaluation.
for
Alkin, M.C., Daillak, R.H., & White, P. (1979). Using e v a l u a t i o n s : Does ev alu atio n m a k e a difference? Beverly Hills, CA: Sage Publications. Alkin, M.C., & Law, A. (1980). A c o n v e r s a t i o n on e v a l u a t i o n utilization. E d u c a t i o n a l Evaluation and Policy Analysis, 2(3), 73-79. Bickel, W.E. (1984). E v a l u a t o r in r e s i d e n c e : New p r o s p e c t s for school district evaluation r e s ear c h. E d u c a t i o n a l E val uat i on a n d Policy_ Analysis, 6(3), 2 9 7 - 3 0 6 .
Fostering Utilization
131
B u n k e r , D.R. (1978). Organizing evaluation to serve the n e e d s of program planners and managers. Evaluation and Program Planning, 1, 129-134. Cohen, L.H. (1977). F a c t o r s affecting the utilization of m e n t a l h e a l t h evaluation research findings. Professional Psycholog~y, 8(4), 526-534. Connolly, T., & Porter, A.L. (1980). A u s e r - f o c u s e d model for the utilization of evaluation. Evaluation and Program Planning, 3, 131-140. Cooley, W.W. (1983). Improving the performance of an educational system. E d u c a t i o n a l Researcher. 1__22,4-12. Cox, G.B. (1977). Managerial style: Implications for the utilization of program evaluation information. Evaluation .Quarterly, 1_.[4), 499-508. Dickey, B. (1980). Utilization of e v a l u a t i o n s of s m a l l - s c a l e innovative e d u c a t i o n a l projects. E d u c a t i o n Evaluation a n d Policy Analysis, 2(6), 65-77. Glaser, E.M., & Taylor, S.H. (1973). F a c t o r s influencing t h e s u c c e s s of applied research. American Psychologist, 2__88(2), 140-146. Gurel, L. (1975). The h u m a n side of evaluating h u m a n services programs: P r o b l e m s a n d p r o s p e c t s . In Guttentag, M. & Striening, E.L. (Eds.), H a n d b o o k of E v a l u a t i o n R e s e a r c h Vol. 2. Beverly Hills, CA: Sage Publications. Hall, G.E. (1979) Using the individual and the innovation as a frame of r e f e r e n c e for r e s e a r c h on change. P a p e r p r e s e n t e d at the A n n u a l Meeting of t h e A u s t r a l i a n Association for R e s e a r c h in E d u c a t i o n in Melbourne, November 1979. Hansen, J.B., Martin, J.M., & Oxford, R.L. (1980). The u s e of evaluation: An 8 n a l y t i c a l review. Portland, OR: N o r t h w e s t Regional E d u c a t i o n Laboratory. Leviton, L.C., & H u g h e s , E.F.X. (1981). R e s e a r c h on the utilization of evaluation: A review and synthesis. Evaluation Re.view, 5(4), 525-548.
132
S. D. Turner eta/.
Loucks, S.F., New Love, B.W., & Hall, G.E. (1975). Measuring Levels of Use of t h e I n n o v a t i o n : A M a n u a l for Trainers, Interviewers, a n d Raters. Austin, TX: University of Texas. O m a n , R.C., & Chitwood, S.R. (1984). M a n a g e m e n t e v a l u a t i o n studies: F a c t o r s affecting the a c c e p t a n c e of r e c o m m e n d a t i o n s . E v a l u a t i o n Review, 8(3), 283-305. Patton, M.Q. (1978). Publication. P a t t o n , M.Q. (1982). Publication.
Utilization focused evaluation. Beverly Hills, CA: Sage
Practical
E v a l u a t i o n , Beverly Hills, CA:
Sage
Patton, M.Q., Grimes, P.S., Guthrie, K.M., B r e n n a n , N.J., F r e n d a , B.D., & Blythe, D.A. (1977). In search of impact: An analysis of the utilization of federal h e a l t h evaluation research. In Weiss, C.H. (Ed.) Using social research in public policy making. Lexington, MA: D.C. Heath. Rich, R.F. (1977). Uses of social science information by federal bureacrats: Knowledge for action v e r s u s knowledge for u n d e r s t a n d i n g . In Weiss, C.H. (Ed.), Using social science r e s e a r c h in public policy m a k i n g . Lexington, MA: D.C. Heath. Van De Vall, M.C., Bolas, C., & Kang, T.S. (1976). Applied social research in industrial organization: An evaluation of functions, theory, a n d methods. The J o u r n a l of Applied Behavioral Science, A p r i l / M a y / J u n e , 158-177. Windle, C., & Bates, P. (1974). Evaluating program evaluation: A suggested a p p r o a c h . In Davidson, P.O. et al. (Eds.), E v a l u a t i o n of behavioral Droarams. Champaign, IL: Research Press. Waller, J.D. (1979). DeveloPing useful evaluation capability: Lessons from the model evaluation program, U.S. D e p a r t m e n t of Justice. Washington, D.C.: Government Printing Office. Young, C.J. (1978). E v a l u a t i o n Utilization. Presented at the Evaluation Research Society Second A n n u a l Meeting, Washington, D.C., November 2-4.
0191-491X/88 $0.00 + .50 Copyright © 1988 Pergamon Press plc
FOSTERING UTILIZATION T H R O U G H MULTIPLE DATA GAT HERI N G M E T H O D S Susan D. Turner, Jayne Hartman, Lore A. Nielsen and Judith Lombana Hi//sborough County Public Schools, Tampa, Florida
Introduction To b o r r o w a p h r a s e from the i n c o m p a r a b l e Casey Stengel, the s t u d y of e v a l u a t i o n utilization h a s "...come along slow, b u t fast." Casey's r e m a r k referred, of course, to baseball's Miracle Mets w hose u s u a l plodding s e a s o n s in th e cellar t u r n e d a r o u n d s u d d e n l y in t he s u m m e r of 1969. While our s u b j e c t pales beside the "old professor's" in t e r m s of d r a m a a n d excitement, his m e s s a g e c a n be a p p r o p r i a t e l y applied to the p a s t d e c a d e ' s r e s e a r c h into a n d d is cu s s io n of evaluation utilization. Who first s p o t t e d t h e u n u s e d e v a l u a t i o n r e p o r t s in social service, e d u c a t i o n , a n d g o v e r n m e n t a g e n c y b a s e m e n t s , or w ho first l a m e n t e d their idleness is u n k n o w n . In the p a s t t e n years, however, dozens of articles have b e e n w r i t t e n , p a p e r s p r e s e n t e d , a n d s t u d i e s c o n d u c t e d to d i s c o v e r and d i s c u s s factors w hi ch influence utilization. Most recently, t he C e n t e r for the S t u d y of E v a l u a t i o n at UCLA h a s developed a tri-category analytic framework with t h e p u r p o s e of pr ovi di ng a formal s t r u c t u r e for u n d e r s t a n d i n g and c o n d u c t i n g r e s e a r c h on ut i l i z a t i on (Alkin, 1984). W i t h i n t h r e e general c ateg o r ies (context factors, h u m a n factors and e v a l u a t i o n factors) multiple d e s c r i p t o r s comprise the framework which is b a s e d on a review and analysis of t h e u t i l i z a t i o n l i t e r a t u r e , s e v e r a l c a s e s t u d i e s a n d t h e c o n t r i b u t o r s ' e x p e r i e n c e s in c o n d u c t i n g e v a l u a t i o n s , B e c a u s e it is i n t e n d e d to guide practitioners', theoreticians' and r e s e a r c h e r s ' efforts in s u b s e q u e n t y e a r s and h a s b e e n d i s t r i b u t e d for t h a t p u r p o s e , t he fram ew ork, s h o w n in figure 1, 113
114
S. D. Turner et aL
s h o u l d be a s inclusive as possible. A. S e t t i n g t h e S t a g e Pre-existing evaluation b o u n d s User identity Project characteristics Organizational features B.
Identifying/Organizing the Participants User characteristics: * i n t e r e s t in e v a l u a t i o n * c o m m i t m e n t to u s e * p r o f e s s i o n a l style Evaluator characteristics: * background and identity *
credibility
* c h o i c e of role * w i l l i n g n e s s to involve u s e r s * rapport with users * c o m m i t m e n t to u s e * political s e n s i t i v i t y E v a l u a t i o n p r o c e d u r e s -- p l a n C.
Operationalizing the Interactive Process E v a l u a t i o n p r o c e d u r e s -- e x e c u t i o n S u b s t a n c e of e v a l u a t i o n i n f o r m a t i o n E v a l u a t o r c o m m i t m e n t to u s e I n f o r m a t i o n d i a l o g u e -- f o r m a t i v e User information processing preferences
D. A d d i n g t h e F i n i s h i n g T o u c h e s Evaluation reporting Evaluation characteristics: * c o m m i t m e n t to u s e * political s e n s i t i v i t y I n f o r m a t i o n d i a l o g u e -- s u m m a t i v e U s e r c o m m i t m e n t to u s e S o u r c e : Alkin, 1 9 8 4 : 9 1 F i g u r e I: F a c t o r P a t t e r n for E v a l u a t i o n Use T h e p u r p o s e of t h i s p a p e r is to p r o p o s e t h e a d d i t i o n of o n e f a c t o r to the framework:
m u l t i p l e x i t y (variety) of d a t a g a t h e r i n g m e t h o d s .
Toward
Fostering Utilization
115
this end, two d i s t i n c t d i s c u s s i o n s comprise the paper. First, supportive a r g u m e n t s for the a s s u m p t i o n t h a t this factor s u b s t a n t i a l l y c o n t r i b u t e s to utilization are p r e s e n t e d and relevant literature is cited. Then, to provide preliminary evidence of the factor's legitimacy, a n exploratory s t u d y of four highly-utilized educational program evaluations is described. Before beginning a d i s c u s s i o n of the declared topic, some prefatory r e m a r k s are n e c e s s a r y to familiarize r e a d e r s with o u r experiences a n d p o s i t i o n s (i.e., biases) r e g a r d i n g the s t u d y of utilization. As i n t e r n a l evaluators in a large school district, we know first-hand the t r u t h of Alkin's, Daillak's a n d White's (2979: 233) statement, "...the degree of utilization of the e v a l u a t i o n a p p a r e n t l y is not d e t e r m i n e d by mere c h a n c e b u t is, to a considerable extent, associated with certain characteristics of the evaluation situation." Plainly, we recognize the decisive interpersonal factors frequently associated with utilization (Patton et al., 1977; Patton, 1978, Gurel, 1975). We are h e s i t a n t , however, to e n d o r s e " o r i e n t a t i o n of t h e evaluator" as "perhaps the m o s t influential [factor] in determining w h e t h e r utilization will occur" (Alkin a n d Law, 1980: 79). E n h a n c i n g utilization is an accepted p a r t of our jobs. While we occasionally (albeit infrequently) are t e m p t e d by Harry T r u m a n ' s notion, "I have found the best way to give advice.., is to find o u t w h a t t h e y w a n t and advise t h e m to do it," the professional literature a n d daily occurrences have proved the m o s t valuable resources in moving toward t h a t goal. As t h e l i t e r a t u r e h a s t e n d e d toward e n d l e s s lists of h y p o t h e s i z e d utilization factors (Oman a n d Chitwood, 1984; Leviton a n d Hughes, 1981; Dickey, 1980; H a n s e n , Martin a n d Oxford, 1980; Alkin, DaiUak a n d White, 1979; Young, 1978; Cox, 1977), we have become increasingly skeptical of the direction a n d value of utilization s t u d y . In the end, we are inclined toward the conviction of Patton and his associates (1977:142). The i s s u e at this time is n o t the s e a r c h for a single f o r m u l a of utilization success, nor the generation of ever-longer lists of possible factors affecting Utilization. The t a s k for the p r e s e n t is tO identify and define a few key variables t h a t m a y m a k e a m a j o r difference in a significant n u m b e r of evaluation cases. Realizing, however, the need for a n d potential value of factor frameworks in guiding, c o n d u c t i n g a n d applying research, we e m b r a c e Alkin's c u r r e n t s t r u c t u r e a n d choose to suggest minor revisions a n d modifications as it is implemented.
116
S. D. Turner et aL
Supportive A r g u m e n t s We w h o l e h e a r t e d l y agree with Connolly a n d Port er (1980) who noted t h a t c h a r a c t e r i s t i c s a s s o c i a t e d with ut i l i zat i on are i n t i m a t e l y i nt ert w i ned. Not one, b u t c o n s t e l l a t i o n s of f a c t o r s c o n t r i b u t e in di fferent s i t u a t i o n s to utilization (Alkin, Daillak, & White, 1979). In a r g u i n g for the i ncl usi on of m u l t i p l e x i t y of d a t a g a t h e r i n g m e t h o d s , we are aware t h a t t he p r e s e n c e of this factor could be inferred to exist a l r e a d y within t he f r a m e w o r k b e c a u s e of its r e l a t i o n s h i p , d i r e c t l y or s u b t l y , to o t h e r e s t a b l i s h e d f a c t o r s . T he t r a d i t i o n a l f u n c t i o n s of multiple d a t a g a t h e r i n g m e t h o d s have b e e n t hose of v alid atio n a n d believability of findings (Patton, 1978; B u n k e r , 1978). Aside from t h e s e p e d e s t r i a n motives, we m a i n t a i n t h a t t he u s e of m u l t i p l e d a t a g a t h e r i n g m e t h o d s c o n t r i b u t e s u n i q u e l y to utilization, u n r e l a t e d to t h r e a t s to validity. Th e following t h e s e s explicate this belief. Multiplexity of d a t a g a t h e r i n g m e t h o d s c a n i n f l u e n c e i n v o l v e m e n t of p r o g r a m d e v e l o p e r s , p a r t i c i p a n t s a n d deci si on m a k e r s in t h e e v a l u a t i o n p ro ces s . Waller (1979) f o u n d t h a t i nvol vem ent of t he u s e r w as the p r i m a r y f acto r a s s o c i a t e d with utilization. Active involvement i n c r e a s e s c o m m i t m e n t to utilization of findings b e c a u s e u s e r s are a n integral p a r t of the evaluation's i m p l e m e n t a t i o n . As r e p o r t e d by O m a n and Chitwood (1984), m a n a g e m e n t s t u d i e s w h i c h i n c l u d e d an interactive p r o c e s s b e t w e e n a n a l y s t and involved u n i t p e r s o n n e l s h o w e d a b o v e - a v e r a g e levels of a c c e p t a n c e . Previously, Leviton a n d H u g h e s (1981) h a d r e p o r t e d positive r e l a t i o n s h i p s b e t w e e n the level of u s e r i n v o l v e m e n t a n d ut i l i z a t ion. In a d d i t i o n , Cooley's (1983) " c l i e n t - o r i e n t a t i o n " a n d P a t t o n ' s (1982) " p r o c e s s - o r i e n t e d " a p p r o a c h e s to e valu atio n in clude this active involvement element. A v a r i e t y of d a t a g a t h e r i n g m e t h o d s c a n i m p a c t c o m m u n i c a t i o n b e t w e e n p r o g r a m decision m a k e r a n d e v a l u a t o r a n d b e t w e e n e v a l u a t o r and p r o g r a m p a r t i c i p a n t s . One i nve s t i gat i on involving m e n t a l h e a l t h agencies found that more useful studies featured more frequent communication b e t w e e n e v a l u a t o r a n d staff (Windle and Bates, 1974). Similarly, Glaser and T ay lo r (1973) c o n c l u d e d t h a t s u c c e s s f u l r e s e a r c h s t u d i e s were distinguished from u n s u c c e s s f u l ones b y close c o m m u n i c a t i o n b e t w e e n r e s e a r c h e r s and consumers. F r e q u e n t c o n t a c t b e t w e e n e v a l u a t i o n p r i n c i p a l s c a n alter d e c i s i o n m a k e r a n d e v a l u a t o r p e r c e p t i o n s of one a n o t h e r . C o n n o l l y a n d P o r t e r (1980) a d v a n c e d t h e i d e a t h a t p r e v i o u s i n t e r a c t i o n s b e t w e e n e v a l u a t o r a n d m a n a g e r i nf l ue nc e t h e t r u s t place in the e v a l u a t i o n findings, a n d t h a t i n c r e a s e d t r u s t l eads to utilization. F u r t h e r , t h e y a r g u e d t h a t
Fostering Utilization
117
t w o - w a y c o m m u n i c a t i o n b e t w e e n m a n a g e r a n d e v a l u a t o r is c r u c i a l to evaluation utilization. Multiplexity of d a t a g a t h e r i n g m e t h o d s can affect credibility of the evaluator a n d evaluation findings. Leviton and Hughes (1981) suggested t h a t a l t h o u g h evidence of its effect on utiliztion is mixed, credibility of the producer of a n evaluation is likely to be important. Use of multiple methods a n d m e a s u r e s allows t h e e v a l u a t o r i n c r e a s e d o p p o r t u n i t i e s in w h i c h to d e m o n s t r a t e h i s / h e r abilities and skills. As a result, the p r o g r a m decision m a k e r comes to rely on the evaluator as a primary source of information. Multiple d a t a gathering m e t h o d s c a n be associated with the visiblity the project a t t a i n s with a n u m b e r of different audiences. B u n k e r (1978:132) c o n t e n d e d t h a t "both a c c o u n t a b i l i t y a n d p r o b a b i l i t y of u t i l i z a t i o n are increased by wider distribution of findings." Similarly, in a s t u d y for the General Accounting Office, Young (1978) noted d i s s e m i n a t i o n a m o n g issues t h a t are i m p o r t a n t to utilization. S o m e w h a t d i s c o u n t i n g the relationship between d i s s e m i n a t i o n (i.e., visibility) a n d utilization, Dickey (1980) pointed out the general agreement in the literature on the value of dissemination b u t she reported finding no research d o c u m e n t i n g this relationship. Evaluator u n d e r s t a n d i n g of a project can be influenced by the n u m b e r a n d variety of data-gathering methods. If a decision m a k e r is removed from the evaluation process, h i s / h e r information needs m a y be unclear. In this case, the u s e of multiple d a t a collection m e t h o d s could act to circumvent u n c e r t a i n t y or misconceptions about questions to be investigated. Oman and Chitwood (1984) reported t h a t studies revolving a r o u n d elaborate large scale q u e s t i o n s h a d a lower level of a c c e p t a n c e t h a n s t u d i e s of more focused i s s u e s . Likewise, Connolly a n d Porter (1980) c o m m e n t e d t h a t decision m a k e r u s e of e v a l u a t i o n i n f o r m a t i o n is more likely if the findings are produced in response to a specific information need. Dickey (1980) a n d Cox (1977) b o t h reported i n s t a n c e s in w h i c h decision m a k e r s were u n a b l e to utilize findings b e c a u s e their questions were not a d d r e s s e d by the studies' designs and procedures. The u s e of m u l t i p l e m e a s u r e s improves the evaluator's ability to u n d e r s t a n d b o t h major and minor project issues, talk more fluently a b o u t the project a n d r e p r e s e n t it more accurately. Also. it allows the evaluator to anticipate questions t h a t m a y arise during the course of the project a n d have readily available data for discussion. Multiplexity of d a t a gathering m e t h o d s can influence the probability t h a t a methodological mix of collection strategies a n d reporting techniques
118
S. D. T u r n e r et a/.
will occur. Th e u s e of a variety of m e t h o d s p r o m o t e s wide appeal to assorted d e c i s i o n - m a k e r styles. O m a n a n d Chitwood (1984) r e p o r t e d s t u d i e s t h a t s h o w differences in a c c e p t a n c e of m a n a g e m e n t r e p o r t s b a s e d on m e t h o d s of i n f o r m a t i o n collection. Along s i m i l ar lines, V an de Vall et al (1976:170) c o n c l u d e d t h a t m o r e utilization will o c c u r for m a n a g e m e n t r e s e a r c h st udi es w h i c h e m p l o y "a hi gh r a t e of e c l e c t i c i s m in t h e e x p l o r a t o r y st age of a project." According to C ohen (1977), h e a l t h care a d m i n i s t r a t o r s v a r y in the d e g r e e to w h i c h t h e y v a l u e a n d u n d e r s t a n d scientific i n f o r m a t i o n . He implied t h a t s u n d r y m e t h o d s of collection a n d r e p o r t i n g m i g h t a s s i s t in i n cr eas in g the i m p a c t of evaluation information. Based on a s t u d y of a group of e d u c a t i o n a l s u b j e c t s , Dickey (1980:76) c o n c l u d e d "If the decision m a k e r d oes n o t u n d e r s t a n d or see as a p p r o p r i a t e t he w a y in w h i c h d a t a are collected, t h e n belief in the d a t a is likely to be d i m i n i s h e d a n d utilization of the ev alu atio n low." Her c o m m e n t s regarding t he face value of validity in the c o n d u c t of e v a l u a t i o n s give f u r t h e r s u p p o r t to t he n o t i o n t h a t multiplexity of d a t a gathering m e t h o d s m a y be an i m p o r t a n t variable in utilization. A v a r i e t y of d a t a g a t h e r i n g m e t h o d s m a y i m p a c t t h e s c o p e of an evaluation. In a n article regarding this topic, B u n k e r (1978) pointed out t h a t e v a l u a t i o n utility a n d c o m p r e h e n s i v e n e s s are directly related. Specifically, he c o m m e n t e d t h a t m u l t i p l e d a t a s o u r c e s , m u l t i p l e t y p e s of dat a, a n d d a t a collection over time i n c r e a s e p r o s p e c t s for s u c c e s s in utilizing e v a l u a t i o n information.
Program Evaluation Descriptions To t e s t s u p p o r t for proposing the multiplexity factor for t he utilization framework, four ongoing highly-utilized educational program evaluations were e x a m i n e d .
Brief d e s c r i p t i o n s of e a c h follow.
S u m m e r I n s t i t u t e Pr ogr am As a r e s u l t of t h e F l o r i d a legislative a c t i o n , s u m m e r i n - s e r v i c e i n s t i t u t e s in science and m a t h e m a t i c s were held a c r o s s t he st at e d u r i n g the s u m m e r of 1984. In H i l l s bor ough C o u n t y a p p r o x i m a t e l y 4 0 0 e l e m e n t a r y , j u n i o r high, a n d s e n i o r high t e a c h e r s part i ci pat ed in one of eighteen training p r o g r a m s r e l a t e d to t h e i r 1 9 8 4 - 8 5 t e a c h i n g a s s i g n m e n t s . Institutes i n c l u d e d i n s t r u c t i o n in t he following areas: c u r r e n t c o n t e n t i n f o r m a t i o n , c o n t e n t p r e p a r a t i o n c o n s i s t e n t with t h e Fl ori da S t a n d a r d s of Excellence, u p d a t i n g in q u a n t i t a t i v e l a b o r a t o r y a n d field d a t a - g a t h e r i n g t e c h n i q u e s ,
Fostering Utilization
119
i n f o r m a t i o n on related careers, a n d clinical practice in t e a c h i n g c o n t e n t learned in the institute. S t r u c t u r e a n d c o n t e n t of the s i x t y - h o u r institutes were o u t l i n e d by d i s t r i c t i n s t r u c t i o n a l s u p e r v i s o r s c o l l a b o r a t i n g with university faculty. Teachers who a t t e n d e d the i n s t i t u t e s were paid for their p a r t i c i p a t i o n a n d were eligible to e a r n credit t o w a r d s r e n e w a l of their t e a c h i n g certificates. Overall, more t h a n n i n e t y percent of the participants completed their program successfully. Multiple d a t a sources were u s e d in the program evaluation to provide p r o g r a m d e s i g n e r s w i t h i n f o r m a t i o n to modify t h e ongoing p r o g r a m in s u b s e q u e n t s u m m e r s and to gather outcome information a b o u t the program effects at three levels: immediate knowledge gain of participants, impact of t h e i n s t i t u t e s on i n - s c h o o l i n s t r u c t i o n , a n d a n y effects on s t u d e n t achievement. T e s t r e s u l t s a n d s u c c e s s f u l p r o g r a m c o m p l e t i o n were s u m m a r i z e d by the institutes as well as t h r o u g h use of extensive participant opinonnaires. The impact on instruction was a s s e s s e d by m a i l - o u t surveys a n d Concerns-Based Adoption Model (CBAM) Levels of Use interviews with a s t r a t i f i e d r a n d o m s a m p l e of a p p r o x i m a t e l y twenty-five p e r c e n t of the p a r t i c i p a n t s . For one of the i n s t i t u t e s in which there was a n identifiable m a t c h between institute c o n t e n t and course content, s t u d e n t s were given a subject area achievement test and their scores c o m p a r e d to a control group from the previous year to gauge effect on s t u d e n t learning. Data gathering activities a n d reporting f u n c t i o n s were s c h e d u l e d over a year's time and a s u b s e q u e n t utilization survey of program designers indicated t h a t not only did evaluation activities fulfill state reporting r e q u i r e m e n t s , b u t t h e y were u s e d to improve the 1985 s u m m e r institute as well. C h a p t e r 1 Reading Microcu0mputer C o m o o n e n t In operation for the third year, the C h a p t e r 1 reading microcomputer c o m p o n e n t is b e i n g i n t e g r a t e d into 116 s e p a r a t e c l a s s r o o m s w i t h i n s e v e n t y - f o u r H i l l s b o r o u g h C o u n t y e l e m e n t a r y s c h o o l s covering g r a d e s three-six. Over 3,500 Chapter 1 s t u d e n t s are given additional reinforcement of reading skills t h r o u g h use of c o m p u t e r - a s s i s t e d instruction. Additionally, these s t u d e n t s are exposed to c o m p u t i n g skills which previously h a d been reserved for s t u d e n t s in above average groupings. The d i s t r i c t - w i d e i m p l e m e n t a t i o n of t h e r e a d i n g m i c r o c o m p u t e r c o m p o n e n t is d e s i g n e d in two p h a s e s : planning and training. Pilot e v a l u a t i o n d a t a provided p r o g r a m developers w i t h t h e f r a m e w o r k for a modified district plan for y e a r two. The t r a i n i n g of t e a c h e r s a n d aides is
120
S. D, T u r n e r et a/.
divided in to six levels with i n s t a l l a t i o n of h a r d w a r e at t h e s c h o o l sites following Level II. Following a carefully c o n s t r u c t e d e v a l u a t i o n model which i n c o r p o r a t e s c o m p o n e n t s of t h e C o n c e r n s - B a s e d A d o p t i o n Model (Hall, 1979), the district p h a s e was initiated in y e a r two and c o n t i n u e s to date. Th e P r o c e d u r e s for Adopting E d u c a t i o n a l I n n o v a t i o n s Project at the R & D C e n t e r for T e a c h e r E d u c a t i o n developed a m odel e m p h a s i z i n g i n n o v a t i o n a d o p t i o n , t h e C o n c e r n s - B a s e d A d o p t i o n Model (CBAM), t h a t a s s u m e s c h a n g e to be a highly p e r s o n a l and l e n g t h y process, one t h a t a f f e c t s i n d i v i d u a l s differently. T he m o d e l h y p o t h e s i z e s two d i m e n s i o n s a l o n g w h i c h i n d i v i d u a l s grow as t h e y b e c o m e m o r e f a m i l i a r w i t h a n d s o p h i s t i c a t e d in u s i n g i n n o v a t i o n s : S t a g e s of C o n c e r n A b o u t t h e I n n o v a t i o n (SoC) a n d Levels of Use of t h e I n n o v a t i o n (I~U). T h e S t a g e s of C o n c e r n d i m e n s i o n f o c u s e s on t he feelings, t h o u g h t s a n d i n f o r m a t i o n n e e d s of t he i n n o v a t i o n "user." Indi vi dual s a p p e a r to be mo r e s e l f - c o n c e r n e d w h e n t h e y first find out a b o u t an innovation, wondering w h a t it will r e q u i r e of t h e m , h o w t h e i r roles m u s t c h a n g e . c o n c e r n s ar e resolved, m o r e " t a s k " - o r i e n t e d c o n c e r n s emerge.
As t h e s e Questions
a b o u t w h a t m a t e r i a l s are n e e d e d daily a n d h o w to s c h e d u l e time m ore effectively are typical. Later c o n c e r n s focus on how t he i n n o v a t i o n affects s t u d e n t s , h o w to c o m b i n e efforts or even replace t he i n n o v a t i o n to e n h a n c e s t u d e n t learning. this p r o g r e s s i o n .
The CBAM des c r i bes seven Stages of C o n c e r n t h a t reflect
T h e o t h e r d i m e n s i o n of t he CBAM, Levels of Use of t he I n n o v a t i o n (LoU), d e s c r i b e s t he b e h a v i o r of i ndi vi d ual s as t h e y b e c o m e m o r e familiar with a n d m o r e skilled in u s i n g an innovation. E a c h of t h e eight identified Levels of Use focuses on behavior t h a t is c h a r a c t e r i s t i c of the innovation u s e r at a p a r t i c u l a r stage of development. For example, at LoU II (Preparation), t he individual is p r e p a r i n g for first use. This involves g a t h e r i n g r e s o u r c e s , finding out detailed requirements, arranging the physical setting and s c h e d u l i n g initial steps. At e a c h of t he eight levels, specific b e h a v i o r s have b e e n defined t h a t de s cr i be c h a r a c t e r i s t i c s of the i n n o v a t i o n user. (Loucks, Newlove, & Hall, 1975, pp. 1-2). Evaluation is i n t e g r a t e d i n t o all p h a s e s of t h e p r o j e c t ' s i m p l e m e n t a t i o n a n d e m b o d i e s t h e o n g o i n g a p p l i c a t i o n of t h e CBAM. Multiple d a t a s o u r c e s are u s e d c o n t i n u o u s l y on a p l a n n e d review cycle to provide f e e d b a c k to t he p r o g r a m developers. Prior to t he s t a r t of in-service
Fostering Utilization
121
training, a configuration was developed w h i c h defined critical c o m p o n e n t s for y e a r two. From this configuration, a checklist was generated and is used on visits by t h e m i c r o c o m p u t e r t r a i n i n g specialist. In addition, all in-service p r o g r a m s are e v a l u a t e d u s i n g a W o r k s h o p e v a l u a t i o n Form c o n s t r u c t e d by t h e federal evaluation office to certify delivery a n d a s s e s s u s e f u l n e s s of workshop information. Via written a n d verbal reports, results of these evaluation forms are provided to supervisors within two days of each w o r k s h o p . To date, the s c h e d u l e d periodic u s e of O p e n - e n d e d Stages of C o n c e r n Q u e s t i o n n a i r e (SoCQ) a n d C o n f i g u r a t i o n C h e c k l i s t d a t a have provided the m o s t useful information for program planners. Academically Gifted Program The E l e m e n t a r y Academically Gifted Program (AGP) is designed to offer a c a d e m i c a l l y challenging c o n t e n t in the a r e a s of m a t h e m a t i c s a n d science to gifted s t u d e n t s in grades three t h r o u g h six. S t u d e n t s receive two h o u r s per d a y of a c c e l e r a t e d a n d e x t e n d e d i n s t r u c t i o n delivered by a specially trained t e a c h e r certified in gifted education. The AGP's p h i l o s o p h y is a significant d e p a r t u r e from typical gifted c u r r i c u l u m m o d e l s a n d the previous local model in t h a t it e m p h a s i z e s specialized academic i n s t r u c t i o n in two highly visible disciplines a n d singles out a particular group of s t u d e n t s from their peers at the school building. For t h e s e r e a s o n s , as well as program improvement, a p r o g r a m evaluation is r e g a r d e d as a n essential e l e m e n t in p r o g r a m i m p l e m e n t a t i o n p r o c e d u r e s and activities. The evaluation model resembles the traditional formative- summative pattern. Initially, descriptive q u e s t i o n s were g e n e r a t e d a n d i n f o r m a t i o n g a t h e r i n g o c c u r r e d for t h e p u r p o s e s of h i s t o r i c a l d o c u m e n t a t i o n , c o m m u n i c a t i o n with interested b u t s o m e w h a t skeptical c o m m u n i t y groups, and program decision-making. The s e c o n d p r o g r a m y e a r ' s e m p h a s i s i n c l u d e d t h e f o r m a t i v e c o m p o n e n t w i t h modified q u e s t i o n s a n d d a t a collection s t r a t e g i e s , a n d a d d e d s u m m a t i v e q u e s t i o n s a n d p r o c e d u r e s . Questions a n d focus for the third program year reflect program changes and m o d i f i c a t i o n s , c o l l a b o r a t i o n with a n i n t e r n a t i o n a l l y - recognized gifted educator, a n d continued summative assessment. D a t a collection p r o c e d u r e s have involved s a m p l e s of every relevant d a t a source from school a n d district a d m i n i s t r a t o r s to s t u d e n t s , parents, t e a c h e r s , u n i v e r s i t y personnel a n d the professional literature. Strategies
122
S. D. Turner et aL
u s e d for i n f o r m a t i o n g a t h e r i n g have i n c l u d e d m a i l - o u t s u r v e y s , t e l e p h o n e and face-to-face interviews, classroom observations, town meetings, p e n c i l - p a p e r tests, a n d r e c or d reviews of s u c h t hi ngs as science fair entries, class rosters, a n d staff d e v e l o p m e n t offerings. The L a b o r a t o r y for Individualized F o u n d a t i o n T e a c h i n g {LIFT} T h e L a b o r a t o r y for I n d i v i d u a l i z e d F o u n d a t i o n T e a c h i n g (LIFT), an a l t e r n a t i v e e d u c a t i o n p r o g r a m , w a s i nit i at ed in one H i l l s b o r o u g h C o u n t y e l e m e n t a r y school d u r i n g the s e c o n d s e m e s t e r of t he 1984-85 school year. T h e p u r p o s e of LIFT is to p r o v i d e a h i g h l y m o t i v a t i n g , e d u c a t i o n a l e n v i r o n m e n t d e s i g n e d to i n c r e a s e t h e r a t e of a c a d e m i c p r o g r e s s a n d d e c r e a s e i n a p p r o p r i a t e s t u d e n t b e h a v i o r s w h i c h i nt erfere with the learning p r o c e s s . S t u d e n t s c o m p r i s i n g t he t a r g e t g r o u p are b o t h male a n d female from e t h n i c g r o u p s r e p r e s e n t e d in t he c o u n t r y , a n d a t t e n d g r a d e s four and five. T h e t e a c h e r , a s s i s t e d by a n i n s t r u c t i o n a l aide, i n s t r u c t s e i g h t e e n s t u d e n t s following m a i n l y t he general e l e m e n t a r y c u r r i c u l u m . Multiple d a t a g a t h e r i n g m e t h o d s w e r e e m p l o y e d in t h e p r o g r a m e v a l u a t i o n providing descriptive i nf or m a ti on to d o c u m e n t , modify, as well as i m p l e m e n t the p r o g r a m at twenty-five additional e l e m e n t a r y school s duri ng the 1 9 8 5 - 8 6 school year. To m o n i t o r LIFT's m o s t critical c h a r a c t e r i s t i c s as t h e y evolved at t he initial site, w e e k l y o b s e r v a t i o n s of LIFT w ere m ade. D u r in g t h e s e weekly visits, t he principal, s t u d e n t s , t eacher, aide and s u p p o r t s t a f f wer e i n t e r v i e w e d a n d r e c o r d s were reviewed. Additional interviews were c o n d u c t e d with district a d m i n i s t r a t o r s a n d p a r e n t s of LIFT s t u d e n t s .
Methodology Th e f o u r p r o g r a m evaluations described above formed the basis of this e x p l o r a t o r y s t u d y . T h u s far, r e s e a r c h on ut i l i zat i on h a s b e e n c o n d u c t e d p r i m a r i l y t h r o u g h r e t r o s p e c t i v e c a s e s t u d i e s a n d s i m u l a t i o n investigations, Hence, t h e s e ongoing s t u d i e s provided a v a r i a n t m o d e for utilization study. I n f o r m a t i o n w a s collected f r om d e c i s i o n m a k e r s a n d e v a l u a t o r s via obs e r v a t i o n s a n d surveys. The following similarities exist a m o n g the p r o g r a m evaluations examined. 1) All ar e p r i m a r i l y formative s t u d i e s w h o s e p u r p o s e is to a s s i s t in decision making. 2) All are c o n d u c t e d by an internal evaluator. 3} All h a v e available r e c e n t w r i t t e n r e p o r t s w h i c h explicitly s t a t e
Fostering Utilization
123
conclusions a n d / o r recommendations. 4) All involve centralized a d m i n i s t r a t o r s or c u r r i c u l u m supervi sors as p r i m a r y decision m a k e r s . 5) All em pl oy multiple m e a s u r e s and multiple d a t a sources. 6) All are ongoing a n d c u r r e n t l y being u s e d in t he decision m aki ng process. To a s s u r e t h a t each evaluation h a d achieved a high level of utilization, decision m a k e r s were observed a n d queried regardi ng t hei r utilization of the evaluation's recommendations.
A r a t h e r n a r r o w definition of u s e - t h a t of
actions b a s e d on r e c o m m e n d a t i o n s a n d / o r c o n c l u s i o n s - was employed. Lack of specification of t he type of utilization h a s b e e n cited as a m e t h o d o l o g i c a l p r o b l e m in t h e s t u d y of ut i l i zat i on (Leviton a n d H ughes, 1981).
To alleviate this w e a k n e s s , Rich's (1977) " i n s t r u m e n t a l use" was
selected as th e utilization definition. Table 1 shows, for e a c h evaluation, the n u m b e r a n d p e r c e n t a g e of c o n c l u s i o n s a n d / o r r e c o m m e n d a t i n s w h i c h were acted u p o n b y at least one decision maker. Table 1:
F r e q u e n c i e s a n d P e r c e n t a g e s b y E v a l u a t i o n Projects of Utilization of C o n c l u s i o n s / R e c o m m e n d a t i o n s N um ber of C o n c l u s i o n s /
Number &
Recommendations
Acted Upon
Evaluation Project
S u m m e r Institute
27
26 (96%)
C h a p t e r 1 Reading Microcomputer Component Academically Gifted Program
6
6 (100%)
32
23 (72%)
13
10 (77%)
L ab o r ato r y for Individualized F o u n d a t i o n Teaching
After e s t a b l i s h i n g t h a t t he s t u d y m e t t he high-utilization criterion (i.e., a t l e a s t t w o - t h i r d s of c o n c l u s i o n s / r e c o m m e n d a t i o n s w ere followed), e a c h d ecis io n m a k e r w a s t h e n a s k e d to com p l et e two s u r v e y s developed by the a u t h o r s . C o n t e n t of t he s u r v e y s was derived by u s i n g c o m p o n e n t s of two Alkin categories ( h u m a n factors a n d e v a l u a t i o n factors), a d d i n g a descri pt or for th e mu ltip le xi t y factor, and including several o t h e r e l e m e n t s occasionally
124 S. D. Turner eta/.
mentioned
in t h e u t i l i z a t i o n l i t e r a t u r e .
The first survey, which presented
seventeen factors hypothesized and/or utilization, required respondents influenced
their
recommendations.
utilization
s h o w n to h a v e s o m e r e l a t i o n s h i p to
to i n d i c a t e t h e e x t e n t to w h i c h e a c h f a c t o r
of w r i t t e n
evaluation
conclusions
and/or
The second survey showed the same seventeen factors
a n d r e q u i r e d t h e r e s p o n d e n t to r a n k o r d e r t h e five f a c t o r s w h i c h e x e r t e d t h e m o s t i n f l u e n c e in h i s / h e r
d e c i s i o n to utilize t h e e v a l u a t i o n i n f o r m a t i o n .
(See
A p p e n d i c e s A a n d B for s u r v e y s . ) O n t h e f i r s t s u r v e y we h y p o t h e s i z e d t h a t t h e m u l t i p l e x i t y f a c t o r w o u l d b e c o n s i d e r e d a m a j o r i n f l u e n c e in u t i l i z a t i o n a s f r e q u e n t l y a s a t l e a s t h a l f t h e other factors.
O n t h e r a n k o r d e r s u r v e y , we did n o t e x p e c t t h e m u l t i p l e x i t y
f a c t o r to o u t r a n k
o t h e r f a c t o r s , b u t to b e a m o n g t h e five m o s t i n f l u e n t i a l
factors as frequently as at least half the other factors.
Results Table indicated
2
shows
the
number
and
percentage
of r e s p o n d e n t s
on the first survey the factors they considered
m o d e r a t e a n d little or n o i n f l u e n c e in e v a l u a t i o n u t i l i z a t i o n . in d e s c e n d i n g
who
to b e of m a j o r , F a c t o r s a r e listed
o r d e r a c c o r d i n g to t h e f r e q u e n c y w i t h w h i c h t h e f a c t o r w a s
r e p o r t e d a s a m a j o r i n f l u e n c e (Table 2). P e r c e n t s in t h e first n u m e r i c a l c o l u m n s h o w t h a t e i g h t of tile s e v e n t e e n factors were considered by more than half the respondents
to b e of m a j o r
i n f l u e n c e in t h e i r d e c i s i o n to utilize t h e e v a l u a t i o n d a t a .
A m o n g these eight
factors
of d a t a
is t h e
methods.
multiplexity
factor descriptor:
variety
collection
Thus, as hypothesized, the multiplexity variable was marked at least
as frequently as half the other factors.
I n d e e d , v a r i e t y of d a t a
collection
m e t h o d s w a s r a t e d a s a m a j o r i n f l u e n c e m o r e f r e q u e n t l y t h a n t e n f a c t o r s , less f r e q u e n t l y t h a n five f a c t o r s a n d w i t h t h e s a m e f r e q u e n c y a s o n e factor. As n o t e d Alkin,
by other utilization researchers
Daillak & White
utilization.
1979), m a n y
(Connally & Porter
factors simultaneously
1980;
contribute
to
Not s u r p r i s i n g l y , t h e n , t h e m u l t i p l e x i t y f a c t o r w a s n o t s e l e c t e d a s a
m a j o r i n f l u e n c e in i s o l a t i o n .
As we r e v i e w e d t h e d a t a , it a p p e a r e d a s if o t h e r
u t i l i z a t i o n f a c t o r s m i g h t h a v e s o m e r e l a t i o n s h i p to t h e m u l t i p l e x i t y factor.
To
investigate further, surveys were separated into two groups: subgroup I c o n s i s t e d of t h e e l e v e n r e s p o n d e n t s w h o s e l e c t e d t h e m u l t i p l e x i t y f a c t o r a s a m a j o r i n f l u e n c e ; s u b g r o u p II i n c l u d e d t h e e i g h t r e s p o n d e n t s w h o did n o t
Fostering Utilization 125
T a b l e 2: F r e q u e n c i e s
and Percentages
b y E x t e n t of I n f l u e n c e for F a c t o r s
(N=I9)
Major Factors
Influence a N
Moderate
Little or No
Influence
Influence
%
N
%
N
16
84
2
11
0
planning S u b s t a n c e of r e p o r t
15 14
79 74
2 3
11 16
0 2
0 11
Evaluator's credentials
13
68
2
11
2
11
T i m e l i n e s s of r e p o r t i n g
%
Y o u r i n v o l v e m e n t in e v a l u a t i o n
M e t h o d s of r e p o r t i n g
12
63
1
5
2
11
V a r i e t y of d a t a collection m e t h o d s b
11
58
5
26
1
5
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
11
58
4
21
1
5
E v a l u a t o r ' s i n t e r e s t in t h e p r o g r a m
10
53
2
11
6
32
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
9
47
6
32
1
5
T y p e of e v a l u a t i o n m o d e l
8
42
4
21
3
16
8
42
5
26
3
16
7
37
4
21
6
32
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a collection activities a n d r e s u l t s Reporting format Rapport with evaluator
7
37
3
16
7
37
I n s t r u m e n t s u s e d to g a t h e r d a t a
6
32
7
37
3
16
F r e q u e n c y of d a t a collection
6
32
5
26
3
16
E x t e n t of l i t e r a t u r e r e v i e w
1
5
5
26
8
42
Evaluation cost
0
0
1
5
16
84
a T h e m e d i a n n u m b e r c h e c k e d a s " m a j o r influence" w a s 7. b H y p o t h e s i z e d m u l t i p l e x i t y factor.
Note : S o m e r e s p o n d e n t s did n o t r e s p o n d to all factors; t h u s , e a c h r o w d o e s n o t s u m to 19 or 100%.
126 S. D, Turner eta/.
T a b l e 3:
Frequencies
a n d P e r c e n t a g e s for F a c t o r s of M a j o r I n f l u e n c e W i t h
and Without Multiplexity Factor
Factors
SubKrouD 1
Subgroup 2
Major Influence
Major Influence
with Multiplexity
w i t h o u t Multiplexity
F a c t o r (N = 11) N
T i m e l i n e s s of r e p o r t i n g
%
F a c t o r (N = 8) N
%
10
91
6
75
10
91
5
63
8
73
6
75
Y o u r i n v o l v e m e n t in e v a l u a t i o n planning S u b s t a n c e of r e p o r t Evaluator's credentials
7
64
6
75
M e t h o d s of r e p o r t i n g
8
73
4
50
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
6
55
5
63
E v a l u a t o r ' s i n t e r e s t in t h e p r o g r a m
7
64
3
38
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
7
64
2
25
T y p e of e v a l u a t i o n m o d e l
6
55
2
25
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a 6
55
2
25
Reporting format
collection activities a n d r e s u l t s
5
45
2
25
Rapport with evaluator
5
45
2
25
I n s t r u m e n t s u s e d to g a t h e r d a t a F r e q u e n c y of d a t a collection
5 5
45 45
1
13
1
13
E x t e n t of l i t e r a t u r e r e v i e w
1
9
0
0
Evaluation cost
0
0
0
0
Note:
T h e m e d i a n n u m b e r c h e c k e d a s a m a j o r i n f l u e n c e for s u b g r o u p I w a s 10. T h e m e d i a n n u m b e r c h e c k e d for s u b g r o u p II w a s 7.
Fostering Utilization
127
i d e n t i f y t h e m u l t i p l e x i t y f a c t o r as a m a j o r i n f l u e n c e . Then, differences b e t w e e n t h e two g r o u p s were noted. Table 3 (with f a c t o r s a r r a n g e d in the s a m e order as Table 2) displays frequencies for b o t h s u b g r o u p s . Due to t he small n u m b e r of r e s p o n d e n t s in b o t h groups, we employed a c a u t i o u s definition of difference (i.e., a "difference" exists w h e n a factor was selected b y a majority of r e s p o n d e n t s in one s u b g r o u p and less t h a n a majority of r e s p o n d e n t s in the ot her subgroup). Based on this rule of t h u m b , evidence of a d i s c r e p a n c y exists for four factors: e v a l u a t o r ' s i n t e r e s t in t he program, a s s o r t m e n t of information sources, type of evaluation model, and f r e q u e n c y of f e e d b a c k c o n c e r n i n g dat ~ e v a l u a t i o n activities a n d r e s u l t s . In e a c h case, r e s p o n d e n t s who identified t he multiplexity factor as a m a j o r i nfl uence were m or e likely to select t h e s e four factors as well. Conversely, r e s p o n d e n t s who did n o t c h o o s e t he multiplexity factor were less likely to identify t hese factors as m a j o r influences. Although v a r i o u s i n t e r p r e t a t i o n s of this o c c u r r e n c e can be posed, th e n a t u r e of t h e s e four factors s u g g e s t an i n t e r a c t i o n a m o n g t h e m a n d th e mu ltip l exi t y factor. Additional s t u d y of t h e s e a s s o c i a t i o n s might be fruitful. Tab le 4 s h o w s two pi e c e s of i n f o r m a t i o n r e g a r d i n g t h e r a n k o r d e r survey. Listed in t he first n u m e r i c a l c o l u m n is t h e n u m b e r of r e s p o n d e n t s w h o r a n k e d th e factor a m o n g t he five m o s t influential factors. To exam i ne e a c h factor's relative i m p o r t a n c e , r a n k i n g s were a s s i g n e d inverse weights as s h o w n at the b o t t o m of t he table. Then. weights were s u m m e d for each factor a n d are s h o w n in the second n u m e r i c a l column. R e s u l t s from t h e r a n k or de r s u r v e y m i r r o r t he earlier r e s u l t s in t h a t ~ u b s t a n c e of r e p o r t a n d t i m e l i n e s s of r e p o r t i n g are f a c t o r s w h i c h m o s t i n f l u e n c e d u s e of t h e e v a l u a t i o n i n f o r m a t i o n . H o w e v e r , y o u r (user) i n v o l v e m e n t in 1;he e v a l u a t i o n p l a n n i n g , t h e s e c o n d h i g h e s t p e r c e n t in the initial survey, does not a p p e a r as frequently a m o n g t he m o s t influential factors w h e n r e s p o n d e n t s r a n k e d only five of t h e s e v e n t e e n factors. B a s e d on the weighted s u m scores, two o t h e r factors, f r e q u e n c y of f e e d b a c k and r e p o r t i n g f o r m a t , edge o u t t he u s e r i n v o l v e m e n t factor for t h e t hi rd a n d f o u r t h place spots. As h y p o t h e s i z e d , t h e m u l t i p l e x l t y f a c t o r w a s m a r k e d a m o n g t he five m o s t influential factors as f r e q u e n t l y as hal f t he o t h e r factors. Its weighted s u m score of s i xt een was obt ai ned b e c a u s e t he multiplexity factor was r a n k e d first, s e c o n d a n d third one time each a n d f o u r t h by two r e s p o n d e n t s .
128
S. D. Turner eta/.
Table 4:
F r e q u e n c i e s a n d W e i g h t s for F a c t o r s R a n k e d A m o n g t h e Five Most I n f l u e n t i a l (N = 19) S u m m a t i o n of N
Weighted R a n k s a
12
29
5
18
16
62
Evaluator's credentials
6
13
M e t h o d s of r e p o r t i n g
7
Factors T i m e l i n e s s of r e p o r t i n g Your i n v o l v e m e n t in e v a l u a t i o n p l a n n i n g S u b s t a n c e of r e p o r t
Variety of d a t a collection m e t h o d s
5
18 16
E v a l u a t o r ' s u n d e r s t a n d i n g of p r o g r a m
6
17
E v a l u a t o r ' s i n t e r e s t in the p r o g r a m
3
15
A s s o r t m e n t of i n f o r m a t i o n s o u r c e s
3
11
Type of e v a l u a t i o n m o d e l
3
9
8
20
Reporting f o r m a t
7
20
R a p p o r t with e v a l u a t o r
2
6
I n s t r u m e n t s u s e d to g a t h e r d a t a F r e q u e n c y of d a t a collection
7 1
18 2
E x t e n t of literature review
0
0
E v a l u a t i o n cost
0
0
a R a n k s w e r e a s s i g n e d w e i g h t s a s follows:
r a n k 1=5, r a n k 2=4, r a n k 3=3,
F r e q u e n c y of f e e d b a c k c o n c e r n i n g d a t a collection activities a n d r e s u l t s
r a n k 4=2, r a n k 5= 1. Then, for e a c h f a c t o r t h e w e i g h t s were a d d e d together.
Discussion T h e p r i m a r y p u r p o s e of t h i s p a p e r w a s to p r o p o s e a n d e x p l o r e the a d d i t i o n of o n e e v a l u a t i o n f a c t o r to A l k i n ' s f r a m e w o r k .
Our preliminary
r e s e a r c h s u p p o r t s t h e s o u n d n e s s of this n o t i o n a n d also i n d i c a t e s revision a n d / o r r e t h i n k i n g of two f r a m e w o r k categories: e v a l u a t i o n f a c t o r s a n d h u m a n factors.
Fostering Utilization
129
R e a s s u r i n g l y , r e s u l t s of t hi s i n v e s t i g a t i o n are c o n g r u e n t with earlier s t u d i e s w h i c h f o u n d t i m e l i n e s s of r e p o r t i n g a n d s u b s t a n c e of r e o o r t to be m a j o r f acto r s in e v a l u a t i o n utilization. Within his framework, Alkin considers timing of i n f o r m a t i o n along with f r e o u e n c v of i n f o r m a t i o n provided a n d f o r m a t p r e s e n t a t i o n as e l e m e n t s of e v a l u a t i o n r e p o r t i n g . Bot h e v a l u a t i o n r e p o r t i n g a n d s u b s t a n c e of e va l ua t i on i nf or m a t i on are designated b y Alkin as evaluation factors. I n c l u d e d within s.ub~tance of e v a l u a t i o n i n f o r m a t i o n are i n f o r m a t i o n r e l e v a n c e a n d i n f o r m a t i o n s p e c i f i c i t y . Indeed, o u r fi ndi ngs s u p p o r t the i n c l u s i o n of t h e s e f a c t o r s w i t h i n t he f r a m e w o r k a n d are f u r t h e r u p h e l d by o t h e r r e s e a r c h e r s , i n c l u d i n g P a u l e y a n d C o h e n (1984) who c o n t e n d "reports m u s t be timely a n d relevant to potential decision makers." In a d d i t i o n to t i m e l i n e s s a n d s u b s t a n c e , Alkin's f r a m e w o r k c o n t a i n s o t h e r factors w h i c h r e s p o n d e n t s in this s t u d y c o n s i d e r e d as m a j o r influences in t h e i r e v a l u a t i o n utilization. T he p r o p o s e d fact or -- m u l t i p l e x i t y of d a t a g a t h e r i n g m e t h o d s -- while n o t i n c l u d e d in t h e exi st i ng f r a m e w o r k , was c o n s i d e r e d as influential to utilization as some w hi ch are al ready present. O u r r e s u l t s lend c r e d e n c e to t h e a s s u m p t i o n t h a t a v a r i e t y of d a t a g a t h e r i n g m e t h o d s n o t only affects u n i q u e l y t he u s e of evaluation information b u t c o n t r i b u t e s to related utilization factors, s u c h as evaluator's willingness to involve u s e r s , e v a l u a t o r ' s r a P P o r t with u s e r s , e v a l u a t o r ' s credibilitv, u s e r s ' c o m m i t m e n t to us e , s u b s t a n c e of e v a l u a t i o n i n f o r m a t i o n , a n d e v a l u a t i o n reDortinff. B e c a u s e t hi s s t u d y h a s s e r i o u s m e t h o d o l o g i c a l l i m i t a t i o n s (several of w h i c h were d i s c u s s e d earlier) a n d b e c a u s e , as C o h e n (1977) notes, "formative e v a l u a t i o n findings are p r o b a b l y m or e easily u s e d t h a n t h o s e obt ai ned from a s u m m a t i v e p r o j ect , " l e g i t i m a c y of t h e m u l t i p l e x i t y f a c t o r for s u m m a t i v e e v a l u a t i o n s m u s t . be i n v e s t i g a t e d . F u r t h e r s t u d i e s also m i g h t i n c l u d e evaluations c o n d u c t e d by external evaluators. Incidentally, r e s u l t s of t hi s s t u d y p r o v o k e d s o m e t h o u g h t s a b o u t t he h u m a n f a c t o r s w i t h i n t he f r am e w or k. In reviewing t he r e s u l t s of t he r a n k o rd er survey, we were s o m e w h a t a s t o n i s h e d to find t he h i g h e s t scoring factors to be r e l a t e d to m e t h o d o l o g y r a t h e r t h a n h u m a n factors. Having c u t o u r e v a l u a t o r s ' t e e t h on t h e i m p o r t a n c e of "evaluator c h a r a c t e r i s t i c s , " this finding s t i m u l a t e d m u c h di s c us s i on a b o u t the clarity of o u r survey i n s t r u m e n t , veracity of th e r e s p o n d e n t group a n d quality of d a t a collection p r o c e d u r e s . Finally, we a g r e e d t h a t h u m a n f a c t o r s in u t i l i z a t i o n lose t h e i r p e d e s t a l p o s i t i o n for p r o j e c t s w h i c h involve i n t e m a l e v a l u a t o r s who h a v e proved t h e i r w o r t h and
130
S. D. Turner et aL
e s t a b l i s h e d t h e i r roles. Also r e l e v a n t to t hi s s t u d y is t h e f a c t t h a t all r e s p o n d e n t s h a ve b e e n e d u c a t e d as to p r o g r a m e v a l u a t i o n ' s p u r p o s e s and p r o c e d u r e s a n d pr evi ous l y have utilized e v al uat i on results. In t h e s e instances, d e c i s i o n m a k e r s m a y c o n c e n t r a t e on m e t h o d o l o g i c a l c o n s i d e r a t i o n s in d e c i d i n g w h e t h e r to u s e e v a l u a t i o n i n f o r m a t i o n . Most i n t e r n a l e v a l u a t o r s o p e r a t e from a d v a n t a g e o u s p o s i t i o n s in this r e g a r d in t h a t t h e y are able to a s s i s t d e c i s i o n m a k e r s in clarifying q u e s t i o n s , a r t i c u l a t i n g t h e e v a l u a t i o n d e sig n a n d u n d e r s t a n d i n g r e s u l t s within t he c o n t e x t of t h e i r organi zat i ons. Bickel (1984) r e c e n t l y o b s e r v e d (and r i g h t l y so) t h a t a r e s i d e n t e v a l u a t o r develops a b e t t e r u n d e r s t a n d i n g of the c o n t e x t in w hi ch d a t a are collected and interpreted. C o n s e q u e n c e s of t h i s e v a l u a t o r role are s u r e l y r e l a t e d to utilization. A s e c o n d i s s u e w o r t h y of f u r t h e r s t u d y is t h e "interest in the evalution" factor. Within the existing framework, this variable is r e g a r d e d as solely a user c h a r a c t e r i s t i c . O u r findings s u g g e s t t h a t i n t e r e s t in t he proj ect also m a y be r e l a t e d to e v a l u a t o r c h a r a c t e r i s t i c s . R e p l i c a t i o n s of t h i s s t u d y s h o u l d i n v e s t i g a t e this di s t i nc t i on. In s u m , we have provi ded p r e l i m i n a r y s u p p o r t t h a t e v a l u a t i o n utilization can be i n f l u e n c e d by the p r e s e n c e of multiple dat a g a t h e r i n g m e t h o d s . We feel strongly, b o t h as a r e s u l t of o u r daily w ork and this exploration, t h a t t h e multiplexity factor is an i m p o r t a n t one in utilization. N e v e r t h e l e s s , we a r e t h e first to a d m i t t h a t a d d i t i o n a l q u a n t i t a t i v e and r i g o r o u s s t u d y of t hi s v a r i a b l e is n e c e s s a r y , a n d we i n t e n d to p u r s u e its c o n t r i b u t i o n s with o t h e r e va l ua t i on projects.
References Alkin, M.C. (1984). O r g a n i z i n g for e v a l u a t i o n u s e : A h a n d b o o k a d m i n i s t r a t o r s . Los Angeles: C e nt e r for the S t u d y of Evaluation.
for
Alkin, M.C., Daillak, R.H., & White, P. (1979). Using e v a l u a t i o n s : Does ev alu atio n m a k e a difference? Beverly Hills, CA: Sage Publications. Alkin, M.C., & Law, A. (1980). A c o n v e r s a t i o n on e v a l u a t i o n utilization. E d u c a t i o n a l Evaluation and Policy Analysis, 2(3), 73-79. Bickel, W.E. (1984). E v a l u a t o r in r e s i d e n c e : New p r o s p e c t s for school district evaluation r e s ear c h. E d u c a t i o n a l E val uat i on a n d Policy_ Analysis, 6(3), 2 9 7 - 3 0 6 .
Fostering Utilization
131
B u n k e r , D.R. (1978). Organizing evaluation to serve the n e e d s of program planners and managers. Evaluation and Program Planning, 1, 129-134. Cohen, L.H. (1977). F a c t o r s affecting the utilization of m e n t a l h e a l t h evaluation research findings. Professional Psycholog~y, 8(4), 526-534. Connolly, T., & Porter, A.L. (1980). A u s e r - f o c u s e d model for the utilization of evaluation. Evaluation and Program Planning, 3, 131-140. Cooley, W.W. (1983). Improving the performance of an educational system. E d u c a t i o n a l Researcher. 1__22,4-12. Cox, G.B. (1977). Managerial style: Implications for the utilization of program evaluation information. Evaluation .Quarterly, 1_.[4), 499-508. Dickey, B. (1980). Utilization of e v a l u a t i o n s of s m a l l - s c a l e innovative e d u c a t i o n a l projects. E d u c a t i o n Evaluation a n d Policy Analysis, 2(6), 65-77. Glaser, E.M., & Taylor, S.H. (1973). F a c t o r s influencing t h e s u c c e s s of applied research. American Psychologist, 2__88(2), 140-146. Gurel, L. (1975). The h u m a n side of evaluating h u m a n services programs: P r o b l e m s a n d p r o s p e c t s . In Guttentag, M. & Striening, E.L. (Eds.), H a n d b o o k of E v a l u a t i o n R e s e a r c h Vol. 2. Beverly Hills, CA: Sage Publications. Hall, G.E. (1979) Using the individual and the innovation as a frame of r e f e r e n c e for r e s e a r c h on change. P a p e r p r e s e n t e d at the A n n u a l Meeting of t h e A u s t r a l i a n Association for R e s e a r c h in E d u c a t i o n in Melbourne, November 1979. Hansen, J.B., Martin, J.M., & Oxford, R.L. (1980). The u s e of evaluation: An 8 n a l y t i c a l review. Portland, OR: N o r t h w e s t Regional E d u c a t i o n Laboratory. Leviton, L.C., & H u g h e s , E.F.X. (1981). R e s e a r c h on the utilization of evaluation: A review and synthesis. Evaluation Re.view, 5(4), 525-548.
132
S. D. Turner eta/.
Loucks, S.F., New Love, B.W., & Hall, G.E. (1975). Measuring Levels of Use of t h e I n n o v a t i o n : A M a n u a l for Trainers, Interviewers, a n d Raters. Austin, TX: University of Texas. O m a n , R.C., & Chitwood, S.R. (1984). M a n a g e m e n t e v a l u a t i o n studies: F a c t o r s affecting the a c c e p t a n c e of r e c o m m e n d a t i o n s . E v a l u a t i o n Review, 8(3), 283-305. Patton, M.Q. (1978). Publication. P a t t o n , M.Q. (1982). Publication.
Utilization focused evaluation. Beverly Hills, CA: Sage
Practical
E v a l u a t i o n , Beverly Hills, CA:
Sage
Patton, M.Q., Grimes, P.S., Guthrie, K.M., B r e n n a n , N.J., F r e n d a , B.D., & Blythe, D.A. (1977). In search of impact: An analysis of the utilization of federal h e a l t h evaluation research. In Weiss, C.H. (Ed.) Using social research in public policy making. Lexington, MA: D.C. Heath. Rich, R.F. (1977). Uses of social science information by federal bureacrats: Knowledge for action v e r s u s knowledge for u n d e r s t a n d i n g . In Weiss, C.H. (Ed.), Using social science r e s e a r c h in public policy m a k i n g . Lexington, MA: D.C. Heath. Van De Vall, M.C., Bolas, C., & Kang, T.S. (1976). Applied social research in industrial organization: An evaluation of functions, theory, a n d methods. The J o u r n a l of Applied Behavioral Science, A p r i l / M a y / J u n e , 158-177. Windle, C., & Bates, P. (1974). Evaluating program evaluation: A suggested a p p r o a c h . In Davidson, P.O. et al. (Eds.), E v a l u a t i o n of behavioral Droarams. Champaign, IL: Research Press. Waller, J.D. (1979). DeveloPing useful evaluation capability: Lessons from the model evaluation program, U.S. D e p a r t m e n t of Justice. Washington, D.C.: Government Printing Office. Young, C.J. (1978). E v a l u a t i o n Utilization. Presented at the Evaluation Research Society Second A n n u a l Meeting, Washington, D.C., November 2-4.