- Email: [email protected]
Serving the needs of various groups of stakeholders
Studies in EducationalEvaluation. Vol, 17, pp. 137-149, 1991 Printed in Great Britain. All rights reserved.
0191-491)(/91 $0.00 + .50 © 1991 Pergamon Press plc
SERVING THE NEEDS OF VARIOUS GROUPS OF STAKEHOLDERS Iris Geva-May* and Hannah Peretzt * Department of Education in Science and Technology, Technion, Haifa, Israel t Department o f Education, Te/-Aviv University, Ramat Aviv, Israel
Introduction : B a c k g r o u n d of the Evaluation S t u d y The aim of this s t u d y is to determine the extent to w h i c h evaluation r e s u l t s c a n a n s w e r t h e n e e d s of a v a r i e t y of i n t e r e s t e d p a r t i e s or s t a k e h o l d e r s (term u s e d by G u b a & Lincoln, 1981; Weiss, 1983], a n d be utilized for a n u m b e r of different purposes. It also investigates the r e a s o n s b e h i n d non-utilization of evaluation findings. The c o n t e x t within which the project evaluation was u n d e r t a k e n was a B u s i n e s s English course for executive a s s i s t a n t s at the Van Leer Institute for the D e v e l o p m e n t of Executive A s s i s t a n t s , Tel-Aviv. The goal of the course w a s to raise the level of the secretary in Israel to t h a t of an Executive Assistant, a position with more responsibility a n d requiring high level skills. The s t u d e n t s s t u d y i n g in the i n s t i t u t e c o n s i s t e d of a d u l t s r a n g i n g in age from 20-50. All of t h e m h a d at least a high school Matriculation Exam, if n o t a Bachelor's or Master's Degree. The general i n s t r u c t i o n a l program consisted of c o u r s e s in accounting, b u s i n e s s s t u d i e s , e c o n o m i c s , office m a n a g e m e n t , s h o r t h a n d , wordp r o c e s s i n g a n d two foreign languages, the p r i m a r y foreign l a n g u a g e being English. S t u d e n t s ' m a s t e r y of general English was at a relatively high level, as w a s revealed b y t h e G e n e r a l E n g l i s h Proficiency E n t r a n c e E x a m . According to the r e s u l t s obtained on this exam the s t u d e n t s were divided at the beginning of the academic y e a r into three s t u d y groups: low intermediate, high intermediate, and nearly native speakers. 137
138
I. Geva-May and H. Peretz
Since English t o d a y is considered to be an i n t e r n a t i o n a l language, or, in Dubin a n d Olshtuln's t e r m s {1986), a l a n g u a g e of wider c o m m u n i c a t i o n , in t h e local Israeli b u s i n e s s world E n g l i s h h a s b e c o m e a n e c e s s i t y for promoting communication with the international business community. This, in t u r n , b r o u g h t a b o u t the need for English l a n g u a g e c o u r s e s w h i c h would a d e q u a t e l y m e e t the needs of the b u s i n e s s c o m m u n i t y . As a result, the Van Leer group financed the design a n d development of a special B u s i n e s s English course at the i n s t i t u t e in order to serve the specific n e e d s of the Israeli s t u d e n t population. This course was i n t e n d e d to deal w i t h t h e skills of oral proficiency, writing (correspondence), a n d r e a d i n g , w h i c h were i d e n t i f i e d as b e i n g of m a j o r i m p o r t a n c e for a population involved in b u s i n e s s interaction. Special stress was m e a n t to be placed on socio-cultural linguistic behavior a n d linguistic variety due to the differences i n h e r e n t in Hebrew a n d English (Geva-May et al., 1987; GevaMay & E d a n - B a m b a g y , 1988). F u r t h e r m o r e , the Van Leer group required, a n d decided to finance, a f o r m a t i v e e v a l u a t i o n of t h e B u s i n e s s E n g l i s h p r o j e c t at different development stages in order to provide feedback to a n u m b e r of interested parties: the a d m in istra tio n who represented the financing body, the s t u d e n t s , the t e a c h e r s a n d the i n s t i t u t e ' s c u r r i c u l u m d e v e l o p m e n t team. This w a s initiated in order to enable the s t a k e h o l d e r s to m a k e decisions r e g a r d i n g v a r i o u s a s p e c t s of the program, i.e. financing, self-improvement in the field of instruction, course p l a n n i n g a n d course development.
The Evaluation S t u d y : The Evaluation Tool Since t h e e v a l u a t i o n project w a s i n t e n d e d to serve a n u m b e r of purposes, it was decided to u s e a m u l t i - p u r p o s e diagnostic t e s t as the m a i n tool of e v a l u a t i o n b e c a u s e of t h e variety of potential a n s w e r s t h a t it can supply. M u c h h a s been said a b o u t multi-purpose tests, b u t very few cases have b e e n reported in w h i c h the feasibility of utilizing t e s t findings for a variety of p u r p o s e s h a s been investigated. R e s e a r c h e r s have d i s c u s s e d the notion of diagnostic t e s t i n g a n d the u s e s for w h i c h the i n f o r m a t i o n from s u c h tests c a n be utilized (Carroll, 1980; Harris, 1969; Lewy, 1988; Oiler, 1979; S h o h a m y , 1985). An i m p o r t a n t point m a d e in t h e s e p a p e r s is t h a t if a d i a g n o s t i c t e s t ' s r e s u l t s c a n provide specific i n f o r m a t i o n r e g a r d i n g a l e a r n e r ' s s t r e n g t h s a n d w e a k n e s s e s , t h e n it will n o t only be possible to
Needs of Stakeholders
139
develop a n d prescribe t h e n e c e s s a r y r e m e d i e s for t h e l e a r n e r ' s problem areas, b u t also supply useful information to other related audiences. This ability of diagnostic tests is a characteristic which can be u s e d at t h r e e different e v a l u a t i o n s t a g e s a n d b y a n u m b e r of s t a k e h o l d e r s for a variety of purposes:
(a)
A t the pre-course stage diagnostic t e s t s c a n be u s e d for the screening of l e a r n e r s ' abilities (Carroll, 1980), a n d for m a k i n g t h e l e a r n e r s aware of their initial knowledge. It enables the school a d m i n i s t r a t i o n to u s e t h e r e s u l t s for p l a c e m e n t p u r p o s e s a n d in o r d e r to s e t r e q u i r e m e n t s for a c o u r s e , as p a r t of w h a t S t u f f l e b e a m (1971) describes as context evaluation. Finally, the c u r r i c u l u m development
t e a m or planning.
i n d i v i d u a l t e a c h e r s c a n utilize t h e f i n d i n g s for c o u r s e
(b)
During the course t h e y c a n be u s e d for c h e c k i n g s t u d e n t progress, providing i n f o r m a t i o n to b o t h the l e a r n e r s a n d t h e teachers, a n d for formative evaluation of the c u r r i c u l u m (Carroll, 1980; Lewy, 1977), so t h a t the c u r r i c u l u m writers c a n m a k e a n y n e c e s s a r y revisions; a n d
(c)
A t t h e e n d o f the course t h e s e t e s t s c a n be u s e d for s t u d e n t a c h i e v e m e n t a n d for s u m m a t i v e evaluation of the c u r r i c u l u m (Carroll
1980), i n f o r m i n g t h e development team a d m i n i s t r a t i o n of the worth of the program.
and
the
school
The m u l t i - p u r p o s e diagnostic t e s t u s e d in this project evaluation was developed at Tel Aviv University for the Van Leer I n s t i t u t e . The t e s t was i n t e n d e d to serve a n u m b e r of p u r p o s e s , f o r e m o s t of w h i c h w a s -improvem e n t of i n s t r u c t i o n , with a focus on the n e e d s of the l e a r n e r s in the field of B u s i n e s s English. The t e s t r e s u l t s offered i n f o r m a t i o n to four interested parties: the financing b o d y t h a t ordered the tests, the s t u d e n t s , t h e teachers a n d the institute's c u r r i c u l u m development team. The test w a s m e a n t to provide i n f o r m a t i o n for t h r e e d i f f e r e n t purposes: (1)
To serve as a springboard for individual language awareness, and t h u s , for s t u d e n t remedial work;
{2}
To offer t h e c u r r i c u l u m planners and English teachers data c o n c e r n i n g t h e s t r a t e g i e s t h e y have to e m p l o y for s t r u c t u r i n g the
140
L Geva-Ma)/and H. Peretz
c o u r s e , de c i di ng u p o n level a n d m e t h o d o l o g y , a n d guiding t he individual s t u d e n t s ; a n d (3)
producing
materials
To e n a b l e t h e s c h o o l a d m i n i s t r a t i o n to d e t e r m i n e t h e v a l u e of t he c o u r s e , t h u s as s i s t i ng t h e m in deciding w h e t h e r to c o n t i n u e financing t h e d e v e l o p m e n t a n d t h e r e vi s i on of t h e m a t e r i a l s p r o d u c e d at t h e i n s t i t u t e for this p a r t i c u l a r population. T h e d i a g n o s t i c t e s t w a s a d m i n i s t e r e d to t h e 52 s t u d e n t s in t he first
two levels for t he skills of oral p r o f i c i e n c y a n d writing a w e e k aft er t h e c o u r s e h a d b e g u n . T h e s e skills were c h o s e n b e c a u s e of t h e i r m a j o r role in b u s i n e s s c o m m u n i c a t i o n . T h e t e s t c o n s i s t e d of two s e p a r a t e sect i ons: a Writing Test c o n c e r n e d with formal a n d i n f o r m a l l et t ers involving t h e act of complaining, a n d a n Oral Test c o n s i s t i n g of a s i m u l a t e d interview a n d a role play.
ORAL TEST: Interview
S t u d e n t ' s Name:
Language feature
Scale 1234567 expert u s e r
non-user -
-
Message c o m m u n i c a t i o n
1234567 1234567
Use of r eg i s t e r
- Fluency - Functional sophistication
1234567
- Lexical s o p h i s t i c a t i o n - Grammatical sophistication
1234567
1234567 1234567 3+4+4+2+3+3 = 1 9 / 4 2 = 47%
Score
Figure I: S a m p l e Rating of an Oral Proficiency T a s k
The
raters
of b o t h
the writing and
the
oral tests
w e r e given
e v a l u a t i o n f or m s for e a c h one of t h e l e a r n e r s on w h i c h t h e i r p e r f o r m a n c e was e v a l u a t e d following set criteria a n d r a t i n g scales. E a c h one of the t est s w a s s u p p o s e d to p r o v i d e i n f o r m a t i o n c o n c e r n i n g t h e l e a r n e r s ' initial knowledge of c e r t a i n f e a t u r e s of s p o k e n l a n g u a g e a n d v o c a b u l a r y n e c e s s a r y
Needs of Stakeholders
141
in B u s i n e s s English. These language f e a t u r e s were either c o m m o n to both oral p r o f i c i e n c y a n d writing, i.e. f u n c t i o n a l s o p h i s t i c a t i o n (appropriate l a n g u a g e formulas), or u n i q u e to e a c h one, i.e. flow of s p e e c h in oral proficiency a n d letter f o r m a t in writing. E a c h criterion, representing a relevant l a n g u a g e feature, was rated on a 1 - 7 0sgood scale r a n g i n g from n o n - u s e r to expert, a l m o s t native speaker. Figure 1 represents a sample rating of a n oral proficiency test. For example, the final m a r k of a s t u d e n t ' s Oral Test w a s r e a c h e d by a d d i n g u p t h e % scores o b t a i n e d s e p a r a t e l y for the interview a n d the roleplay a n d dividing by two. T h u s , the interested parties could observe each s t u d e n t ' s p e r f o m a n c e a n d relate to t h e specific deficiencies w i t h i n each criterion.
Utilization of Results The issue of utilization being of great importance, it is n o t the actual r e s u l t s of the diagnostic t e s t t h a t we are i n t e r e s t e d in reporting here, b u t r a t h e r w h e t h e r a n d to w h a t e x t e n t t h e s e findings served the n e e d s of the various stakeholders, a n d w h e t h e r t h e y were a c t u a l l y utilized by them. The r e s u l t s obtained were reported to the different s t a k e h o l d e r s for a v a r i e t y of p u r p o s e s . A l t h o u g h detailed r e c o m m e n d a t i o n s were made, the final i m p l e m e n t a t i o n of the findings ultimately d e p e n d e d on them. The S t u d e n t s The first set of s t a k e h o l d e r s w a s the s t u d e n t s . The pedagogic p u r p o s e of the diagnostic tests was to enable t h e m to become aware of their s t r e n g t h s a n d w e a k n e s s e s . With this a w a r e n e s s t h e y could t h e n focus on remedial work on the a r e a s of deficiency. S t u d e n t Profile C a r d s were p r e p a r e d for e a c h of t h e s t u d e n t s informing t h e m individually of l a n g u a g e abilities a n d deficiencies, for both oral proficiency a n d b u s i n e s s writing. The e v a l u a t i o n criteria a n d the s t u d e n t ' s final score were listed on e a c h card. E a c h s t u d e n t was also provided w i t h a detailed e x p l a n a t i o n of the m e a n i n g of the criteria a n d of the scores, a n d of their implications in t e r m s of m a s t e r y of needed abilities. S u g g e s t i o n s as to possible remedial work were also included. (see Figure 2).
142
L Geva-May and H. Peretz
S t u d e n t ' s Name: XY Final Mark: 46 Writing: f o r the Informal T a s k you received: 3.5/7 for the Formal Task ~Iou received: 3/7 Your ability to convey y o u r message a n d y o u r u s e of register are m o d e s t (4/7 a n d 3/7). You are not aware of the r e q u i r e m e n t s of letter format (3.5/7). • Your m a i n problem, however, lies with: u s e of functional expressions (3/7 a n d 2/7), - m a s t e r y of vocabulary (3/7 a n d 2/7), - g r a m m a r (2/7), a n d spelling (2/7). -
-
Your letters are comprehensible, b u t y o u r v o c a b u l a r y is limited; y o u have c o n s i s t e n t errors of g r a m m a r a n d of spelling w h i c h m a k e y o u a m a r g i n a l writer. I would suggest t h a t you write as m u c h as possible. Write s h o r t p a r a g r a p h s on a n y subject of interest a n d give t h e m to y o u r t e a c h e r to correct. Try to analyze the corrections a n d learn from them.
Figure 2: Sample Profile Card
Individual conferences t a k i n g from t e n to t h i r t y m i n u t e s were held with e a c h of t h e s t u d e n t s in order to d i s c u s s the r e m e d i a l action to be taken, a n d to a n s w e r a n y questions w h i c h the learners m i g h t have. During t h e c o n f e r e n c e s the l e a r n e r s expressed a very favorable a t t i t u d e t o w a r d s t h e diagnostic t e s t a n d were grateful for h a v i n g b e e n helped to p i n p o i n t their problem areas. The g e n e r a l r e s p o n s e given b y the l e a r n e r s w a s t h a t t h e y h a d h a d a v a g u e or intuitive feeling r e g a r d i n g t h e i r deficiencies, b u t t h a t t h e y h a d been u n a b l e "to p u t a finger on it". They claimed t h a t the test reports fairly a n d a c c u r a t e l y d i a g n o s e d t h e i r ability levels. However, t h e s t u d e n t s a d m i t t e d to h a v i n g felt some a p p r e h e n s i o n a n d s k e p t i c i s m u p o n first h e a r i n g a b o u t the t e s t s a n d their aims. Despite this, a major c h a n g e of
Needs o f Stakeholders
143
a t t i t u d e was obs e r ved in t h e c o n f e r e n c e s held. This c a m e a b o u t as a result, t h e y claimed, of bei ng able to a s k q u e s t i o n s a b o u t w h a t t he r e s u l t s m e a n t , h o w t h e y would be affected b y t h e m and w h a t r o u t e s t h e y s h o u l d take. T h e y h i n t e d to t h e f a c t t h a t it w a s t h i s f a c e - t o - f a c e c o m m u n i c a t i o n w h i c h h a d m a d e all t h e di f f er e nc e b o t h in t h e i r a c c e p t a n c e of t h e d i a g n o s i s a n d in t h e i r following t h r o u g h in t h e i r r e m e d i a l work. F u r t h e r m o r e , l a t e r o n in t h e y e a r , a s e c o n d d i a g n o s t i c t e s t w a s c o n d u c t e d to e n a b l e t h e s t u d e n t s to see t h e i r p r o g r e s s in m a s t e r i n g e a c h o n e of t h e l a n g u a g e f e a t u r e s a n d to a n a l y s e t h e i r a c h i e v e m e n t s . Nevertheless,
in
time,
due
stakeholders, the students
to
the
lack
of c o o p e r a t i o n
from
other
c a m e to feel t h a t t h e y w ere bei ng left to cope
alone a n d t h e y b e g a n to lose interest. As a result, findings was only partial.
t he utilization of the t est
The Teachers T h e s e c o n d s e t of s t a k e h o l d e r s w a s t h a t of t h e t e a c h e r s . They r e c e i v e d w h a t m i g h t be c o n s i d e r e d v a l u a b l e i n f o r m a t i o n r e g a r d i n g t h e i r s t u d e n t s ' s t r e n g t h s a n d w e a k n e s s e s , on b o t h a class a n d a n individual basis: i.e. t h e y r ecei ved a full r e p o r t of t h e d i a g n o s t i c t e s t s w h i c h det ai l ed t h e t e n d e n c i e s of e a c h class level as a whole, as well as a complete set of profile cards
for t h e i r r e s p e c t i v e s t u d e n t s .
Thus
the
teachers
had
a wide
i n f o r m a t i o n s p a n w h i c h w a s i n t e n d e d to enabl e t h e m to a d j u s t t hei r lessons a c c o r d i n g to t he l e a r n e r s ' level of knowledge, a n d to p l a n t h e a p p r o p r i a t e r e m e d i a l w o r k for t he individual s t u d e n t s . T h e t e a c h e r s s h o w e d i n t e r e s t in t h e g e n e r a l d i a g n o s t i c r e p o r t a n d r e f e r r e d to it for t h e p u r p o s e of a d j u s t i n g t h e i r c o u r s e p l a n to t he overall class level. However, t h e y r a r el y referred to t he individual s t u d e n t profiles, with t h e e x c e p t i o n of a ver y limited n u m b e r of e x t r e m e l y p r o b l e m a t i c cases. To a c e r t a i n e x t e n t t h i s a p p r o a c h t a k e n b y t h e t e a c h e r s d e f e a t e d t he p u r p o s e of th e diagnostic t e s t s in t h a t t he s t u d e n t s ' e x p e c t a t i o n s regardi ng a n i n d i v i d u a l i z e d a p p r o a c h w e r e n o t m et . It also c o n f l i c t e d w i t h t h e t e a c h e r s ' initial active c o o p e r a t i o n in t he d e v e l o p m e n t a n d a d m i n i s t r a t i o n of t h e t e s t a n d w i t h t h e i r s t a t e d u n d e r s t a n d i n g of its i m p o r t a n c e . Interviews wi t h t he t e a c h e r s revealed t h e fact v e r y t l m e - c o n s u m i n g a n d r e q u i r e d a great deal th eir p a r t for w h i c h t h e y would n o t have b e e n no real s u p p o r t for this a p p r o a c h w as provided
t h a t s u c h a n a p p r o a c h w as of e x t r a w o r k a n d effort on compensated. Furthermore, by t he a d m i n i s t r a t i o n of the
in s titu te, despite its initial e n t h u s i a s m and financing of the diagnostic tests.
144
I. Geva-May and H. Peretz
The C u r r i c u l u m Team The third set of s t a k e h o l d e r s w a s t h e c u r r i c u l u m team, w h i c h was developing t e a c h i n g m a t e r i a l s for B u s i n e s s English for the institute. These m a t e r i a l s were m e a n t to a n s w e r t h e specific l a n g u a g e n e e d s of t h i s particular population. Prior to the development of the materials, open interviews as well as Needs a n d Expectations Q u e s t i o n n a i r e s were d i s t r i b u t e d to a representative s a m p l e of 120 executives a n d executive a s s i s t a n t s . (Geva-May & E d a n B a m b a g y , 1988). These tools, w h i c h s o u g h t to identify the l a n g u a g e needs of the learners, formed the basis for material development. To some extent the detailed r e s u l t s of the diagnostic t e s t s c a m e to validate a n d e x p a n d u p o n t h e f i n d i n g s w h i c h were o b t a i n e d t h r o u g h t h e q u e s t i o n n a i r e , a n d provided a more explicit framework u p o n w h i c h to b a s e the materials. This framework was i n t e n d e d to be u s e d for formative evaluation purposes, in two ways:
{i}
{2)
As a m e a n s of d e t e r m i n i n g the overall s t r u c t u r e a n d l a y o u t of the materials, i.e. w h i c h skills or l a n g u a g e f e a t u r e s s h o u l d be highlighted in t h e m a t e r i a l s , in t e r m s of a d d i t i o n a l e x p l a n a t i o n s , p r a c t i c e activities, etc.; a n d As a m e a n s of revising a n d modifying already p r o d u c e d units.
In a c t u a l i t y , t h e c u r r i c u l u m t e a m m a d e u s e of t h e r e s u l t s for p l a n n i n g a n d s t r u c t u r i n g t h e prospective u n i t s w h i c h h a d n o t y e t b e e n written. However, t h e initial f o r m a t i v e p u r p o s e c o n c e r n i n g already existing u n i t s was not met. Due to a decision t a k e n by the a d m i n i s t r a t i o n of t h e i n s t i t u t e to p o s t p o n e revision, t h e r e s u l t s w e r e n e v e r fully implemented. The A d m i n i s t r a t i o n A full r e p o r t of t h e d i a g n o s t i c t e s t s c o n s i s t i n g of p r o c e d u r e s , description of t e s t s a n d results, w a s given to the f o u r t h s t a k e h o l d e r , the a d m i n i s t r a t i o n of t h e i n s t i t u t e , r e p r e s e n t e d b y the general m a n a g e r . He was the m a i n decision-maker regarding the c o n t i n u a t i o n of f u n d i n g for the revision of the materials a n d their eventual publication. The a d m i n i s t r a t i o n w a s i n t e r e s t e d in discovering the c o m p a r a b l e w o r t h of locally-produced m a t e r i a l s w h i c h specifically a n s w e r e d the n e e d s of a local p o p u l a t i o n , as opposed to c o m m e r c i a l l y - p r o d u c e d materials available on the m a r k e t , a n d
Needs of Stakeholders
145
d i r e c t e d t o w a r d s a more u n i v e r s a l a u d i e n c e (Dubin & O1shtain, 1986; DeEscorcia, 1985). It also s u p p o r t e d the c u r r i c u l u m project a n d initiated t h e d i a g n o s t i c t e s t w h i c h w a s i n t e n d e d to p i n p o i n t t h e local linguistic deficiencies a n d needs, a n d t h u s help in the revision of the materials. At t h e s a m e time, a c o m m i t t e e of expert j u d g e s in t h e fields of foreign l a n g u a g e c u r r i c u l u m design a n d t e a c h i n g w a s c o m m i s s i o n e d to p r e p a r e a formative e v a l u a t i o n r e p o r t of t h e m a t e r i a l s (Olshtain et al., 1987). Their aim w a s to consider w h e t h e r the materials m e t the needs of the learners, a n d w h e t h e r t h e y were w o r t h revising a n d p u b l i s h i n g . Despite the fact t h a t the experts' report f o u n d the m a t e r i a l s to be suitable for the i n t e n d e d a u d i e n c e a n d potentially very good (Olshtain et al., 1987), a n d despite the fact t h a t this s t a k e h o l d e r ' s initial i n t e r e s t in the test was in c o n t i n u i n g the revision project a n d i m p l e m e n t i n g the test findings in the materials, i~mal revision a n d publication of the materials was postponed. D i s c u s s i o n s w i t h t h i s s t a k e h o l d e r p o i n t e d to t h e e x i s t e n c e of o t h e r financial priorities. The r e s u l t s of t h e diagnostic tests were n o t utilized in this case a n d the materials c o n t i n u e d to be u s e d in their original, unrevised form.
Discussion As was m e n t i o n e d earlier, diagnostic tests can be used, a n d are useful, at different stages of a course of study, a n d can provide relevant information to different i n t e r e s t e d parties, according to their n e e d s (Nevo, 1983) for a variety of purposes. Nevertheless, r e s e a r c h e r s (Alkin et al., 1979; Alkin & Coyle, 1988; C o u s i n s & Leithwood 1986; Leithwood et al., 1981; Lewy 1988; P a t t o n , 1977) have f o u n d t h a t t h e r e are m a n y v a r i a b l e s w h i c h influence i m p l e m e n t a t i o n a n d utilization decisions. For example, Cousins & Leithwood (1986) identified i n t e n s i t y of i n f o r m a t i o n n e e d s , type of i n f o r m a t i o n required a n d variance in a u d i e n c e n e e d s for information to be the i n d e p e n d e n t variables influential in several s t u d i e s c o n d u c t e d , Patton (1977) categorized u t i l i z a t i o n f a c t o r s into t h r e e g r o u p s : organizational, referring to the organizations expected to use the results, personal, relating to the evaluator a n d the user, a n d the characteristics of the s t u d y itself. Additional factors affecting the utilization or the n o n - u t i l i z a t i o n of diagnostic t e s t r e s u l t s are t h o s e f o u n d in the p r e s e n t s t u d y , a n d which i n c l u d e p e r s o n a l involvement, risk, i m m e d i a c y of profit, a n d d e p e n d e n c y between audiences.
146
L Geva-May and H. Peretz
Despite t h e fact t h a t all t h e s t a k e h o l d e r s were p a r t n e r s in the evaluation, w h i c h is s u p p o s e d to increase the utilization of r e s u l t s (Cousins & Leithwood, 1986; Lewy, 1988; Weiss, 1983), it a p p e a r s t h a t t h e closer a s t a k e h o l d e r is to t h e r e s u l t s of a d i a g n o s t i c test, the s t r o n g e r t h e i m p a c t a n d the s t r o n g e r the c h a n c e s for i m p l e m e n t a t i o n . As noted in this s t u d y , the personal involvement of the individual s t u d e n t s led to a motivated effort to u n d e r s t a n d a n d utilize the r e s u l t s . On the other h a n d , impersonal involvement, as well as other factors, s u c h as financial and a d m i n i s t r a t i v e c o n s i d e r a t i o n s , d e t e r r e d i m p l e m e n t a t i o n , as w a s t h e case Personal
Involvement
-
with the m a n a g e m e n t . learners, by implementing the results, could only gain from their efforts, the other s t a k e h o l d e r s h a d more at risk. The t e a c h e r s would have h a d to invest a great deal of extra time a n d energy in a d a p t i n g their l e s s o n s to s u i t the n e e d s of each individual s t u d e n t . F u r t h e r m o r e , t h e y t e n d e d to s u s p e c t t h a t t h e y would not be c o m p e n s a t e d for their efforts. In addition, the m a n a g e m e n t h a d already s p e n t a s u b s t a n t i a l a m o u n t of m o n e y on d e v e l o p i n g t h e m a t e r i a l s , a n d b e i n g i g n o r a n t of t h e c u r r i c u l u m development process, w h i c h c a n be a l e n g t h y a n d expensive endeavor, felt t h a t the materials were satisfactory as t h e y were a n d could be u s e d in their original form w i t h o u t a n y f u r t h e r investment. Alkin & Coyle refer to this as utilization m i s u s e or a b u s e , a n d categorize it as "shelving t h e e v a l u a t i o n findings to reduce the likelihood of utilization of results." (1988:337). In f u r t h e r s u p p o r t of our findings, C o u s i n s & Leithwood (1986) have claimed t h a t evaluation Fmdings which were c o n g r u e n t with decision-maker e x p e c t a t i o n s were u s u a l l y easily accepted a n d utilized, while i n c o n g r u e n t findings were u s u a l l y ignored. In our case the findings were, to a certain extent, i n c o n g r u e n t with the a d m i n i s t r a t i o n ' s expectations of n o t having to Risk
- While
the
risk a n y more f u n d s for revision. additional issue is the i m m e d i a c y of feedback or profit gained b y , t h e utilization of the results. While the s t u d e n t s received i m m e d i a t e feedback regarding a n y positive c h a n g e s in their skill mastery, the l e n g t h of time it would have t a k e n the m a n a g e m e n t to profit from its i n v e s t m e n t in the project prevented it from i m p l e m e n t i n g the findings of Immediacy
of Profit -
An
the diagnostic tests. Finally, we found t h a t since the s t u d e n t s were not entirely d e p e n d e n t u p o n the o t h e r s t a k e h o l d e r s in a p p l y i n g the r e s u l t s of the diagnostic tests for their purposes, utilization could be partially e n h a n c e d .
Dependency
-
Needs of Stakeholders
147
However, t h e t e a c h e r s a n d t h e c u r r i c u l u m t e a m were d e p e n d e n t u p o n d e c i s i o n s t a k e n b y the i n s t i t u t e ' s a d m i n i s t r a t i o n a n d b y t h e r e s o u r c e s provided b y it. Therefore, a decision t a k e n by the a d m i n i s t r a t i o n to deter or p o s t p o n e i m p l e m e n t a t i o n resulted in the non-utilization of the results on the p a r t of the d e p e n d e n t stakeholders.
Conclusion By t h e i r very n a t u r e diagnostic t e s t s provide i n f o r m a t i o n regarding deficiencies t h a t can be remedied. Employed in a n evaluation project t h e y c a n serve t h e n e e d s of v a r i o u s g r o u p s of s t a k e h o l d e r s . Their detailed criteria-based r e s u l t s c a n facilitate utilization b y the interested a u d i e n c e s at different stages of the program a n d for a variety of purposes. Nevertheless, since utilization is d e p e n d e n t u p o n d e c i s i o n - m a k i n g a n d c o m m i t m e n t on t h e p a r t of t h e s t a k e h o l d e r s , it is u l t i m a t e l y d e t e r m i n e d by a variety of affective factors. This s t u d y points to the fact t h a t a l t h o u g h the e v a l u a t i o n project a t t e m p t e d to serve the n e e d s of the v a r i o u s s t a k e h o l d e r s b y e n h a n c i n g initial i n t e r e s t a n d r e s p o n s i v e n e s s , w h e n factors s u c h as low p e r s o n a l involvement a n d benefit, v e r s u s high risk and dependency, are at play, t h e probability of n o n - u t i l i z a t i o n increases.
References Alkin, M.C., Daillak, R. & White, P. (1979). Using evaluation: evaluation make a diference? Beverly Hills: Sage.
Does
Alkin, M.C. & Coyle, K. (1988). T h o u g h t s on e v a l u a t i o n u t i l i z a t i o n , misutilization a n d non-utilization. Studies in Educational Evaluation, 14, 331-340. Carroll, B.(1980). Press.
Testing communicative performance.
Oxford: Pergamon
C o u s i n s , J.B. & Leithwood, K.A. (1986). C u r r e n t empirical r e s e a r c h on e v a l u a t i o n utilization. Review of Educational Research, 56 (3), 331364. De Escorsia, B. (1985). ESP and beyond - A quest for real world. In R. Quirk a n d H . G . W i d d o w s o n (Eds.). English in the world. Teaching and learning the language and literature. Cambridge: Cambridge University Press a n d the British Council.
148
I. Geva-May and H. Peretz
Dubin, F. & O1shtain, E. (1986). Course design: Developing programs and materials for language learning. Cambridge: C a m b r i d g e University Press. G e v a - M a y , I. & E d a n - B a m b a g y , O. (1988). A curricular study: A functional~situational approach in business English in Israel. Paper p r e s e n t e d at the 2nd International TESOL Conference, J e r u s a l e m . Geva-May, I., E d a n - B a m b a g y , O., Levitin, H., Peretz, H. & Silbenberg, C. (1987). G.R.O.W. for business, grammar, reading, oral proficiency, wr/ting. Tel Aviv: ESP Curriculum Center, Van Leer - IDEA. G u b a , E.G., a n d Lincoln, Y.S. (1981). Jossey-Bass. Harris,D.P. (1969). McGraw-Hill.
Effective evaluation.
S a n Francisco:
Testing English as a second language.
New York:
Leithwood, K.A., Wilson, R.J. & Marshall, A.R. (1981). I n c r e a s i n g the influence of evaluation s t u d i e s on p r o g r a m decision making. In A. Lewy and D. Nevo (Eds.), Evaluation roles in education. New York: Gordon & Breach. Lewy, A. (Ed.), (1977). Longman.
Handbook o f curriculum evaluation. New York:
Lewy, A. (1986). Formative evaluation of educational pragrammes: The state o f the art. P a p e r p r e s e n t e d at t h e T e a c h e r O p p o r t u n i t y Programmes, J o h a n n e s b u r g . Lewy, A. (1988). Issues in curriculum evaluation. J e r u s a l e m : Education, Israel Curriculum Center.
Ministry of
Nevo, D. (1983). The c o n c e p t u a l i z a t i o n of e d u c a t i o n a l e v a l u a t i o n : An analytical review of literature. Review of Educational Research, 1 5 (I), 1 1 7 - 1 2 8 OUer, J. (1979). Language tests at school. London: Longman. Olshtain,E., Feuerstin, T., Begerano,Y., Scholnik, M., & Smith, K. (1987). A summative evaluation of the Van-Leer IDEA materials for teaching English for professional purposes. I n t e r n a l Report. Tel-Aviv: D e p a r t m e n t of Education, Tel-Aviv University and the Van-Leer IDEA Institute. Patton, M.Q., Grimes, P.S., Guthrie, K.M., B r e n n a n , N.G., French, B.D. & Blyth, D.A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making. New York: Health. S h o h a m y , E. (1985). A practical handbook in language testing for the second language teacher. Tel Aviv: Tel Aviv University Press.
Needs o f Stakeholders
149
S t u f f i e b e a m , D . L . (1971). An i n t r o d u c t i o n to t h e PDK book. In D.L. Stufflebeam, W.J. Foley, W.J. Gephart, E.G. Guda, R.L. H a m m o n d , H.D. M e r r i m a n & M.M. Provus. Education evaluation and decision-making. Itasca, Illinois: F.E. Peacock. Weiss,C.H. (1983). The s t a k e h o l d e r a p p r o a c h to e v a l u a t i o n origins a n d promise. In A.S. Bryk, (Ed.), The stakeholder approach to evaluation: Origins and promise. San-Francisco: Jossey-Bass.
The A u t h o r s IRIS GEVA-MAY is a lecturer in the D e p a r t m e n t of E d u c a t i o n in Science a n d Technology, T e c h n i o n - Israel I n s t i t u t e of Technology. Her fields of r e s e a r c h are project e v a l u a t i o n , c u r r i c u l u m design, LSP (Language for Special Purposes) a n d foreign language learning. HANNAH PERETZ is a post g r a d u a t e s t u d e n t at the School of Education, Tel-Aviv University. She is also a h i g h - s c h o o l t e a c h e r of E n g l i s h in Jerusalem.
0191-491)(/91 $0.00 + .50 © 1991 Pergamon Press plc
SERVING THE NEEDS OF VARIOUS GROUPS OF STAKEHOLDERS Iris Geva-May* and Hannah Peretzt * Department of Education in Science and Technology, Technion, Haifa, Israel t Department o f Education, Te/-Aviv University, Ramat Aviv, Israel
Introduction : B a c k g r o u n d of the Evaluation S t u d y The aim of this s t u d y is to determine the extent to w h i c h evaluation r e s u l t s c a n a n s w e r t h e n e e d s of a v a r i e t y of i n t e r e s t e d p a r t i e s or s t a k e h o l d e r s (term u s e d by G u b a & Lincoln, 1981; Weiss, 1983], a n d be utilized for a n u m b e r of different purposes. It also investigates the r e a s o n s b e h i n d non-utilization of evaluation findings. The c o n t e x t within which the project evaluation was u n d e r t a k e n was a B u s i n e s s English course for executive a s s i s t a n t s at the Van Leer Institute for the D e v e l o p m e n t of Executive A s s i s t a n t s , Tel-Aviv. The goal of the course w a s to raise the level of the secretary in Israel to t h a t of an Executive Assistant, a position with more responsibility a n d requiring high level skills. The s t u d e n t s s t u d y i n g in the i n s t i t u t e c o n s i s t e d of a d u l t s r a n g i n g in age from 20-50. All of t h e m h a d at least a high school Matriculation Exam, if n o t a Bachelor's or Master's Degree. The general i n s t r u c t i o n a l program consisted of c o u r s e s in accounting, b u s i n e s s s t u d i e s , e c o n o m i c s , office m a n a g e m e n t , s h o r t h a n d , wordp r o c e s s i n g a n d two foreign languages, the p r i m a r y foreign l a n g u a g e being English. S t u d e n t s ' m a s t e r y of general English was at a relatively high level, as w a s revealed b y t h e G e n e r a l E n g l i s h Proficiency E n t r a n c e E x a m . According to the r e s u l t s obtained on this exam the s t u d e n t s were divided at the beginning of the academic y e a r into three s t u d y groups: low intermediate, high intermediate, and nearly native speakers. 137
138
I. Geva-May and H. Peretz
Since English t o d a y is considered to be an i n t e r n a t i o n a l language, or, in Dubin a n d Olshtuln's t e r m s {1986), a l a n g u a g e of wider c o m m u n i c a t i o n , in t h e local Israeli b u s i n e s s world E n g l i s h h a s b e c o m e a n e c e s s i t y for promoting communication with the international business community. This, in t u r n , b r o u g h t a b o u t the need for English l a n g u a g e c o u r s e s w h i c h would a d e q u a t e l y m e e t the needs of the b u s i n e s s c o m m u n i t y . As a result, the Van Leer group financed the design a n d development of a special B u s i n e s s English course at the i n s t i t u t e in order to serve the specific n e e d s of the Israeli s t u d e n t population. This course was i n t e n d e d to deal w i t h t h e skills of oral proficiency, writing (correspondence), a n d r e a d i n g , w h i c h were i d e n t i f i e d as b e i n g of m a j o r i m p o r t a n c e for a population involved in b u s i n e s s interaction. Special stress was m e a n t to be placed on socio-cultural linguistic behavior a n d linguistic variety due to the differences i n h e r e n t in Hebrew a n d English (Geva-May et al., 1987; GevaMay & E d a n - B a m b a g y , 1988). F u r t h e r m o r e , the Van Leer group required, a n d decided to finance, a f o r m a t i v e e v a l u a t i o n of t h e B u s i n e s s E n g l i s h p r o j e c t at different development stages in order to provide feedback to a n u m b e r of interested parties: the a d m in istra tio n who represented the financing body, the s t u d e n t s , the t e a c h e r s a n d the i n s t i t u t e ' s c u r r i c u l u m d e v e l o p m e n t team. This w a s initiated in order to enable the s t a k e h o l d e r s to m a k e decisions r e g a r d i n g v a r i o u s a s p e c t s of the program, i.e. financing, self-improvement in the field of instruction, course p l a n n i n g a n d course development.
The Evaluation S t u d y : The Evaluation Tool Since t h e e v a l u a t i o n project w a s i n t e n d e d to serve a n u m b e r of purposes, it was decided to u s e a m u l t i - p u r p o s e diagnostic t e s t as the m a i n tool of e v a l u a t i o n b e c a u s e of t h e variety of potential a n s w e r s t h a t it can supply. M u c h h a s been said a b o u t multi-purpose tests, b u t very few cases have b e e n reported in w h i c h the feasibility of utilizing t e s t findings for a variety of p u r p o s e s h a s been investigated. R e s e a r c h e r s have d i s c u s s e d the notion of diagnostic t e s t i n g a n d the u s e s for w h i c h the i n f o r m a t i o n from s u c h tests c a n be utilized (Carroll, 1980; Harris, 1969; Lewy, 1988; Oiler, 1979; S h o h a m y , 1985). An i m p o r t a n t point m a d e in t h e s e p a p e r s is t h a t if a d i a g n o s t i c t e s t ' s r e s u l t s c a n provide specific i n f o r m a t i o n r e g a r d i n g a l e a r n e r ' s s t r e n g t h s a n d w e a k n e s s e s , t h e n it will n o t only be possible to
Needs of Stakeholders
139
develop a n d prescribe t h e n e c e s s a r y r e m e d i e s for t h e l e a r n e r ' s problem areas, b u t also supply useful information to other related audiences. This ability of diagnostic tests is a characteristic which can be u s e d at t h r e e different e v a l u a t i o n s t a g e s a n d b y a n u m b e r of s t a k e h o l d e r s for a variety of purposes:
(a)
A t the pre-course stage diagnostic t e s t s c a n be u s e d for the screening of l e a r n e r s ' abilities (Carroll, 1980), a n d for m a k i n g t h e l e a r n e r s aware of their initial knowledge. It enables the school a d m i n i s t r a t i o n to u s e t h e r e s u l t s for p l a c e m e n t p u r p o s e s a n d in o r d e r to s e t r e q u i r e m e n t s for a c o u r s e , as p a r t of w h a t S t u f f l e b e a m (1971) describes as context evaluation. Finally, the c u r r i c u l u m development
t e a m or planning.
i n d i v i d u a l t e a c h e r s c a n utilize t h e f i n d i n g s for c o u r s e
(b)
During the course t h e y c a n be u s e d for c h e c k i n g s t u d e n t progress, providing i n f o r m a t i o n to b o t h the l e a r n e r s a n d t h e teachers, a n d for formative evaluation of the c u r r i c u l u m (Carroll, 1980; Lewy, 1977), so t h a t the c u r r i c u l u m writers c a n m a k e a n y n e c e s s a r y revisions; a n d
(c)
A t t h e e n d o f the course t h e s e t e s t s c a n be u s e d for s t u d e n t a c h i e v e m e n t a n d for s u m m a t i v e evaluation of the c u r r i c u l u m (Carroll
1980), i n f o r m i n g t h e development team a d m i n i s t r a t i o n of the worth of the program.
and
the
school
The m u l t i - p u r p o s e diagnostic t e s t u s e d in this project evaluation was developed at Tel Aviv University for the Van Leer I n s t i t u t e . The t e s t was i n t e n d e d to serve a n u m b e r of p u r p o s e s , f o r e m o s t of w h i c h w a s -improvem e n t of i n s t r u c t i o n , with a focus on the n e e d s of the l e a r n e r s in the field of B u s i n e s s English. The t e s t r e s u l t s offered i n f o r m a t i o n to four interested parties: the financing b o d y t h a t ordered the tests, the s t u d e n t s , t h e teachers a n d the institute's c u r r i c u l u m development team. The test w a s m e a n t to provide i n f o r m a t i o n for t h r e e d i f f e r e n t purposes: (1)
To serve as a springboard for individual language awareness, and t h u s , for s t u d e n t remedial work;
{2}
To offer t h e c u r r i c u l u m planners and English teachers data c o n c e r n i n g t h e s t r a t e g i e s t h e y have to e m p l o y for s t r u c t u r i n g the
140
L Geva-Ma)/and H. Peretz
c o u r s e , de c i di ng u p o n level a n d m e t h o d o l o g y , a n d guiding t he individual s t u d e n t s ; a n d (3)
producing
materials
To e n a b l e t h e s c h o o l a d m i n i s t r a t i o n to d e t e r m i n e t h e v a l u e of t he c o u r s e , t h u s as s i s t i ng t h e m in deciding w h e t h e r to c o n t i n u e financing t h e d e v e l o p m e n t a n d t h e r e vi s i on of t h e m a t e r i a l s p r o d u c e d at t h e i n s t i t u t e for this p a r t i c u l a r population. T h e d i a g n o s t i c t e s t w a s a d m i n i s t e r e d to t h e 52 s t u d e n t s in t he first
two levels for t he skills of oral p r o f i c i e n c y a n d writing a w e e k aft er t h e c o u r s e h a d b e g u n . T h e s e skills were c h o s e n b e c a u s e of t h e i r m a j o r role in b u s i n e s s c o m m u n i c a t i o n . T h e t e s t c o n s i s t e d of two s e p a r a t e sect i ons: a Writing Test c o n c e r n e d with formal a n d i n f o r m a l l et t ers involving t h e act of complaining, a n d a n Oral Test c o n s i s t i n g of a s i m u l a t e d interview a n d a role play.
ORAL TEST: Interview
S t u d e n t ' s Name:
Language feature
Scale 1234567 expert u s e r
non-user -
-
Message c o m m u n i c a t i o n
1234567 1234567
Use of r eg i s t e r
- Fluency - Functional sophistication
1234567
- Lexical s o p h i s t i c a t i o n - Grammatical sophistication
1234567
1234567 1234567 3+4+4+2+3+3 = 1 9 / 4 2 = 47%
Score
Figure I: S a m p l e Rating of an Oral Proficiency T a s k
The
raters
of b o t h
the writing and
the
oral tests
w e r e given
e v a l u a t i o n f or m s for e a c h one of t h e l e a r n e r s on w h i c h t h e i r p e r f o r m a n c e was e v a l u a t e d following set criteria a n d r a t i n g scales. E a c h one of the t est s w a s s u p p o s e d to p r o v i d e i n f o r m a t i o n c o n c e r n i n g t h e l e a r n e r s ' initial knowledge of c e r t a i n f e a t u r e s of s p o k e n l a n g u a g e a n d v o c a b u l a r y n e c e s s a r y
Needs of Stakeholders
141
in B u s i n e s s English. These language f e a t u r e s were either c o m m o n to both oral p r o f i c i e n c y a n d writing, i.e. f u n c t i o n a l s o p h i s t i c a t i o n (appropriate l a n g u a g e formulas), or u n i q u e to e a c h one, i.e. flow of s p e e c h in oral proficiency a n d letter f o r m a t in writing. E a c h criterion, representing a relevant l a n g u a g e feature, was rated on a 1 - 7 0sgood scale r a n g i n g from n o n - u s e r to expert, a l m o s t native speaker. Figure 1 represents a sample rating of a n oral proficiency test. For example, the final m a r k of a s t u d e n t ' s Oral Test w a s r e a c h e d by a d d i n g u p t h e % scores o b t a i n e d s e p a r a t e l y for the interview a n d the roleplay a n d dividing by two. T h u s , the interested parties could observe each s t u d e n t ' s p e r f o m a n c e a n d relate to t h e specific deficiencies w i t h i n each criterion.
Utilization of Results The issue of utilization being of great importance, it is n o t the actual r e s u l t s of the diagnostic t e s t t h a t we are i n t e r e s t e d in reporting here, b u t r a t h e r w h e t h e r a n d to w h a t e x t e n t t h e s e findings served the n e e d s of the various stakeholders, a n d w h e t h e r t h e y were a c t u a l l y utilized by them. The r e s u l t s obtained were reported to the different s t a k e h o l d e r s for a v a r i e t y of p u r p o s e s . A l t h o u g h detailed r e c o m m e n d a t i o n s were made, the final i m p l e m e n t a t i o n of the findings ultimately d e p e n d e d on them. The S t u d e n t s The first set of s t a k e h o l d e r s w a s the s t u d e n t s . The pedagogic p u r p o s e of the diagnostic tests was to enable t h e m to become aware of their s t r e n g t h s a n d w e a k n e s s e s . With this a w a r e n e s s t h e y could t h e n focus on remedial work on the a r e a s of deficiency. S t u d e n t Profile C a r d s were p r e p a r e d for e a c h of t h e s t u d e n t s informing t h e m individually of l a n g u a g e abilities a n d deficiencies, for both oral proficiency a n d b u s i n e s s writing. The e v a l u a t i o n criteria a n d the s t u d e n t ' s final score were listed on e a c h card. E a c h s t u d e n t was also provided w i t h a detailed e x p l a n a t i o n of the m e a n i n g of the criteria a n d of the scores, a n d of their implications in t e r m s of m a s t e r y of needed abilities. S u g g e s t i o n s as to possible remedial work were also included. (see Figure 2).
142
L Geva-May and H. Peretz
S t u d e n t ' s Name: XY Final Mark: 46 Writing: f o r the Informal T a s k you received: 3.5/7 for the Formal Task ~Iou received: 3/7 Your ability to convey y o u r message a n d y o u r u s e of register are m o d e s t (4/7 a n d 3/7). You are not aware of the r e q u i r e m e n t s of letter format (3.5/7). • Your m a i n problem, however, lies with: u s e of functional expressions (3/7 a n d 2/7), - m a s t e r y of vocabulary (3/7 a n d 2/7), - g r a m m a r (2/7), a n d spelling (2/7). -
-
Your letters are comprehensible, b u t y o u r v o c a b u l a r y is limited; y o u have c o n s i s t e n t errors of g r a m m a r a n d of spelling w h i c h m a k e y o u a m a r g i n a l writer. I would suggest t h a t you write as m u c h as possible. Write s h o r t p a r a g r a p h s on a n y subject of interest a n d give t h e m to y o u r t e a c h e r to correct. Try to analyze the corrections a n d learn from them.
Figure 2: Sample Profile Card
Individual conferences t a k i n g from t e n to t h i r t y m i n u t e s were held with e a c h of t h e s t u d e n t s in order to d i s c u s s the r e m e d i a l action to be taken, a n d to a n s w e r a n y questions w h i c h the learners m i g h t have. During t h e c o n f e r e n c e s the l e a r n e r s expressed a very favorable a t t i t u d e t o w a r d s t h e diagnostic t e s t a n d were grateful for h a v i n g b e e n helped to p i n p o i n t their problem areas. The g e n e r a l r e s p o n s e given b y the l e a r n e r s w a s t h a t t h e y h a d h a d a v a g u e or intuitive feeling r e g a r d i n g t h e i r deficiencies, b u t t h a t t h e y h a d been u n a b l e "to p u t a finger on it". They claimed t h a t the test reports fairly a n d a c c u r a t e l y d i a g n o s e d t h e i r ability levels. However, t h e s t u d e n t s a d m i t t e d to h a v i n g felt some a p p r e h e n s i o n a n d s k e p t i c i s m u p o n first h e a r i n g a b o u t the t e s t s a n d their aims. Despite this, a major c h a n g e of
Needs o f Stakeholders
143
a t t i t u d e was obs e r ved in t h e c o n f e r e n c e s held. This c a m e a b o u t as a result, t h e y claimed, of bei ng able to a s k q u e s t i o n s a b o u t w h a t t he r e s u l t s m e a n t , h o w t h e y would be affected b y t h e m and w h a t r o u t e s t h e y s h o u l d take. T h e y h i n t e d to t h e f a c t t h a t it w a s t h i s f a c e - t o - f a c e c o m m u n i c a t i o n w h i c h h a d m a d e all t h e di f f er e nc e b o t h in t h e i r a c c e p t a n c e of t h e d i a g n o s i s a n d in t h e i r following t h r o u g h in t h e i r r e m e d i a l work. F u r t h e r m o r e , l a t e r o n in t h e y e a r , a s e c o n d d i a g n o s t i c t e s t w a s c o n d u c t e d to e n a b l e t h e s t u d e n t s to see t h e i r p r o g r e s s in m a s t e r i n g e a c h o n e of t h e l a n g u a g e f e a t u r e s a n d to a n a l y s e t h e i r a c h i e v e m e n t s . Nevertheless,
in
time,
due
stakeholders, the students
to
the
lack
of c o o p e r a t i o n
from
other
c a m e to feel t h a t t h e y w ere bei ng left to cope
alone a n d t h e y b e g a n to lose interest. As a result, findings was only partial.
t he utilization of the t est
The Teachers T h e s e c o n d s e t of s t a k e h o l d e r s w a s t h a t of t h e t e a c h e r s . They r e c e i v e d w h a t m i g h t be c o n s i d e r e d v a l u a b l e i n f o r m a t i o n r e g a r d i n g t h e i r s t u d e n t s ' s t r e n g t h s a n d w e a k n e s s e s , on b o t h a class a n d a n individual basis: i.e. t h e y r ecei ved a full r e p o r t of t h e d i a g n o s t i c t e s t s w h i c h det ai l ed t h e t e n d e n c i e s of e a c h class level as a whole, as well as a complete set of profile cards
for t h e i r r e s p e c t i v e s t u d e n t s .
Thus
the
teachers
had
a wide
i n f o r m a t i o n s p a n w h i c h w a s i n t e n d e d to enabl e t h e m to a d j u s t t hei r lessons a c c o r d i n g to t he l e a r n e r s ' level of knowledge, a n d to p l a n t h e a p p r o p r i a t e r e m e d i a l w o r k for t he individual s t u d e n t s . T h e t e a c h e r s s h o w e d i n t e r e s t in t h e g e n e r a l d i a g n o s t i c r e p o r t a n d r e f e r r e d to it for t h e p u r p o s e of a d j u s t i n g t h e i r c o u r s e p l a n to t he overall class level. However, t h e y r a r el y referred to t he individual s t u d e n t profiles, with t h e e x c e p t i o n of a ver y limited n u m b e r of e x t r e m e l y p r o b l e m a t i c cases. To a c e r t a i n e x t e n t t h i s a p p r o a c h t a k e n b y t h e t e a c h e r s d e f e a t e d t he p u r p o s e of th e diagnostic t e s t s in t h a t t he s t u d e n t s ' e x p e c t a t i o n s regardi ng a n i n d i v i d u a l i z e d a p p r o a c h w e r e n o t m et . It also c o n f l i c t e d w i t h t h e t e a c h e r s ' initial active c o o p e r a t i o n in t he d e v e l o p m e n t a n d a d m i n i s t r a t i o n of t h e t e s t a n d w i t h t h e i r s t a t e d u n d e r s t a n d i n g of its i m p o r t a n c e . Interviews wi t h t he t e a c h e r s revealed t h e fact v e r y t l m e - c o n s u m i n g a n d r e q u i r e d a great deal th eir p a r t for w h i c h t h e y would n o t have b e e n no real s u p p o r t for this a p p r o a c h w as provided
t h a t s u c h a n a p p r o a c h w as of e x t r a w o r k a n d effort on compensated. Furthermore, by t he a d m i n i s t r a t i o n of the
in s titu te, despite its initial e n t h u s i a s m and financing of the diagnostic tests.
144
I. Geva-May and H. Peretz
The C u r r i c u l u m Team The third set of s t a k e h o l d e r s w a s t h e c u r r i c u l u m team, w h i c h was developing t e a c h i n g m a t e r i a l s for B u s i n e s s English for the institute. These m a t e r i a l s were m e a n t to a n s w e r t h e specific l a n g u a g e n e e d s of t h i s particular population. Prior to the development of the materials, open interviews as well as Needs a n d Expectations Q u e s t i o n n a i r e s were d i s t r i b u t e d to a representative s a m p l e of 120 executives a n d executive a s s i s t a n t s . (Geva-May & E d a n B a m b a g y , 1988). These tools, w h i c h s o u g h t to identify the l a n g u a g e needs of the learners, formed the basis for material development. To some extent the detailed r e s u l t s of the diagnostic t e s t s c a m e to validate a n d e x p a n d u p o n t h e f i n d i n g s w h i c h were o b t a i n e d t h r o u g h t h e q u e s t i o n n a i r e , a n d provided a more explicit framework u p o n w h i c h to b a s e the materials. This framework was i n t e n d e d to be u s e d for formative evaluation purposes, in two ways:
{i}
{2)
As a m e a n s of d e t e r m i n i n g the overall s t r u c t u r e a n d l a y o u t of the materials, i.e. w h i c h skills or l a n g u a g e f e a t u r e s s h o u l d be highlighted in t h e m a t e r i a l s , in t e r m s of a d d i t i o n a l e x p l a n a t i o n s , p r a c t i c e activities, etc.; a n d As a m e a n s of revising a n d modifying already p r o d u c e d units.
In a c t u a l i t y , t h e c u r r i c u l u m t e a m m a d e u s e of t h e r e s u l t s for p l a n n i n g a n d s t r u c t u r i n g t h e prospective u n i t s w h i c h h a d n o t y e t b e e n written. However, t h e initial f o r m a t i v e p u r p o s e c o n c e r n i n g already existing u n i t s was not met. Due to a decision t a k e n by the a d m i n i s t r a t i o n of t h e i n s t i t u t e to p o s t p o n e revision, t h e r e s u l t s w e r e n e v e r fully implemented. The A d m i n i s t r a t i o n A full r e p o r t of t h e d i a g n o s t i c t e s t s c o n s i s t i n g of p r o c e d u r e s , description of t e s t s a n d results, w a s given to the f o u r t h s t a k e h o l d e r , the a d m i n i s t r a t i o n of t h e i n s t i t u t e , r e p r e s e n t e d b y the general m a n a g e r . He was the m a i n decision-maker regarding the c o n t i n u a t i o n of f u n d i n g for the revision of the materials a n d their eventual publication. The a d m i n i s t r a t i o n w a s i n t e r e s t e d in discovering the c o m p a r a b l e w o r t h of locally-produced m a t e r i a l s w h i c h specifically a n s w e r e d the n e e d s of a local p o p u l a t i o n , as opposed to c o m m e r c i a l l y - p r o d u c e d materials available on the m a r k e t , a n d
Needs of Stakeholders
145
d i r e c t e d t o w a r d s a more u n i v e r s a l a u d i e n c e (Dubin & O1shtain, 1986; DeEscorcia, 1985). It also s u p p o r t e d the c u r r i c u l u m project a n d initiated t h e d i a g n o s t i c t e s t w h i c h w a s i n t e n d e d to p i n p o i n t t h e local linguistic deficiencies a n d needs, a n d t h u s help in the revision of the materials. At t h e s a m e time, a c o m m i t t e e of expert j u d g e s in t h e fields of foreign l a n g u a g e c u r r i c u l u m design a n d t e a c h i n g w a s c o m m i s s i o n e d to p r e p a r e a formative e v a l u a t i o n r e p o r t of t h e m a t e r i a l s (Olshtain et al., 1987). Their aim w a s to consider w h e t h e r the materials m e t the needs of the learners, a n d w h e t h e r t h e y were w o r t h revising a n d p u b l i s h i n g . Despite the fact t h a t the experts' report f o u n d the m a t e r i a l s to be suitable for the i n t e n d e d a u d i e n c e a n d potentially very good (Olshtain et al., 1987), a n d despite the fact t h a t this s t a k e h o l d e r ' s initial i n t e r e s t in the test was in c o n t i n u i n g the revision project a n d i m p l e m e n t i n g the test findings in the materials, i~mal revision a n d publication of the materials was postponed. D i s c u s s i o n s w i t h t h i s s t a k e h o l d e r p o i n t e d to t h e e x i s t e n c e of o t h e r financial priorities. The r e s u l t s of t h e diagnostic tests were n o t utilized in this case a n d the materials c o n t i n u e d to be u s e d in their original, unrevised form.
Discussion As was m e n t i o n e d earlier, diagnostic tests can be used, a n d are useful, at different stages of a course of study, a n d can provide relevant information to different i n t e r e s t e d parties, according to their n e e d s (Nevo, 1983) for a variety of purposes. Nevertheless, r e s e a r c h e r s (Alkin et al., 1979; Alkin & Coyle, 1988; C o u s i n s & Leithwood 1986; Leithwood et al., 1981; Lewy 1988; P a t t o n , 1977) have f o u n d t h a t t h e r e are m a n y v a r i a b l e s w h i c h influence i m p l e m e n t a t i o n a n d utilization decisions. For example, Cousins & Leithwood (1986) identified i n t e n s i t y of i n f o r m a t i o n n e e d s , type of i n f o r m a t i o n required a n d variance in a u d i e n c e n e e d s for information to be the i n d e p e n d e n t variables influential in several s t u d i e s c o n d u c t e d , Patton (1977) categorized u t i l i z a t i o n f a c t o r s into t h r e e g r o u p s : organizational, referring to the organizations expected to use the results, personal, relating to the evaluator a n d the user, a n d the characteristics of the s t u d y itself. Additional factors affecting the utilization or the n o n - u t i l i z a t i o n of diagnostic t e s t r e s u l t s are t h o s e f o u n d in the p r e s e n t s t u d y , a n d which i n c l u d e p e r s o n a l involvement, risk, i m m e d i a c y of profit, a n d d e p e n d e n c y between audiences.
146
L Geva-May and H. Peretz
Despite t h e fact t h a t all t h e s t a k e h o l d e r s were p a r t n e r s in the evaluation, w h i c h is s u p p o s e d to increase the utilization of r e s u l t s (Cousins & Leithwood, 1986; Lewy, 1988; Weiss, 1983), it a p p e a r s t h a t t h e closer a s t a k e h o l d e r is to t h e r e s u l t s of a d i a g n o s t i c test, the s t r o n g e r t h e i m p a c t a n d the s t r o n g e r the c h a n c e s for i m p l e m e n t a t i o n . As noted in this s t u d y , the personal involvement of the individual s t u d e n t s led to a motivated effort to u n d e r s t a n d a n d utilize the r e s u l t s . On the other h a n d , impersonal involvement, as well as other factors, s u c h as financial and a d m i n i s t r a t i v e c o n s i d e r a t i o n s , d e t e r r e d i m p l e m e n t a t i o n , as w a s t h e case Personal
Involvement
-
with the m a n a g e m e n t . learners, by implementing the results, could only gain from their efforts, the other s t a k e h o l d e r s h a d more at risk. The t e a c h e r s would have h a d to invest a great deal of extra time a n d energy in a d a p t i n g their l e s s o n s to s u i t the n e e d s of each individual s t u d e n t . F u r t h e r m o r e , t h e y t e n d e d to s u s p e c t t h a t t h e y would not be c o m p e n s a t e d for their efforts. In addition, the m a n a g e m e n t h a d already s p e n t a s u b s t a n t i a l a m o u n t of m o n e y on d e v e l o p i n g t h e m a t e r i a l s , a n d b e i n g i g n o r a n t of t h e c u r r i c u l u m development process, w h i c h c a n be a l e n g t h y a n d expensive endeavor, felt t h a t the materials were satisfactory as t h e y were a n d could be u s e d in their original form w i t h o u t a n y f u r t h e r investment. Alkin & Coyle refer to this as utilization m i s u s e or a b u s e , a n d categorize it as "shelving t h e e v a l u a t i o n findings to reduce the likelihood of utilization of results." (1988:337). In f u r t h e r s u p p o r t of our findings, C o u s i n s & Leithwood (1986) have claimed t h a t evaluation Fmdings which were c o n g r u e n t with decision-maker e x p e c t a t i o n s were u s u a l l y easily accepted a n d utilized, while i n c o n g r u e n t findings were u s u a l l y ignored. In our case the findings were, to a certain extent, i n c o n g r u e n t with the a d m i n i s t r a t i o n ' s expectations of n o t having to Risk
- While
the
risk a n y more f u n d s for revision. additional issue is the i m m e d i a c y of feedback or profit gained b y , t h e utilization of the results. While the s t u d e n t s received i m m e d i a t e feedback regarding a n y positive c h a n g e s in their skill mastery, the l e n g t h of time it would have t a k e n the m a n a g e m e n t to profit from its i n v e s t m e n t in the project prevented it from i m p l e m e n t i n g the findings of Immediacy
of Profit -
An
the diagnostic tests. Finally, we found t h a t since the s t u d e n t s were not entirely d e p e n d e n t u p o n the o t h e r s t a k e h o l d e r s in a p p l y i n g the r e s u l t s of the diagnostic tests for their purposes, utilization could be partially e n h a n c e d .
Dependency
-
Needs of Stakeholders
147
However, t h e t e a c h e r s a n d t h e c u r r i c u l u m t e a m were d e p e n d e n t u p o n d e c i s i o n s t a k e n b y the i n s t i t u t e ' s a d m i n i s t r a t i o n a n d b y t h e r e s o u r c e s provided b y it. Therefore, a decision t a k e n by the a d m i n i s t r a t i o n to deter or p o s t p o n e i m p l e m e n t a t i o n resulted in the non-utilization of the results on the p a r t of the d e p e n d e n t stakeholders.
Conclusion By t h e i r very n a t u r e diagnostic t e s t s provide i n f o r m a t i o n regarding deficiencies t h a t can be remedied. Employed in a n evaluation project t h e y c a n serve t h e n e e d s of v a r i o u s g r o u p s of s t a k e h o l d e r s . Their detailed criteria-based r e s u l t s c a n facilitate utilization b y the interested a u d i e n c e s at different stages of the program a n d for a variety of purposes. Nevertheless, since utilization is d e p e n d e n t u p o n d e c i s i o n - m a k i n g a n d c o m m i t m e n t on t h e p a r t of t h e s t a k e h o l d e r s , it is u l t i m a t e l y d e t e r m i n e d by a variety of affective factors. This s t u d y points to the fact t h a t a l t h o u g h the e v a l u a t i o n project a t t e m p t e d to serve the n e e d s of the v a r i o u s s t a k e h o l d e r s b y e n h a n c i n g initial i n t e r e s t a n d r e s p o n s i v e n e s s , w h e n factors s u c h as low p e r s o n a l involvement a n d benefit, v e r s u s high risk and dependency, are at play, t h e probability of n o n - u t i l i z a t i o n increases.
References Alkin, M.C., Daillak, R. & White, P. (1979). Using evaluation: evaluation make a diference? Beverly Hills: Sage.
Does
Alkin, M.C. & Coyle, K. (1988). T h o u g h t s on e v a l u a t i o n u t i l i z a t i o n , misutilization a n d non-utilization. Studies in Educational Evaluation, 14, 331-340. Carroll, B.(1980). Press.
Testing communicative performance.
Oxford: Pergamon
C o u s i n s , J.B. & Leithwood, K.A. (1986). C u r r e n t empirical r e s e a r c h on e v a l u a t i o n utilization. Review of Educational Research, 56 (3), 331364. De Escorsia, B. (1985). ESP and beyond - A quest for real world. In R. Quirk a n d H . G . W i d d o w s o n (Eds.). English in the world. Teaching and learning the language and literature. Cambridge: Cambridge University Press a n d the British Council.
148
I. Geva-May and H. Peretz
Dubin, F. & O1shtain, E. (1986). Course design: Developing programs and materials for language learning. Cambridge: C a m b r i d g e University Press. G e v a - M a y , I. & E d a n - B a m b a g y , O. (1988). A curricular study: A functional~situational approach in business English in Israel. Paper p r e s e n t e d at the 2nd International TESOL Conference, J e r u s a l e m . Geva-May, I., E d a n - B a m b a g y , O., Levitin, H., Peretz, H. & Silbenberg, C. (1987). G.R.O.W. for business, grammar, reading, oral proficiency, wr/ting. Tel Aviv: ESP Curriculum Center, Van Leer - IDEA. G u b a , E.G., a n d Lincoln, Y.S. (1981). Jossey-Bass. Harris,D.P. (1969). McGraw-Hill.
Effective evaluation.
S a n Francisco:
Testing English as a second language.
New York:
Leithwood, K.A., Wilson, R.J. & Marshall, A.R. (1981). I n c r e a s i n g the influence of evaluation s t u d i e s on p r o g r a m decision making. In A. Lewy and D. Nevo (Eds.), Evaluation roles in education. New York: Gordon & Breach. Lewy, A. (Ed.), (1977). Longman.
Handbook o f curriculum evaluation. New York:
Lewy, A. (1986). Formative evaluation of educational pragrammes: The state o f the art. P a p e r p r e s e n t e d at t h e T e a c h e r O p p o r t u n i t y Programmes, J o h a n n e s b u r g . Lewy, A. (1988). Issues in curriculum evaluation. J e r u s a l e m : Education, Israel Curriculum Center.
Ministry of
Nevo, D. (1983). The c o n c e p t u a l i z a t i o n of e d u c a t i o n a l e v a l u a t i o n : An analytical review of literature. Review of Educational Research, 1 5 (I), 1 1 7 - 1 2 8 OUer, J. (1979). Language tests at school. London: Longman. Olshtain,E., Feuerstin, T., Begerano,Y., Scholnik, M., & Smith, K. (1987). A summative evaluation of the Van-Leer IDEA materials for teaching English for professional purposes. I n t e r n a l Report. Tel-Aviv: D e p a r t m e n t of Education, Tel-Aviv University and the Van-Leer IDEA Institute. Patton, M.Q., Grimes, P.S., Guthrie, K.M., B r e n n a n , N.G., French, B.D. & Blyth, D.A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making. New York: Health. S h o h a m y , E. (1985). A practical handbook in language testing for the second language teacher. Tel Aviv: Tel Aviv University Press.
Needs o f Stakeholders
149
S t u f f i e b e a m , D . L . (1971). An i n t r o d u c t i o n to t h e PDK book. In D.L. Stufflebeam, W.J. Foley, W.J. Gephart, E.G. Guda, R.L. H a m m o n d , H.D. M e r r i m a n & M.M. Provus. Education evaluation and decision-making. Itasca, Illinois: F.E. Peacock. Weiss,C.H. (1983). The s t a k e h o l d e r a p p r o a c h to e v a l u a t i o n origins a n d promise. In A.S. Bryk, (Ed.), The stakeholder approach to evaluation: Origins and promise. San-Francisco: Jossey-Bass.
The A u t h o r s IRIS GEVA-MAY is a lecturer in the D e p a r t m e n t of E d u c a t i o n in Science a n d Technology, T e c h n i o n - Israel I n s t i t u t e of Technology. Her fields of r e s e a r c h are project e v a l u a t i o n , c u r r i c u l u m design, LSP (Language for Special Purposes) a n d foreign language learning. HANNAH PERETZ is a post g r a d u a t e s t u d e n t at the School of Education, Tel-Aviv University. She is also a h i g h - s c h o o l t e a c h e r of E n g l i s h in Jerusalem.