Formative evaluation of the individualized science program

STUDIES IN EDUCATIONAL EVALUATION

Volume 1, No. 2

Summer 1975

F O R M A T I V E E V A L U A T I O N OF T H E I N D I V I D U A L I Z E D SCIENCE PROGRAM 1 LEO. E. K L O P F E R

& AUDREY

B. C H A M P A G N E

Learning Research and Developn~ent Center. University of Piltsb~rg/~

I n the course of developing the Individualized Science (IS) program, a t t e n t i o n to formative evaluation has been a c o n s t a n t c o m m i t m e n t . Indeed, even if we were not advocates of formative evaluation ourselves, the social pressure of the organizational milieu wherein IS is n u r t u r e d would enjoin us to conduct appropriately thoughtful. careful evaluations and assessments t h r o u g h o u t the long process of curriculum development. The Learning Research and Development Center (LRDC) at the University of P i t t s b u r g h considers formative evaluation a necessary component in the develop m e n t of instructional materials, and n u m e r o u s tbrmative evaluation studies on LRDC's various programs have been carried out in recent years (e.g. Lindvall and Cox, 1969; Wang, Resnick and Boozer, 1971 ; Wang, 1973). We are not alone atLt{DC, then in believing t h a t it is essential to do a conscientious job of evaluation and assessm e n t when developing a science program t h a t will be used by m a n y i n n o c e n t children. The formative evaluation activities which accompany the developnlent of IS take n u m e r o u s forms and are carried out in various settings and at various times. Since we cannot give a full account of all of these activities in a brief compass, fbr tiffs paper we have chosen only to delineate the four stages in the formative evaluation of IS a n d to discuss our way of answering representative formative evaluation questions. To provide a context for considering the formative evaluation of IS, howcv~'r, w(, should first describe some of the salient t~atures of the IS program, e IS is an individualized elementary-school science program designed to serve children in present school grades K through 8. There are seven levels in the IS program, each of which consists of a p p r o x i m a t e l y one year's work in science. The science content of IS is drawn from both the biological and the physical sciences. One unique aspect of the program is its emphasis on the cultm'al aspects of science and the interactions of science and society. IS is a comprehensively individualized program : not only does IS provide mechanisms for a child to help plan his or her science activities, to manage his or her own instructional materials, and to take part in the assessment of his or her learning, b u t it also provides opportunities tbr the child to make selections from among alternative learning resources and from among alternative units of study. 1 Presented at the 47th Annual Meeting of' the National Association for Research in Sci(,nee Teaching, Chicago, Illinois, 16 April 1974. z Of necessity, this short description of the IS program is sketchy. For more detailed descript ion of the rationale, organization, and operation of IS see Champagne and Kh)pfer (1974a). 109

1 10

KIA)I)FEI{ AND (!HAMPA(;NE

In t h e p l a n for t h e full IS p r o g r a m , t h e r e a r e 19 (.ore s('ien(.e c o n t e n t u n i t s , called M a i n s t r e a m u n i t s , a n d a n a d d i t i o n a l 12 A l t e r n a t i v e P a t h w a y u n i t s . T h e r e a r e also 3 i n t r o d u c t o r y u n i t s t h a t p r o v i d e for e n t r y of t h e s t u d e n t i n t o I S e i t h e r a t t h e b e g i n n i n g o f t h e i ) r o g r a m . Level A. or a t o n e of t h e u p p e r levels. T h e M a i n s t r e a m u n i t s comprise t h e ('ore seienee c o n t e n t t h a t e v e r y s t u d e n t is e x p e c t e d t o s t u d y in t h e IS eurri('u l u m . A l t e r n a t i v e t)athxva5 ' u n i t s a r e o p t i o n a l e x p h ) r a t i o n s whie]~ a r e a v a i l a b l e to t h e s t u ( | e n t , b e g i n n i n g a t Level 1). Ea('h s t u d e n t ix e x p e c t e d t() sele('t a t least o n e A l t e r n a t i v e P a t h w a y u n i t w h e n h e or s h e is in Level E. a n d tw() or m o r e A l t e r n a t i v ( ' P a t h w a y u n i t s in I)oth L e v e l s F a n d (1. T h e n a m e s a,n d t o p i c s o f all t h e lN i n s t r u t ' t i o n a l u n i t s a r e s h o w n in t h e a ( ' ( ' ( ) m p a n y i n g Filzure 1.

lntrodu~t()ry ['nits START and (management a('a(lemi(' skills)

Mainstr'eam ['nits

Alternative Pathway Units

L~rd A

hasic

SIM PS()N (sorting) ( ~,ALI LE() (ohs('rvin~z) 311(!H E LSON (measuring) L~,rd H B I "1~B A N K (('lassitying)

H ( )( ) K E (fi)r('es) ( ' U R l E (physi('al states)

LAUN('H (mangement ha(,kground)

Lerel (~

an(t

s(,ieIlCe

I,AG]IAN(IE (metric mea Sll i'('lll(![|( )

V ESA 1,1 ITS (,,;ystems) BLA(!K (ch('mi('al systems) Lecd l)

LAV()ISI El{ (burning) I)A LT()N (atoms and mole('ules) H A I,I)AN E (hreatlfing) lANK (management t)aekground)

Lerd E alla

s('ien(.e

J() UI,E (energy) BEAI'M()NT (digestion) V()IT (nutrition) L~'r,,I F H A l l V E Y (civculation)

f)OWFLL (water) Lerd (/

B()blELLI (motion) q)UETELET (variation)

('()MESTO('K (plants and animals) IANNAEUN (phmt growth) VOLTA (eleetri,dty) AI~(~H IMEI)E~; (ma('hines) (~OPERNI(!US (solar system) I,YELL (geology) A R R H E N I U S (chemical s<)lutior~s)

HAE( IK El, (rej)rodu('tion) I)P~EW (blood) DARWIN (evolution) MENDEl, (geneti(.s) NEI BERT (mi(q'oorganisms and disclose)

Figure l. Summary of Instructional (;nits in the Individualized Science Program F o r t h e p u r p o s e of a n a l y s i s , we view t h e 1~ p r o g r a m as a s c i e n c e c u r r i c u l u m closely i n t e g r a t e d w i t h a n i n d i v i d u a l i z e d - l e ~ r n i n g m a n a g e m e n t s y s t e m . T h e science c o n t e n t a n d s c i e n c e l e a r n i n / g o a l s of t h e e u r r i c u l u m reflect o u r view t h a t a n e l e m , . m t a r v

INDIVIDUALIZED SCIENCE PROGRAM

II 1

school science program should be multifaceted. The individualized-learning managem e n t system reflects our belief t h a t a child can be t h a u g h t to be responsible fbr his or her own learning. When we say t h a t the c o n t e n t of the science curriculum of IS is multifaceted, we mean t h a t the science c o n t e n t incorporate m a n y kinds of science learning that the child can participate in. Among elementary-school science curriculums promulgated in recent years, there has been a t e n d e n c y in each one to emphasize only one or two kinds of science learning. But, in our view, neither focusing chiefly on the concepts or conceptual schemes of science nor emt)hasizing only the processes used in science is an adequate science c o n t e n t for a science curriculum in these times, and some facile c o m b i n a t i o n of science subject m a t t e r with the processes of scientific inquiry is also insufficient. I n addition to dealing with the concei)ts and conceptual schemes of science and its processes of inquiry, the science c o n t e n t of a contemporary science curriculum must a t t e n d to the social aspects of science, to the nature of scientific inquiry, to means for developing informed a t t i t u d e s toward science and scientists, to information about the functioning of the h u m a n body t h a t children puzzle about, and to applications of science in everday life, so t h a t the curriculum will be of m a x i m u m value to the m a n y different students who are exposed to it. All of these concerns are reflected in the m a n y facets of the science content in the science curriculum of the IS program. I n t e g r a t e d with the IS science curriculum is an individualized l e a r n i . g m~uagem e n t system. A learning m a n a g e m e n t system is a practical necessity for an individualized program which utilizes m a n y different kinds of materials, as IS does. The materials to be managed in the IS program include m a n y separately printed booklets for each lesson and investigation, reading select ions, activity cards, various worksheets. answer sheets, p l a n n i n g booklets, and keys books. There are m a n y kinds of manipulative materials, some of which are contained in kit boxes, others in a central supply area. E v e r y activity with a large a m o u n t of printed m a t t e r to be read by the s t u d e n t has a read-along audiotape. B u t materials m a n a g e m e n t is only a part of the challenge of m a n a g i n g an individualized program. Students in IS progress at different rates. engage in different learning activities, and follow different learning sequences. Students have different interests and have opportunities to make choices from among available learning resources and alternative units of study. Learning plans for each s t u d e n t have to be made and recorded, and the s t u d e n t ' s progress through his or her plan needs to be documented. All these aspects of individualized learning, as well as the m a n a g e m e n t of the large n u m b e r and variety of learning materials, are facilitated by the IS individualized-learning m a n a g e m e n t system. I t is a system designed for each s t u d e n t to learn how to manage and take increasing responsibility for his or her own learning. Even from this short description of the IS program, it should be clear t h a t the formative evaluation of the program must be cognizant of both the effectiveness and effects of the IS science curriculum and the practicality and effectiveness of the IS individualized-learning m a n a g e m e n t systen. Since the IS science curriculum is closely integrated with the m a n a g e m e n t system, making a sharp distinction between them has not always been possible as we persued the development and formative evaluation of the program. Nevertheless, neither c o m p o n e n t has been ignored at a n y step in the process of developing IS. The steps in this lengthy process, as it applies to the developm e n t of a u n i t in IS, m a y be encapsulated as tollows: conceptualization of the unit

1 12

KIA)I)FER AND (!HAMt'A(~NE

anserw~tions, aneess form the ha,ekground for. anpment arc' examined by respeeted educators and suhjeet m a t t e r experts. Their assignment is to compare the IS p r o g r a m s goals and rationale with accepted trhilosophieal and psychological prineiph~s, s<><'ietal ~oals of education, and the structure of the p e r t i n e n t seienee disciplines. The second substage of Stage A is the critical review of the stu¢tent instructional material of rhe program. The materials ave evaluated on the basis of their consistency with generally accepted lwineiples of pedagogy, with the sta, ted philosophy of the curriculum, and with the accepted facts and ideas of science. In the n in Stage B is t IS is usetirl fi)r i(tentit~dng weaknesses within learning activities and instructional materials. Revisi(ms Oflwot,otyl~e materials and producers are made on the basis of this inflwmati
INDIVIDUALIZED SCIENCE PROGR,AM

l 13

The Stage C formative evaluation of IS seeks answers to questions a b o u t the IS science eurrieulum's i n t e r m e d i a t e - t e r m effects on s t u d e n t s ' behaviors relating to science content, the n a t u r e of the learning e n v i r o n m e n t and its effeet.s on s t u d e n t s and teachers' behavior, and the ease with which the program can be implemented and used by teachers in various types of n o n e x p e r i m e n t a l classroom. Many of the questions addressed in the Stage C evaluation are the same as those addressed in the Stage B evaluation. There are two notable differences. The first is that, during the Stage B evaluation, observations focus on the ease of use and effectiveness of IS learning activities in modifying the individual s t u d e n t ' s short-term behaviors related to seienee eontent, whereas the Stage C evaluation, while not neglecting the short-term behaviors, focuses more on the intermediate term behaviors of students. The second difference is t h a t the Stage B evaluation of IS is carried out in one or two prototype testing schools with small samples of students a n d often only a single teacher is involved ; the Stage C evaluation on the other hand, is conducted in a larger numl)er of field test schools and involves a larger n u m b e r of teaehers. 4 At the completion of the StcLge C formative evaluation of IS. the effectiveness of the instructional materials, the ease of their i m p l e m e n t a t i o n , and the effectiveness of the teacher preparation program have been evaluated with a limited sample of students, teachers, and classroom types. During the Stage D evaluation, more comprehensive answers to questions concerning the effectiveness of the teacher preparation program and the ease with which the IS program can be implemented are sought. Data collected from a larger, selfselected population, are evaluated on the same basis as in the Stage C evaluation. The m a r k e t a b i l i t y of the IS program is also evaluated during this stage by d e t e r m i n i n g the rate at which the program is purchased by schools and whether or not schools continue to use IS after their first, year's experience with it. Moreover, data from children, teachers, and schools using IS will be eompared with data from control sehools. The results of the Stage D evaluation will, of course, be of considerable academic interest and will be of use in preparing revisions of the IS program. At this writing, we have only initiated the p l a n n i n g for the Stage D evaluation, so that. this is not as yet a reality t h a t can be reported upon. On the other hand. we can report on the ways we have used to seek answers to several questions during the earlier stages of the formative evaluation of IS.

ACHIEVEMENT IN SCIENCE CONTENT As we have pointed out above, the science e o n t e n t of the IS science eurrieulum is multifaeeted, b u t we can e o n v e n i e n t l y illustrate our fi)rmative evaluation procedures with respeet to achievement in science content by using examples of aehievement in seienee subject m a t t e r knowledge. Our formative evaluations at, Stages B and (! have asked primarily the following questions concerning subject m a t t e r achievement. ~ 1. How well does the student, learn what. the developers of the curriculum materials intended him to learn ! 4 The Stage C formative evaluation of IS is carried out in eonjunction with Research fiw Better Schools, Inc., (1RBS) in Philadelphia. The obvious reason for this i~ that I~BS is responsible fi~vfield testing the IS program. 5 These questions are taken from Table 3 of our paper cited in flx)tnote 3.

14

K IXH'Iq~]I( AND (:HAMPA(;NE

"2. How ~'an the materials be improved so that the s t u d e n t will learn better 3. Does tile s t u d e n t learn something fi'om the eurrieulum materials other than or more than what the developers intended he would learn ! 4. Upon completion of his interaction with the eurrieuh~m materiMs does the s t u d e n t (lemonstrate the behaviors that the materials purport to teach '! 5. For a n y instructional unit. are the expeeted s t u d e n t behaviors generally too simple or too difficult ti)r the s t u d e n t s in the target population ? In the Stage B f'(wmative evaluation of IS. our approaeh to answering these questions is essentially, lo (,h~selv ()hserve and query s t u d e n t s in the classroom. As students begin to stud 5 a newly (,reate(I unit in one /)f our prototype test, ing sites, a member of the 1S (levelol)nmnt staff is generally on hand to observe and interaet with the stu(lents. In a(tdition to being familiar with the IS program overall, the observer is arme(l with the t)ehavioral obje(.tives for the new unit, so t h a t there is little diflieultv in d e t e r m i n i n g what tilt, (levelT)pers" inten(led outcomes are or what behaviors the materials are designed to teauh. Naturally, in addition to the observer from the development staff', the classroom t e a d l e r is also invoh-ed in the observation of s t u d e n t behav iors and in questioning s t u d e n t s about what they have learned. Her observations are partieularly valuable for discovering what u n i n t e n d e d outcomes s t u d e n t s are learning and what i m p r o v e m e n t s should be made in the instruetional materials. Ahnost all of the science e o n t e n t lessons in IS units have (?heek-Up test, s at tile end t h a t provide a diagnosis of how well the s t u d e n t has learned the subjeet m a t t e r t h a u g h t in the lesson. By reviewing the s t u d e n t s ' (!heek-[Tp answer sheets, we obtain a good indieation of the quality <>f the instruetion in the lessons ~md of whether or not the expected s t u d e n t behaviors are too diffleult flw students in the target population. In St'um~,, (? tiwmatiw, evMuations earried out during the field testing of IS units. somewhat more formal |u'oeedures are used for assessing s t u d e n t s subject m a t t e r aehiewmlent to supplement the procedures described for Stage B. All IS units are fitted out with a pretest, called a placement test, and from the third level of the progranl on. units also have posttests. Both the placement test and posttest of a unit are criterionq'eferenced and are based on the u n i t ' s behavioral objeetives. All items on tile two tests are not necessarily identical, but there is usually sufficient overlap between the two tests to make pre-p<~st comparisons possible. In the field testing version of the ,Ioule Unit in IN Level E, the placement test and posttest consisted of 24 and 25 items, respeetively. Subtests of 19 items from eaeh test were used for a pre-post comparison. The data are shown in Table 1. The t for correlated Table 1. Comparison of 19-1tern Subtests of Joule Unit Placement Test and Posttest (Field Testing Version) ])lacellwllt Test Mean S. I). N l l c l i a h i l i t y (K 1-~ 20)

Posttest

13.32 2.67 !);5 0.,52 t 5.S4(p <

15, 10 2.69 9,5 0.(i2 I~O1)

INDIVIDUALIZED SCIENCE PROGRAM

1 15

groups is 5.84, which is significant beyond the .001 level. This is evidence t h a t the students in the field testing schools gained in their knowledge of the Joule U n i t ' s science subject matter, as measured by the 19-item subtest. Two other points in the data m a y be noted. The posttest mean (15.10) for the 19-item subtest is just at the 80% mastery level (19 × 0.80 = 15.20). While this is very good achievement h),' the group, it also means t h a t not all students a t t a i n e d an 80% mastery level on the subject m a t t e r tested for. Also, the placement test mean (13.32) for the 19-item subtest shows t h a t the s t u d e n t s already had a considerable a m o u n t of knowledge of the Joule U n i t ' s subject m a t t e r before they began to s t u d y the unit. We interpret this finding as an indication t h a t the topic of energy is appropriate for inclusion in an elementary-school science curriculum for students in the 4th to 6th grades, where IS Level E is normally used. Although the test mean d a t a shown in Table 1 provide information a b o u t the s t u d e n t s ' science subject m a t t e r achievement in the, ,Joule Unit, they are not specific enough concerning what the students did or did not learn a b o u t energy. To obtain some clues a b o u t this, we examine individual items on the placement test and posttest and group them according to the lesson topics to which they pertain. Examples of this procedure are shown in Table 2, where proportions of correct stmh,,~t v'~'sponses

Table 2. Proportions of Correct Student Responses (p) to Corresponding Items on Joule Unit Placement Test and Posttest (Field Testing Version) (N = 95) l)la('t,lll~,ll [

Lesson Topic Work

Test Item No.

l%sttest

12

.85

13 14 15 16

.61 .88 .79 .81

Item No.

p

6 7 9 10 11

.98 .78 .99 .93 .95

15 13 20 21

.48 .77 .87 .78

3 4 5

.93 .79 .74

.78 Energy ('.nvcrsion

3 4 7 S

.51 .71 .~14 .66

21 22 23

.83 .75 .75

.93

.63 Measuring Heat Energy

.77

~5

.72

.~2

to three groups of corresponding pre-post items are presented. Figure 2 shows, in reduced form, the actual test items from the Joule P l a c e m e n t Test t h a t correspond to the three lesson topics included in Table 2. E x a m i n a t i o n of Table 2 reveals t h a t the s t u d e n t s who studied the ,Joule U n i t during field testing did very well with the lesson topic of work. The mean proportion of' correct responses to items u n d e r this lesson topic on the posttest was 0.93, and sizable

Work Energy Conversion

Circle the letter of the one best answer on your answer sheet.

8. The energy stored in objects that have been lifted up is called a. sound energy b. kinetic energy c, ~lvitational energy d. heat energy e. elastic energy

Circle the letter of the one best answer on your answer sheet.

7. The energy in a stretched rubberband or a compressed spring is called a. gravitational energy b. ~ n e ~ c energy e. sound energy d. elastic energy

Circle the letter of the best answer on your answer sheet,

4. Which of the following energy conversions takes place in a light bulb? 8. electrical to light b. electrical to sound c. light to kinetic d. kinetic to electrical e. elastic to light

On your answer sheet, circle the letter of the correct answer.

3. Which of the following is a system that converts chemical energy into electrical energy? e. light bulb b. battesy c. buzzer d. candle e. person

Lesson Topic:

35°C

40Oc

Measuring Heat Energy

>

35Oc

On your answer sheet circle the letter of the one best answer.

23. HOW much heat energy must you add to a kilogram of water at 30°C to raise the temperature to 35°C? a. I kilocalorie b, 5 kilocaloties c. 6 kilogram C d. 6 5 C e. 33 kilogram C

~)Oc~

Circle your answer.

22. Which has more heat energy in iL a teacup filled with water at 75°C or a bowl Idled with water at 75°C?

Circle A, B, or C on the answer sheet.

21. Each of three beakers has the same mass of water in it. Which water contains the most heat energ-/?

3~°C

Lesson Topic:

Figure 2. Sample Items from Joule Unit Placement Test (Field Testing Version)

16. Which block has the same gravitational energy as block A? Circle B, C, D, or E on the answer sheet,

15. Which block will do the least work when it falls: A, B, C, D, or E? Circle your answer on the answer sheet,

14. Which block will do the most work when it falls: A, B, C, D, or E? Circle your answer orL the answer sheet.

A l l five blocks are made of the same kind of wood.

13. All three girls drop their balls from the top of the walt onto the sidewalk. Whose ball will bounce the highest? Circle the name of the girl whose bag will bounce the highest.

12. Who does the most work lifting her ball? Circle the name of the gift who does the most work.

The girts lift their balls up to the top of the walt.

Sally, Jane, and Pat are identical triplets. Each ~¢~1has a ball. All thlee balls are made of the same kind of robber.

Lesson Topic:

>

7,

INDIVIDUALIZED SCIENCE PROGRAM

117

gains were registered over the p l a c e m e n t test performance. The case was somewhat different for the lesson topic of energy conversion. Though there was some gain from p l a c e m e n t test to p o s t t e s t here, the mean p r o p o r t i o n of correct responses to items under this lesson topic on the p o s t t e s t was only 0.72, with considerable variation in the proportions of correct responses to individual items. F o r the lesson topic of measuring heat energy, the comparison of mean p r o p o r t i o n s of correct responses to items reveals t h a t there was essentially no gain from p l a c e m e n t test to posttest. These findings are of value to the curriculum developer, for t h e y provide clues a b o u t where revisions m a y be needed in the instructional materials. I n this instance. we learned t h a t the lessons t h a t d e a l t with the topic of work were p r o b a b l y satisfactory. On the other hand, the instruction in the lessons on energy conversion a n d measuring h e a t energy could benefit from a r e e x a m i n a t i o n . The net result of this kind of procedure is t h a t an i m p r o v e d version of the IS science curriculum is finally produced, and this gives us greater confidence for disseminating the IS p r o g r a m to children in schools.

LEARNING MANAGEMENT A very i m p o r t a n t consideration in the f o r m a t i v e e v a l u a t i o n of IS is the determination of the o p e r a b i l i t y and effectiveness of the individualized learning m a n a g e m e n t system. Our a p p r o a c h to this problem rests on the realization t h a t an assessment of a learning m a n a g e m e n t system cannot be m a d e unless its elements are specified as carefully and completely as possible. W i t h regard to the IS program, this specification begins with the s t a t e m e n t s of two of the p r o g r a m ' s five goals. Following are the s t a t e m e n t s of the S t u d e n t Self-Direction and S t u d e n t C o - E v a l u a t i o n Goals, which relate to the individualized-learning m a n a g e m e n t system of IS, and an e x p l i c a t o r y p a r a g r a p h for each goal. 6 STUDENT SELF-DIRECTION

G O A L : The student views the learning process as

primarily self-directed and self-initiated. This goal emphasizes the s t u d e n t ' s d e v e l o p m e n t into a c o m p e t e n t and confident i n d e p e n d e n t learner. As an i n d e p e n d e n t learner, he is able to select and utilize a suitable learning e n v i r o n m e n t and instructional materials t h a t will lead him t o w a r d desired knowledge, insight, or satisfaction. He is also able to specify and follow a fairly longterm plan for his own learning. S T U D E N T C O - E V A L U A T I O N G O A L : The student plays a major role in evaluating

the quality, extent, and rapidity of his lerning. As the s t u d e n t develops into an i n d e p e n d e n t learner, e v a l u a t i o n of his learning b y a teacher or someone else should g r a d u a l l y decrease. The s t u d e n t should assume continually increasing responsibility for judging how" well he performs in learning new information, ideas, a n d procedures. W h e n a s t u d e n t has become responsible for e v a l u a t i n g his own learning, he will be able to set criteria for the completion or m a s t e r y of a learning t a s k and recognize t h a t he has completed his t a s k upon meeting these criteria. He will also be able to assess his progress on a learning t a s k as he

6 Excerpted from Individualized Science, Level C, Teacher's Man~tal.

1 18

KLOI'FER

AND CHAMPAGNE

ln'Oeeeds, hy a n a l y z i n v t h e d i f f i c u l t i e s he e n c o u n t e r s , r~,\i-iug his a p p r o a c h if' n e c e s s a r y , a n d s e e k i n g o u t a s s i s t a n c e if n e e d e d . T h o u g h t h e s t a t e m e n t s o f g o a l s tell us w h e r e we are h e a d e d , t h e y are still m u c h too g e n e r a l t,o g u i d e t h e d e v e l o p m e n t o f a l e a r n i n g m a n a g e m e n t s y s t e m or its e v a l u a t i o n . F o r t h i s r e a s o n , w e h a v e specified a set o f e o m p e t e n e i e s u n d e r e a c h goal for e v e r y level o f t h e I S p r o g r a m . E x a m p l e s o f t h e s e e o m p e t e n e i e s in s t u d e n t s e l f - d i r e c t i o n a n d c o - e v a l u a t i o n at, s e v e r a l I S levels are s h o w n in F i g u r e 3. T h e r e a d e r will n o t e t h a t t h e eompet, e n e i e s are s t a t e d in b e h a v i o r a l t e r m s , m a k i n g t h e i r a s s e s s m e n t a f e a s i b l e proposition. Level in Whieti (!ompetency Is Introduced

l,evel A

"-~ ~l,l,,.t S e l f ' - l ) i r . . t i, ,r,

~,lufh,nt ~',, I']\ahmti,m

When given the opportunity to ~ork in seiem.e, the student obtains his or her own folder and proceeds to work aeeording to the infm'mation in the folder.

After <,ompleting a self-<,heek page (identified by a key around page numbers) in a lesson booklet, the studellt compares his or her own answers with those on a model [)age which is completed c¢wrectly.

Using his Planning Booklet or Planning Sheet. the student identifies the activity he or she is to do and obtains the designated materiMs needed. Upon eomplet ing the activity, the student returns the materials to their correct storage ph~ees. I,evel B

Given the opportunity to select a Student Activity. the student chooses one which interests him or her fi'om among those avail able. and records on the Plamfing Booklet or Planning Sheet the code for the avtivity and the date on which he or she does it.

The student corrects his or her x~ork on selected booklet pages usin}z a key.

Level ('

Upon the completion of each day's activities in science, the student writes the date in the appropriate spaces on his or her Planning Booklet or Planning Sheet for each ¢~etivity as a record of what was done that day.

After eonqdeting a vheek up and obtaining a key fbr it. the student vompares his or her completed eheek up with the key. assigns point

( ; i v e n the list of Minature E x p l o r a t i o n s ( M i n E x ' s ) a v a i l a b l e in a unit att(t having examined the M i n E x

booklets and kits, the student chooses the MinEx's which he or she would like to do and records his choiees in the Planning Booklet.

wdues

to correct

answers,

all(] eet]-

eulates the total points for the cheek H]).

In a unit with one or more integrat ing cheek ups (i. e., a cheek up which ties together the content of sew~ral lessons, readings, and/or other aeti vities), the student partieipateswit h the teacher in making a decision about when he or site is ready to work on a cheek up.

I N D I V I D U A L I Z E D SCIENCE PROGRAM

Level in Which Competency Is Introduced

Level I)

Student Self-Direction

Given the list of Readings in Science (RIS's) associated with particular MinEx's, individual lessons, or other learning resources in a unit, the student selects the RIS which he or she wants to do next and reads it at the appropriate time.

Upon completion of the Unit Overview" for an Alternative Pathway unit, the student decides whether or not he or she will work in that unit and discuss the reasons tbr' the decision in a Student-Teacher Conference The student decides which Invitations to Explore (ITE's) in an :\lternative Pathway unit he or ~he will read and records his or her ,.hoices in the Planning Booklet. Level E

(:liven the opportunity to read through a Placement Test, the student decides whether or not he or she will attempt the test. If the decision is not to take the test, he or she informs the teacher of this dceision and includes learning activ ities from all the unit's topics in his or her' individualized learning plan.

119

Student Co-Evaluation

In a unit where a unit mastery test or activity is available, the student participates with the teacher in making a decision about when he or she is ready to attempt the test or activity. As a part of participation in an Alternative Pathway unit, the student in a Student-Teacher Conference assesses his or her accomplishments in the unit, and suggests ways in which his or her work in the unit might have been improved.

After completing and correcting a unit mastery test and given a list that relates questions on the test to topics considered in the unit. the student identifies those topics he or she has not learned satisfactorily. Upon completion of the analysis of a unit mastery test. the student uses activities related to the topics not learned satisfactorily to design a remedial learning plan and discusses this plan with the teacher.

Figure 3. Representative Competeneies in Student Self-Direction and Co-Evaluation for the IS Program I n t h e S t a g e B f o r m a t i v e e v a l u a t i o n o f IS, t h e s t u d e n t s e l f - d i r e c t i o n a n d c o - e v a l u a t i o n c o m p e t e n c i e s are a s s e s s e d b y d i r e c t o b s e r v a t i o n d u r i n g p r o t o t y p e t e s t i n g . T h e o b s e r v a t i o n a l t e c h n i q u e s t h a t w e use for t h i s p u r p o s e a r e n e i t h e r n o v e l n o r e x t r a o r d i n a r y . Since t h e b e h a v i o r s t o b e l o o k e d for h a v e b e e n s p e c i f i e d for e a c h c o m p e t e n c y . it is n o t a difficult m a t t e r t o p r e p a r e a n o b s e r v e r ' s c h e c k l i s t t o b e u s e d w h e n o b s e r v a tions on a s t u d e n t are made. Periodic o b s e r v a t i o n s by a person familiar with the I S p r o g r a m , o r s o m e t i m e s b y t h e c l a s s r o o m t e a c h e r , allow us t o d e t e r m i n e if a s t u d e n t

120

KL()PFER AND (!HAMI'AGNE

d i s p l a y s a speeific b e h a v i o r c o n s i s t e n t l y , occasionally, or not at all. For c o m p e t e n c i e s t h a t call f~{r t h e s t u d e n t to r e c o r d i n f o r m a t i o n on his or her p l a n n i n g b o o k l e t , e x a m i n a tions of t h e w r i t t e n record tell us if t h e s t u d e n t has m a s t e r e d t h e skill i n v o l v e d . F r o m f o r m a t i v e e v a l u a t i o n s t h a t we carried o u t q u i t e e a r l y in the ( l e v e l o p m e n t of t h e IS p r o g r a m , we f o u n d t h a t t h e s t u d e n t ' s l e a r n i n g of t h e skills i n v o l v e d in m a n a g ing m a t e r i a l s , p l a n n i n g l e a r n i n g , and assessing progress could n o t be left to ehan<'e. C o n s e q u e n t l y , we s y s t e m a t i c a l l y i n t r o d u c e d into t h e p r o g r a m ' s i n s t r u c t i o n a l m a t e r i a l s and p r o c e d u r e s t h e n e c e s s a r y sermativc e v a l u a t i o n s also i n f o r m e d us of t h e need f<)r and t h e a p p r o p r i a t e c o n t e n t of an i n t r o d u c t o r y u n i t for children new to t h e IS p r o g r a m . (!l)eration. An essential p a r t of ()ur f o r m a t i v e e v a l u a t i o n a c t i v i t i e s is to i d e n t i f y t h e s e facets and to assess t h e i r effects
INDIVIDUALIZED SCIENCE PROGRAM

121

individualized-learning m a n a g e m e n t system itself. How this happened can best be described by means of an historical narrative. I n the earliest days of our d e v e l o p m e n t efforts, the focus of the formative evaluation was on the interactions of children with instructional materials a n d on the outcomes of these interactions. These studies were carried out in the science classroom of an experimental e l e m e n t a r y school. Elaborate procedures were i n s t i t u t e d to observe children's interactions with instructional materials a n d to test children's learning of the prescribed science content. Once obvious problems with the instructional materials were solved, our observations of the classroom's operation convinced us t h a t we did indeed have a smoothly operating individualized science curriculum. However, our security vanished suddenly when it was decided to make IS widely available to schools. Our decision to "go public" drastically changed the perspective of our classroom observations. No longer could we observe our science curriculum as researchers studying the learning process in an individualized setting. We were now faced with the problem of exporting an individualized science curriculum to the real world. The classroom we saw was hardly the real world; for one thing, the room had too m a n y adults - a teacher plus an aide. How, we asked ourselves, can this curriculum function in a real-world elementary school classroom with t h i r t y children and one teacher .~ We began our a t t a c k on this question by observing the adults in the classroom to find out what they were doing. Then we analyzed our data to determine those activities the adults were engaged in t h a t were essential to the curriculum and those which were not. This analysis convinced us t h a t most of the adults' activities were essential for operating the individualized curriculum. We faced a dilemma: we wanted children to use our IS curriculum, b u t we knew that if an aide were necessary for the curriculum to function, most schools would not be able to afford IS. Therefore. we focused our observations on the aide's activities. We saw t h a t most of her time was spent in dispensing and collecting printed and m a n i p u l a t i v e materials, grading paper and pencil tests, keeping records, and administering oral tests. All of these activities were essential to the smooth functions of the individualized curriculum. Clearly, if the aide were to be eliminated, these activities must be carried out by some other person or persons. Our observations of the teacher indicated t h a t her time was already completely taken up with activities more directly related to instruction. The only other people power in the classroom resided in the children. The next question to which we addressed ourselves was: Could the children take over ~ny of the responsibilities of the aide ? The answer, we decided, was yes. With some organization of materials and a mechanism for informing the child what he or she needed for a particular activity~ the children, we believed, could take the responsibility for m a n a g i n g their own learning materials. And, upon ~ little reflection, we realized t h a t giving the children this responsibility was a first step in the realization of the S t u d e n t Self-Direction Goal t h a t we had set for the IS program. Our original conception of the S t u d e n t Self-Direction Goal had been considerably more lofty and philosophical t h a n the very practical way in which we were now viewing it. However, we had no difficulty now with the rationale t h a t the organization of the things of learning is a necessary first step in a person's development toward becoming a selfdirected learner.

122

KLOPFER AND CHAMPAGNE

Two other t i m e - c o n s u m i n g activities of the aide were the grading of s t u d e n t tests and the keeping of records. The process t h a t brought us the realization t h a t the child could take responsibility for checking and scoring his or her own tests was quite similar to the process described above for materials m a n a g e m e n t . We realized that giving the child the responsibility tbr test checking and scoring solved a practical problem, had potential for e n h a n c i n g learning, and was philosophically quite consistent with our S t u d e n t C o - E v a l u a t i o n Goal. As we i n v e n t e d mechanisms to i n s t i t u t e the responsibility for materials m a n a g e m e n t and test evaluation, the responsibility for recordkeeping n a t u r a l l y t~ll to the child. A p l a n n i n g booklet was the mechanism we had devised to inform the child a b o u t what learning materials were needed to complete an activity, and this p l a n n i n g booklet also became the place where the teacher and the child recorded the child's progress through the IS eurrieulum. The responsibility the aide had tbr a d m i n i s t e r i n g oral tests was the most difficult to reassign. I t was clear t h a t earefully structured oral tests provided the most valid information to us regarding the child's learning of seience content. B u t the mechanism ibr their a d m i n i s t r a t i o n was much to() cumbersome for an elementary school teacher to handle for 30 children. After some t h o u g h t and discussion, we came to realize t h a t the elaborate oral testing procedures were artifact of the formative evaluation of children's learning t h a t could be eliminated in the f n a l version of the IS program. Consequently, we s u b s t i t u t e d self-administered, diagnostic Check Up tests in the science c o n t e n t lessons of" the IS units. Of course, our formative evaluation data provided us with m a n y good distractors to use in our ('heck Up test items. Data for the formative evaluations of the individualized-learning m a n a g e m e n t system were collected ibr different reasons and from different perspectives. I n certain respects, the d a t a and the changes they implied seem trivial, but the m a n y small modifications made in m a n a g e m e n t procedures had a signifieant c u m m u l a t i v e effect on the individualized-learning m a n a g e m e n t system. We believe t h a t our formative evaluations have been crucial in helping us to incorporate an operational and effective individualized-learning m a n a g e m e n t system into the IS program.

REFERENCES ('HAMPAGNE, A. B. and KLOPFER. 1,. E. An Individualized Elementary-School Science tq'o7d'am. Theory Into Practice, 1974, 13 (2), 136-148, (a). (!HAMPAGNE. A. B. and KLOPFER, L. E. Formative Evaluation in Science (!urrieulum Development. Jo~l,',al of Re.~earch i~ Science Teaching, 1974, l I (3), 185 203, (b). Individualized Science, Level C, Teacher's Manual. Kankakee. Ill. : Imperial International Learning Corp., 1973.6 7. LINDVALL, C. 3I. ~nd COX. R.. C. The Role of Evaluation in Programs of Individualized lnstrue tion. In: 681h Yearbook of tl~e Xatio~al Society jbr the St~My oJ" Educatio~. ('hieago: XSSE, 1969. WANG, M. C.. RESNICK, L. B. and BOOZER. R. F. The Sequence of the Development of some Early Mathematics Behaviors. (Tffld Developn~etlt, 1971. 42, 1967 1978. WANG, M.C. The Accuracy of Teacher's Predictions of(!hildren's Learning Perfbrmanee. Jo,r~,d qf Educational Re,vearct,, 1973, 66, 462 665.