- Email: [email protected]
Evaluation of impact in Israel: Israel curriculum centre
Studies in EducationalEvaluation.Vol. 13, pp. 263--274, 1987
0191-491X/87 $0.00 + .50 Copyright O 1987 Pergamon Journals Ltd.
Printed in Great Britain. All rights reserved.
EVALUATION OF IMPACT IN ISRAEL: ISRAEL CURRICULUM CENTRE Arieh Lewy School of Education, University of Tel Aviv, Tel Aviv, Israel
The Israel Curriculum Centre (ICC) started its professional activities in 1966. Seldom has an institute had such favourable conditions at its birth as did the Israel Curriculum Centre. Seldom could an institute begin its work after such meticulous preparation. By the time of the Centre's establishment the State of Israel had suceeded in solving one of its most vexing problems. It had put all schools under a roof, and had provided physical facilities such as classrooms, furniture, etc. for all pupils despite the unprecedented growth in the school population. Instructional materials, such as textbooks and syllabuses for all subjects taught in schools became available for teachers and learners. At the beginning of the 1960's the Minister of Education could say that no child in Israel was deprived of attending a school operating under decent conditions. The achievement enabled the authorities to orient their efforts towards the improvement of educational practice, and to focus on the content of education rather than on physical facilities. It was the time of great messages. The re-discovery of the Tyler rationale, the publication of the Bruner and Gagne books. Schwab's conceptualization of curriculum, programmes known by the acronyms, BSCS, PSSC, etc. created enthusiasm among curriculum experts, and Nuffield ceased to be only a geographical term. Educational experts believed in the power of rational planning and in the merits of 'teacherless' and 'errorless' programmes. This atmosphere Curriculum Centre.
no doubt
influenced
the Ministry's
decision
to establish
a
When the Curriculum Centre was established, Israel had a highly centralized educational system. It was taken for granted that curriculum development, too, was to be carried out by the central agencies of the Ministry and that the products of these developmental activities would be desseminated throughout the system. Accordingly, the Israel Curriculum Centre began working as a national research and development institute. It adopted the principles of the R-D-D (Research, Development and Diffusion) approach (Havelock 1969), and operated according to the Central-Periphery line of production (Eden, 1985).
263
264
A. Lewy
Since then, changes have occurred in the dominant pedagogical conceptions of the country. Greater autonomy was bestowed upon the six regional offices of the Ministry of Education and Culture, and autonomy of schools themselves was increased considerably. Parallel to these changes the Curriculum Centre increased the scope of its activities, and in addition to the R-D-D production pattern it became involved in joint collaborative endeavours with the regional offices of the Ministry. The ICC has encouraged and monitored local and school-based development. It invited other institutes and organizations to contribute to the national effort of curriculum development. University departments, teacher training colleges, museums, research institutes, health institutions, and voluntary organizations responded to the invitation and many became involved in curriculum development activities. In 1986 the ICC no longer had a 'monopoly' as a curriculum development institute in Israel. It became one of several institutes doing such work. Under these conditions it became necessary to establish a national co-ordinating body for curriculum development. Due to historical circumstances, such as the seniority of the Institute, its affiliation to the Ministry of Education and Culture, the academic qualifications of its staff members, and so on, its suitability as a co-ordinating body was not seriously contested. In 1986 the ICC has approximately forty five full-time job-equivalent professional positions, which are occupied by about 120 persons. Only a few of them work at the Centre in a full-time capacity. The rest have a part-time position at the ICC, and complement their work by teaching in schools, teacher training institutions, universities, or by acting as school supervisors. The ICC also monitors the work of curriculum development activities at various institutions outside the ICC. These bodies employ a comparable number of professional curriculum specialists. Its annual budget including staff salary and funding external projects, equivalent to three million dollars.
is
The Modus Operandi of the Israel Curriculum Centre From the very beginning the major operational units within the ICC have been the Subject Teams. A Subject Team had to take care of all professional activities which were related to the school curriculum in one subject matter, such as mother tongue, mathematics, geography, science, history, music, etc. Originally the sole responsibility of a Subject Team was to produce new p r o g r a m e kits (including textbooks, teacher's guide, instructional aids, etc.) in a single subject. Thus, for example, the Science Team prepared the above mentioned items for all three grades of the comprehensive school. They also designed laboratory equipment, kits containing materials for experiments, etc. Gradually, the work of the Subject Team became more and more diversified. First, the need to diffuse curricular products in the whole system imposed upon the teams the task of contacting supervisiors, school principals, teachers, teacher training institutes, etc. They were requested to organize curriculum oriented in-service training activities. Later, they had to assume responsibility for monitoring school-based curriculum development activities, preparing computerised learning modules,
Israel Curriculum Centre developing monitoring
a 'second generation set' of curriculum materials, and the curriculum development activities of other institutes.
above
265
all,
In addition to the Subject Teams which operated as production units, several service units were established in the Centre such as the Graphics and Printing Unit, the Library and the Evaluation Unit. As a result of the diversification in the team's work several new non-subject related support units were created within the ICC; an Implementation Unit, a School-Based Curriculum Development Unit, and a Computerised Instructional Materials Unit which provided support for various Subject Teams. The new units came to assist the Subject Teams in their work rather than to restrict the freedom of their operation. Of course, the mere existence of an Evaluation Unit or an implementation Unit reflected a broad consensus within the ICC that these functions should be attended to, and no Subject Team could fully disregard them.
Current Evaluation Activities The empirical evaluation studies grouped into two major classes:
carried
out by the Evaluation
unit
can be
(i)
Small scale and short duration (approximately one to six months) formative evaluation studies related to material produced by the Teams or difused by them.
focused Subject
(2)
Large scale studies which touched upon pervasive issues of the whole educational system, and cut across limits of particular subject areas. Thus in such studies several Subject Teams were often involved, for example: identifying Educational Needs, Policy Research, Implementation Evaluation, Criteria for Evaluating Computerised instructional Units etc.
An Overall Evaluation of ICC Activities Evaluation has constituted an integral part of the ICC since its establishment. Originally it concerned itself with evaluating the products prepared by the Subject Teams and the activities initiated or carried out by their members. Studies of this type improved the quality of the evaluation targets, demonstrated the merits of various programmes, and persuaded the prospective clientele to use the products. Beyond ensuring and demonstrating the quality of a particular product or the efficiency of a particular action, however, the management of the ICC was challenged to answer more pervasive questions about its operations. Questions were asked about the capability of the ICC to carry out all tasks delegated to it in due time. Could the ICC effectively reach the potential clientele of its products in various sectors of the system? Was it desirable to maintain a national centre for curriculum development? Opinions were expressed that the existence of a national diminished private initative in curriculum development.
curriculum
centre
To answer these questions in the 1980's, the Evaluation Unit of ICC became involved in several studies which examined ICC activities in a holistic way and tried to provide a global perspective of its operation. It would be impossible
266
A. Lewy
to present details about all studies which touched upon the global operation of the ICC, and therefore we will limit our report to illustrative examples of three types: empirical studies, analytical pattern of institutional self-evaluation, and an external auditing study paradigm.
Focusing on the Whole Institute Empirical
Studies
Two empirical studies ethnographic study.
will
be
reported
here:
a statistical
survey
and
an
Survey of utilization. The Graphics and Printing units regularly provide annual statistics about the sale of products prepared by the Subject Teams (Dorr 1985). While the figures of these reports provide information needed about the diffusion of the products, it was felt that more information was needed about the diffusion and utilization of curricular materials. The sales figures do not provide sufficient information about the extent of use. A national survey based on a sample of about I00 schools examined the degree of utilization of various curricular products prepared by the ICC and by other curriculum development institutes in the country. The teachers' perceptions of these products, the teachers' attitudes towards them and difficulties encountered in using these materials were examined (Lewy 1980). The study provided complementary information to the statistics about the annual sale of the products. The survey revealed that only 5% of users did not like About 75% expressed high satisfaction, and the rest satisfaction with the programme. The new programmes were the schools. About 60% of the schools indicated that they instructional materials in more than five subject areas.
the new programmes. expressed moderate highly 'visible' in used the ICC's new
Some programmes were extremely popular and became operational in more than 90% of the schools. Nevertheless, the respondents complained that the new curriculum materials could not readily be used in classes of low ability students or in mixed ability classes. An ethnosraphic
study of four schools.
To broaden the information obtained from the Utilization Survey described above, the Evaluation Unit initiated another impact study which was structured according to the tradition of ethnographic studies in education (Jacobs 1979). It was decided to send an observer to four different schools. The observers were permanent staff members of the Evaluation Unit. They spent three or four days in a particular school, interviewed the school principal and teachers, and observed modes of utilizing curricular materials. The purpose of this study was to examine to what extent the instructional products of the ICC were used in schools, the reactions of teachers and learners to these materials, the teachers expectations with regard to curriculum development institutes, and the extent to which these expectations were met. Some of the findings are described below:
Israel Curriculum Centre
267
Well-structured textbooks describing what to teach and how are more popular among teachers, than those which impose the burden of structuring the programme upon the teacher. The teachers act skillfully in supplementing the existing programme with materials of local character, but they prefer to restrict their autonomy to the selection and supplementation of curricular contents, rather than structuring or organizing. Teachers believe that materials prepared by professional teams are better than those which they are able to produce. The same attitude is felt with regard to the in-service activities. Teachers, and especially principals and supervisors attributed greater importance to the structured courses which cover a specific body of knowledge, than to ad hoc discussions of the problem-solving type (Ebaz et al 1987). These findings may present a gloomy picture for those who strongly advocate school-based curriculum development. Nevertheless, in assessing the impact of the ICC it would be a mistake to interpret these findings as a move towards restricting teacher initiative in the field of curriculum development. One has to view these facts in an historical context. Before the establishment of the ICC the textbooks used in the system were heavily loaded with facts, and teaching focused on cramming. Teachers followed the prescribed syllabus and had little freedom to deviate from what was contained in the textbook. In contrast, the textbooks developed at the ICC invited the teachers to exercise at least a restricted level of autonomy related to some aspects of the curriculum. Teachers responded positively to this challenge, and the discourse among teachers about matters of curriculum changed. Teachers developed a critical approach to textbooks, learned to select and to supplement textbooks, and became familiar with a broad repertoire of criteria for judging the quality of instructional materials. Those who would like to see a shift toward a larger scope of school-based curriculum development in Israeli schools are certainly not fully satisfied with what happened. Nevertheless some supporters of the school-based curriculum movement conceive the impact of the ICC described above as an educative step that may have positive consequences for increasing school-based curriculum development activities in the country. It can be assumed that the establishment of a School-Based Curriculum support unit in the ICC will precipitate such development, and will present to the teacher the alternatives of either selecting structured and semi-structured materials developed by professional curriculum developers, or of developing their own sets of instructional materials.
Analytical
Pattern of Institutional
Self-Evaluation
The ICC regularly runs annual planning workshops. The 1984 workshops focused on assessing successes and failures of past activities. The discussion was based on position papers and analytical summaries of past experiences. Subcommittees tried to review the scholarly merits and the pragmatic consequences of the documents and oral presentations at the Workshop. For this reason one may conceive such an event as an attempt of institutional self-evaluation. The discussion of the Workshop examined the attainments of the Centre according to three criteria suggested by Israeli (1983) to assess the efficiency of organizations: goal attainment, survival of the organization, and acquisition of resources.
268
A. Lewy
In applying these three criteria to the activities of the ICC over the past two decades, one may assign to it a very high rating of success. The changes which occured in the work of the ICC in response to emerging needs were surveyed. It was found that the Centre continuously generated new goals and thus provided justification for its survival; it diversified the areas of its activities, and it increased the scope of institutes and persons with whom it worked and whose work it monitored (Lewy 1984). Analyzing
the problem of the ICC it was stated:
The changes generated problems and tensions which have not been successfully handed. Their solution or reduction is crucial to the continued functioning of the Centre. Firstly, not only have the innovations introduced by the Centre been successfully disseminated in the educational system, but the "know how" of producing such innovative programmes also became a public asset. In other words we have been very good teachers and taught others all our "professional secrets". Unlike industrial companies, we could not and did not want to keep our secrets. Nowadays, we competitors. They contribute to the good, but what is (or do not want to
are not the only tradesmen in the market. There are are our students, and we are proud of them. Their activities improvement of the quality of curricula in schools. This is not good is that many members of the Centre are not cognizant be cognizant) of these facts.
The second problem relates to growth. We grew by splintering, which meant that a Subject Team could obtain additional staff members on the condition that they agreed to assume responsibility for additional tasks. For example a Subject Team could increase the scope of its work, by starting to prepare computerized instructional materials. To carry out this new task, the team split into two, and consequently the size of the original teams became smaller. Parallel with broadening the scope of the centre's responsibility and increasing the size of the staff, under-staffing became a growing concern. Staff enlargement has never matched the enlargement of the scope of responsibilities. As a result of the process of splintering, the number of "one-person teams" or even part-time teams is gradually increasing in the Centre. This has resulted in decreased contact between people working in the Centre. Some people feel lonely inside the Centre and yearn for those old days when larger teams worked together on narrowly defined tasks. Finally, assuming responsibility for new goals and new tasks required the Centre to appoint staff ~embers with different specializations. This required a shift in criteria to evaluate the work of staff members. At the initial stage clear conceptualization and writing ability were the most needed skills for doing good work. Staff members had an academic orientation. The majority held part-time positions at universities and at other teaching institutes and had acquired the habit of publishing papers in scientific journals. The working ethos was akin to that which prevails in academic circles, and the approval granted by the academic community to work done in the Centre was extremely important to staff members. Many tasks of the Centre require interpersonal rather than academic skills, as quite often the skill of speaking is more relevant than the skill of writing. These shifts went together with changes in the relationship between the academic community and staff members of the Centre. Twenty years ago the
Israel Curriculum Centre
269
experts of the Centre were leaders in producing curricular knowledge of academic value. Today the Centre is gradually becoming a place where staff members use academic knowledge produced elsewhere. The Staff Workshop served as a self-evaluation mechanism. Decisions were made to create large working units through combining serveral Subject Teams into one, to establish closer co-operation with various branches of the Ministry of Education and Culture, and to give priority to monitoring the work of all curriculum development institutions in the country, even if it necessiated reducing the scope of direct curriculum development activities of the Subject Teams. These decisions were incorporated into the official policy of the Centre, and at present continuous effort is being made to implement them. Nevertheless, their realization may require much more time than was originally contemplated.
The State Comptroller's
Report
Since the ICC is a branch of the Ministry of Education and Culture, it is subject to the control of the State Comptroller's office. The office examines the operations of all institutes funder by the State, and it publishes an annual report specifying details of any malfunctioning it may uncover. The State Comptroller's office enjoys a high reputation in the ranks of the state bureaucracy. Its director has a status comparable with that of a High Court judge, though the office itself is not a part of the judiciary system. The publication of the annual report is considered an event of national importance and the daily newspapers give it extensive coverage. Institutes that are criticized are expected to react to the criticism and the forthcoming annual reports of the State Comptroller's office usually acknowledge these reactions. While the previous reports contained some comments about the ICC, a team of the State Comptroller's office thoroughly examined the operational routines of the Centre in 1984. The team spent about two weeks at the Centre, reviewed a variety of documents, publication products, etc. interviewed staff members and the summary of their observations was published in the 1984 annual report (Israel 1985). By the very nature of such reports, the focus was on flaws detected in the operation of the Centre. Those aspects of the ICC's work which were found to be satisfactory were not mentioned. Some of the issues dealt with in the report are reported here; the report touched on issues of priority, planning, organizational structure, contact with schools, and co-ordination of work with other branches of the Ministry of Education and Culture.
Priority. The formal goal statements following goal priorities:
issued
i.
Publishing syllabuses for revising or up-dating them.
all
2.
Developing instructional materials contained in the syllabuses.
3.
The diffusion and and textbooks).
implementation
by
study
of
the
areas
ICC
in
1984
(subjects)
which
correspond
the
instructional
to
specified
and
the
the
continuously
specifications
products
(syllabuses
270
A. Lewy
4.
Guiding organizations, institutes and schools materials as specified in the syllabuses.
The report by all Subject distribution of differed widely effort, without
to
develop
instructional
indicated that the order of priority was not strictly observed Teams. Moreover, no sufficient guidelines were given to the efforts across these four different goals. The Subject Teams from each other with respect to the distribution of their providing justification for their idiosyncratic decisions.
Plannin$. The report complained that planning activities of the Centre were deficient. Subject Teams had to prepare an annual plan of work which was submitted for approval to the directorial board of the Centre. The duration of the development work mostly exceeded the limits of a single year, however, and therefore a long-term work plan encompassing activities for two to three years would be needed.
Organizational
structure.
At the time the survey was carried out thirty Subject Teams operated in the Centre. All of them reported directly to the director of the institute. Direct contact with thirty production teams imposed an insurmountable task upon the director. Since Subject Teams represent a great variety of specialisations in science, art and humanities, a single person cannot be adequately qualified to review all these products. Another organizational problem, according to the report, was the confusion between line and staff operational modes within the Centre. The Centre itself assumed responsibility for preparing instructional materials and at the same time also served as a facilitator for other institutes in developing instructional products. There is a contradiction between these activities, since the ICC had to monitor the development of products which could compete with products developed by the ICC itself.
Contact with schools. The report pointed out that in a substantial proportion of schools teachers had never seen the official syllabuses, and that the single tool they used for planning their school programme was the textbook. The situation was deplorable, certainly. The State Comptroller's report blamed the ICC for not ensuring that all teachers could have access to the syllabuses, and thus be able to familiarize themselves with their content.
Co-ordination with other departments
and institutes.
The report stated that the activities of the ICC and other Ministry of Education departments were not sufficiently co-ordinated. As an example, they referred to the discrepancy between the goals set by in-service teacher training activities in the ICC, the Department of In-service teacher training and the teacher training colleges.
Israel Curriculum Centre
271
The Impact of Impact Studies The studies reported in this article examined the impact of the ICC and its products in the educational system. The three types of impact studies represent what Rossi and Freeman (1982) called methodological heterogeneity. One may add that it also represents heterogeneity of study aims. At least three areas of aims were dealt with, or in other words three kinds of questions were asked: Did the institute or a particular product of the institute have an impact? Which characteristics of an entity of of an action may precipitate impact? Is the impact congruent with the goal priority of a certain client? By no means do these questions constitute a full list of aims for impact studies, but they suggest that it might be useful to construct a taxonomy of relevant questions. The legitimacy of methodological heterogeneity brought up the question whether different methods and research paradigms are equally effective. Is it possible to identify certain circumstances and/or specific study aims, where one method produces better results than another one? In an attempt to answer this question hard-line empiricists collected and summarized data to find out whether evaluation makes a difference (Alkin 1974, 1979). Others approached this problem in an axiomatic way (Zetterberg 1965), or in a reflective, analytical way (Lindblom and Cohen 1979). Alkin developed a mapping sentence to distinguish between types of evaluation utilization (such as dominant influence, one of multiple influences, cumulative influence, etc.) and also specified conditions which may contribute to the impact of evaluation studies. At the ICC no studies were carried out to examine the impact of evaluation activities. Therefore, each statement made about the impact of evaluation studies in the ICC has only tentative validity. Nevertheless, it can be said that, within the framework of the operational procedures, great importance has always been given to the utilization of evaluation results. Subject Teams systematically used formative evaluation results to improve the quality of their products, and the pattern of the programme-diffusion activities. Little experience has been accumulated in using evaluation results which touch on the operation of the whole institution. The reorganization of the ICC is at present in process, and sincere efforts are being made to use evaluation results in the process of decision-making. There are signs, however, that it is more difficult to implement recommendations of studies which deal with a whole institution than recommendations of formative evaluation. Cooperative links between the Evaluation Team and each Subject Team pave the way for the utilization of formative evaluation findings. In contrast, the operational pattern of an entire institution represents a delicate compromise between conflicting interests of institutional branches, and of competing institutions. Under such circumstances political and power considerations gain great influence in the process of decision making and they may weaken the impact of evaluation findings. It is unlikely that conflicting parties will be overwhelmed by altruistic motives, and will disregard their particularistic interests in favour of the overall institutional interests (Shapiro 1984). In an institutional setting, impact is a function of adequate con~munication, persuasion, and above all inter-organizational and intra-organizational politics. It is not enough to be right, one has to be able to convince others and to weaken resistance.
272
A. Lewy
During the twenty years of its operation the Evaluation Unit has succeeded in creating a favourable climate towards evaluation. Since this is a crucial determinant of evaluation utilization, it seems appropriate to describe strategies employed by the Evaluation Unit for creating such a climate. These strategies proved to be effective in enhancing the utilization of formative evaluation results, and one may assume that they will contribute also to the utilization of impact evaluation results.
Professional Visibility The Evaluation Unit, from the very beginning of its operation, made an attempt to maintain visibility for its work in professional circles. Priority was assigned to the information needs of direct cliencts, but the clientele was broadly defined. The Subject Teams, therefore, were never thought of as the sole clients. The management of ICC, other branches of the Ministry of Education and Culture, teachers, as well as the mass media, were all considered clients of the Evaluation Unit. Beyond dealing with the needs of such multiple clientele, it has been the policy of the Evaluation Unit to share information with the community of professional educators. Studies published by the Evaluation Unit became incorporated in the syllabuses of university courses and in teacher training study materials, and they were also frequently published in international journals. The professional visibility had two important consequences. One, it increased the reputation of the Evaluation Unit, and two, it obliged decision makers to pay attention to the available evaluative information. The Evaluation Unit successfully avoided the pitfall of being too theoretical, which could be seen as non-relevant to the practioner or alternatively, of dealing with pragmatic issues in a way which did not lead towards generalizations.
Sharin~ Responsibilit~ The Evaluation Team shared responsibility with its clients in specifying the parameters of evaluation studies, developing instruments, interpreting the findings, and formulating recommendations for action (Lewy 1981). One may say that the practice of evaluation at the ICC reflected the principles of self-evaluation (Duby 1985). The involvement of the development or implementation team was very intensive, and most studies resulted in published papers written in collaboration by persons from both the Evaluation and the Subject Teams. Such cooperation enhanced the validity and credibility of the study results, and facilitated their utilization.
Keeping a Low Profile The Evaluation Unit has not striven to grow or to elevate its hierarchical status within the Centre. It operated as a service unit and was aware of the commonly shared view that the production units should fulfil the dominant roles within the institution. Instead of increasing its own staff, the Evaluation Team continuously strove to increase the Subject Teams' share in carrying out the evaluation studies relevant to their work.
Israel Curriculum Centre
273
Meeting the Test of Timeliness Contemporary evaluation literature mentions timeliness as one of the most important criteria for judging the adequacy of an evaluation study. Stufflebeam (1969) defined timeliness as the availability of the information at the time it is needed for the decision maker. The Evaluation Unit puts great emphasis on meeting this requirement. The time schedule of each particular study was always negotiated between the client and the evaluator, and in most cases deadlines were met. Not only did this enhance the utilization of evaluation results, but it also developed trust in the promises of the Evaluation Unit. The clients knew what kind of information they would receive and when.
Indirect Indicators of Impact As indicated above, no studies have been carried out to assess the impact of evaluation in the ICC. An indirect indicator of impact may be the simple fact of the continuous operation of the Evaluation Unit over twenty years. It was mentioned previously that the overall impact of the ICC was examined by employing three criteria: survival of the organization, its ability to generate new goals, and growth in resources. Employing these three criteria for examining the success of the Evaluation Unit and its impact, one may say that it met the requirements of the first two criteria. It managed to survive. It had operated during the whole life span of the ICC. Its survival was characterized by periodic changes in its working patterns. It generated new goals and it adjusted itself to the changing goal priorities of the ICC. The Evaluation Unit, however failed to meet the requirements of the third criterion, it has not increased its resources. The growth of its budget and of its staffing has not matched that of the whole institution. Paradoxically, its failure to grow, in this particular case, has contributed to its ability to survive. Nobody questioned the legitmacy of operating a service unit, which gradually reduced the proportion of its scope, within the framework of the whole institution, and could still maintain the level of its visibility.
References AKIN, M.C. (1974). Evaluation and Decision Making: The Title VII Experience. Los Angeles, University of California, Center for Study of Education. AKIN, M.C. (1979). Using Evaluation: Does Evaluation Make a Difference? Beverly Hills, Sage. DORR, E. (1985). Annual Report about Sales of Curriculum Materials. Jerusalem, Ministry of Education and Culture, Israel Curriculum Center. (Hebrew, unpublished manuscript). DUBY, A. (1985). Self Formulative Evaluation of Instructional Materials. Dissertation submitted to University of South Africa. EDEN, S, (1985). Curriculum Development Centers, in T. Husen and T.N. Postlethwaite (eds.) International Encyclopedia of Education. Oxford, Pergamon Press. ELBAZ, F., RAZ, R., and PADUA, M. (1987). School Level Curriculum Implementation (Hebrew). HAVELOCK, R.G. (1969). Planning for Innovation Through Dissemination and Utilization of Knowledge. Ann Arbor, University of Michigan, Center for Research on Utilization of Scientific Knowledge.
274
AoLewy
Israel, State of, 1985. The State Comptroller's Annual Report. Jerusalem, Government Printer. ISRAELI, R. (1983). Developing a Model for Measuring Organisational Effectiveness in an Institution of Higher Education. Doctor of Science Thesis submitted to Technion - Israel Institute of Technology. JACOBS, G. (ed.) (1979). The Participant Observer. New York, Braziller LEWY, A. (1980). New Curricula: Their Diffusion in Israeli Schools (Hebrew). Jerusalem, Ministry of Education and Culture, Israel Curriculum Center. LEWY, A. (1981). Sharing Responsibilities for Evaluation in A. Lewy et al. Decision Oriented Evaluation: The CAse of Israel. Philadelphia, International Science Services. LEWY, A. (1984). Survival and Revival of Organisations. Paper presented at the Kiriat Anavim Conference of the Israel Curriculum Center (Hebrew). LINDBLOM, C.E. and COHEN, D. (1979). Usable Knowledge: Social Sciences and Social Problem Solving. New Haven, Yale University Press. ROSSI, P.H. and FREEMAN, E.H. Evaluation: A Systematic A~proach. Beverly Hills, Sage. SHAPIRO, J.Z. (1984). Conceptualizing Evaluation Use in Ross F. Conner (ed.) Evaluation Studies Review Annual, vol. 9, Beverly Hills, Sage. STUFFLEBEAM, D. (1969). Evaluation as Enlightment for Decision Making in W.H. Beatty (ed.) Improvin~ Educational Assessment. Washington D.C., Association for Supervision and Curriculum Development. ZETTERBERG, H. (1965). On Theory and Verification in Sociology. Tottowa, N.J., Bedminster
The Author ARIEH LEWY is a professor at the School of Education of the University of Tel Aviv, and has been involved in many ways in the work of ICC and especially in evaluation.
0191-491X/87 $0.00 + .50 Copyright O 1987 Pergamon Journals Ltd.
Printed in Great Britain. All rights reserved.
EVALUATION OF IMPACT IN ISRAEL: ISRAEL CURRICULUM CENTRE Arieh Lewy School of Education, University of Tel Aviv, Tel Aviv, Israel
The Israel Curriculum Centre (ICC) started its professional activities in 1966. Seldom has an institute had such favourable conditions at its birth as did the Israel Curriculum Centre. Seldom could an institute begin its work after such meticulous preparation. By the time of the Centre's establishment the State of Israel had suceeded in solving one of its most vexing problems. It had put all schools under a roof, and had provided physical facilities such as classrooms, furniture, etc. for all pupils despite the unprecedented growth in the school population. Instructional materials, such as textbooks and syllabuses for all subjects taught in schools became available for teachers and learners. At the beginning of the 1960's the Minister of Education could say that no child in Israel was deprived of attending a school operating under decent conditions. The achievement enabled the authorities to orient their efforts towards the improvement of educational practice, and to focus on the content of education rather than on physical facilities. It was the time of great messages. The re-discovery of the Tyler rationale, the publication of the Bruner and Gagne books. Schwab's conceptualization of curriculum, programmes known by the acronyms, BSCS, PSSC, etc. created enthusiasm among curriculum experts, and Nuffield ceased to be only a geographical term. Educational experts believed in the power of rational planning and in the merits of 'teacherless' and 'errorless' programmes. This atmosphere Curriculum Centre.
no doubt
influenced
the Ministry's
decision
to establish
a
When the Curriculum Centre was established, Israel had a highly centralized educational system. It was taken for granted that curriculum development, too, was to be carried out by the central agencies of the Ministry and that the products of these developmental activities would be desseminated throughout the system. Accordingly, the Israel Curriculum Centre began working as a national research and development institute. It adopted the principles of the R-D-D (Research, Development and Diffusion) approach (Havelock 1969), and operated according to the Central-Periphery line of production (Eden, 1985).
263
264
A. Lewy
Since then, changes have occurred in the dominant pedagogical conceptions of the country. Greater autonomy was bestowed upon the six regional offices of the Ministry of Education and Culture, and autonomy of schools themselves was increased considerably. Parallel to these changes the Curriculum Centre increased the scope of its activities, and in addition to the R-D-D production pattern it became involved in joint collaborative endeavours with the regional offices of the Ministry. The ICC has encouraged and monitored local and school-based development. It invited other institutes and organizations to contribute to the national effort of curriculum development. University departments, teacher training colleges, museums, research institutes, health institutions, and voluntary organizations responded to the invitation and many became involved in curriculum development activities. In 1986 the ICC no longer had a 'monopoly' as a curriculum development institute in Israel. It became one of several institutes doing such work. Under these conditions it became necessary to establish a national co-ordinating body for curriculum development. Due to historical circumstances, such as the seniority of the Institute, its affiliation to the Ministry of Education and Culture, the academic qualifications of its staff members, and so on, its suitability as a co-ordinating body was not seriously contested. In 1986 the ICC has approximately forty five full-time job-equivalent professional positions, which are occupied by about 120 persons. Only a few of them work at the Centre in a full-time capacity. The rest have a part-time position at the ICC, and complement their work by teaching in schools, teacher training institutions, universities, or by acting as school supervisors. The ICC also monitors the work of curriculum development activities at various institutions outside the ICC. These bodies employ a comparable number of professional curriculum specialists. Its annual budget including staff salary and funding external projects, equivalent to three million dollars.
is
The Modus Operandi of the Israel Curriculum Centre From the very beginning the major operational units within the ICC have been the Subject Teams. A Subject Team had to take care of all professional activities which were related to the school curriculum in one subject matter, such as mother tongue, mathematics, geography, science, history, music, etc. Originally the sole responsibility of a Subject Team was to produce new p r o g r a m e kits (including textbooks, teacher's guide, instructional aids, etc.) in a single subject. Thus, for example, the Science Team prepared the above mentioned items for all three grades of the comprehensive school. They also designed laboratory equipment, kits containing materials for experiments, etc. Gradually, the work of the Subject Team became more and more diversified. First, the need to diffuse curricular products in the whole system imposed upon the teams the task of contacting supervisiors, school principals, teachers, teacher training institutes, etc. They were requested to organize curriculum oriented in-service training activities. Later, they had to assume responsibility for monitoring school-based curriculum development activities, preparing computerised learning modules,
Israel Curriculum Centre developing monitoring
a 'second generation set' of curriculum materials, and the curriculum development activities of other institutes.
above
265
all,
In addition to the Subject Teams which operated as production units, several service units were established in the Centre such as the Graphics and Printing Unit, the Library and the Evaluation Unit. As a result of the diversification in the team's work several new non-subject related support units were created within the ICC; an Implementation Unit, a School-Based Curriculum Development Unit, and a Computerised Instructional Materials Unit which provided support for various Subject Teams. The new units came to assist the Subject Teams in their work rather than to restrict the freedom of their operation. Of course, the mere existence of an Evaluation Unit or an implementation Unit reflected a broad consensus within the ICC that these functions should be attended to, and no Subject Team could fully disregard them.
Current Evaluation Activities The empirical evaluation studies grouped into two major classes:
carried
out by the Evaluation
unit
can be
(i)
Small scale and short duration (approximately one to six months) formative evaluation studies related to material produced by the Teams or difused by them.
focused Subject
(2)
Large scale studies which touched upon pervasive issues of the whole educational system, and cut across limits of particular subject areas. Thus in such studies several Subject Teams were often involved, for example: identifying Educational Needs, Policy Research, Implementation Evaluation, Criteria for Evaluating Computerised instructional Units etc.
An Overall Evaluation of ICC Activities Evaluation has constituted an integral part of the ICC since its establishment. Originally it concerned itself with evaluating the products prepared by the Subject Teams and the activities initiated or carried out by their members. Studies of this type improved the quality of the evaluation targets, demonstrated the merits of various programmes, and persuaded the prospective clientele to use the products. Beyond ensuring and demonstrating the quality of a particular product or the efficiency of a particular action, however, the management of the ICC was challenged to answer more pervasive questions about its operations. Questions were asked about the capability of the ICC to carry out all tasks delegated to it in due time. Could the ICC effectively reach the potential clientele of its products in various sectors of the system? Was it desirable to maintain a national centre for curriculum development? Opinions were expressed that the existence of a national diminished private initative in curriculum development.
curriculum
centre
To answer these questions in the 1980's, the Evaluation Unit of ICC became involved in several studies which examined ICC activities in a holistic way and tried to provide a global perspective of its operation. It would be impossible
266
A. Lewy
to present details about all studies which touched upon the global operation of the ICC, and therefore we will limit our report to illustrative examples of three types: empirical studies, analytical pattern of institutional self-evaluation, and an external auditing study paradigm.
Focusing on the Whole Institute Empirical
Studies
Two empirical studies ethnographic study.
will
be
reported
here:
a statistical
survey
and
an
Survey of utilization. The Graphics and Printing units regularly provide annual statistics about the sale of products prepared by the Subject Teams (Dorr 1985). While the figures of these reports provide information needed about the diffusion of the products, it was felt that more information was needed about the diffusion and utilization of curricular materials. The sales figures do not provide sufficient information about the extent of use. A national survey based on a sample of about I00 schools examined the degree of utilization of various curricular products prepared by the ICC and by other curriculum development institutes in the country. The teachers' perceptions of these products, the teachers' attitudes towards them and difficulties encountered in using these materials were examined (Lewy 1980). The study provided complementary information to the statistics about the annual sale of the products. The survey revealed that only 5% of users did not like About 75% expressed high satisfaction, and the rest satisfaction with the programme. The new programmes were the schools. About 60% of the schools indicated that they instructional materials in more than five subject areas.
the new programmes. expressed moderate highly 'visible' in used the ICC's new
Some programmes were extremely popular and became operational in more than 90% of the schools. Nevertheless, the respondents complained that the new curriculum materials could not readily be used in classes of low ability students or in mixed ability classes. An ethnosraphic
study of four schools.
To broaden the information obtained from the Utilization Survey described above, the Evaluation Unit initiated another impact study which was structured according to the tradition of ethnographic studies in education (Jacobs 1979). It was decided to send an observer to four different schools. The observers were permanent staff members of the Evaluation Unit. They spent three or four days in a particular school, interviewed the school principal and teachers, and observed modes of utilizing curricular materials. The purpose of this study was to examine to what extent the instructional products of the ICC were used in schools, the reactions of teachers and learners to these materials, the teachers expectations with regard to curriculum development institutes, and the extent to which these expectations were met. Some of the findings are described below:
Israel Curriculum Centre
267
Well-structured textbooks describing what to teach and how are more popular among teachers, than those which impose the burden of structuring the programme upon the teacher. The teachers act skillfully in supplementing the existing programme with materials of local character, but they prefer to restrict their autonomy to the selection and supplementation of curricular contents, rather than structuring or organizing. Teachers believe that materials prepared by professional teams are better than those which they are able to produce. The same attitude is felt with regard to the in-service activities. Teachers, and especially principals and supervisors attributed greater importance to the structured courses which cover a specific body of knowledge, than to ad hoc discussions of the problem-solving type (Ebaz et al 1987). These findings may present a gloomy picture for those who strongly advocate school-based curriculum development. Nevertheless, in assessing the impact of the ICC it would be a mistake to interpret these findings as a move towards restricting teacher initiative in the field of curriculum development. One has to view these facts in an historical context. Before the establishment of the ICC the textbooks used in the system were heavily loaded with facts, and teaching focused on cramming. Teachers followed the prescribed syllabus and had little freedom to deviate from what was contained in the textbook. In contrast, the textbooks developed at the ICC invited the teachers to exercise at least a restricted level of autonomy related to some aspects of the curriculum. Teachers responded positively to this challenge, and the discourse among teachers about matters of curriculum changed. Teachers developed a critical approach to textbooks, learned to select and to supplement textbooks, and became familiar with a broad repertoire of criteria for judging the quality of instructional materials. Those who would like to see a shift toward a larger scope of school-based curriculum development in Israeli schools are certainly not fully satisfied with what happened. Nevertheless some supporters of the school-based curriculum movement conceive the impact of the ICC described above as an educative step that may have positive consequences for increasing school-based curriculum development activities in the country. It can be assumed that the establishment of a School-Based Curriculum support unit in the ICC will precipitate such development, and will present to the teacher the alternatives of either selecting structured and semi-structured materials developed by professional curriculum developers, or of developing their own sets of instructional materials.
Analytical
Pattern of Institutional
Self-Evaluation
The ICC regularly runs annual planning workshops. The 1984 workshops focused on assessing successes and failures of past activities. The discussion was based on position papers and analytical summaries of past experiences. Subcommittees tried to review the scholarly merits and the pragmatic consequences of the documents and oral presentations at the Workshop. For this reason one may conceive such an event as an attempt of institutional self-evaluation. The discussion of the Workshop examined the attainments of the Centre according to three criteria suggested by Israeli (1983) to assess the efficiency of organizations: goal attainment, survival of the organization, and acquisition of resources.
268
A. Lewy
In applying these three criteria to the activities of the ICC over the past two decades, one may assign to it a very high rating of success. The changes which occured in the work of the ICC in response to emerging needs were surveyed. It was found that the Centre continuously generated new goals and thus provided justification for its survival; it diversified the areas of its activities, and it increased the scope of institutes and persons with whom it worked and whose work it monitored (Lewy 1984). Analyzing
the problem of the ICC it was stated:
The changes generated problems and tensions which have not been successfully handed. Their solution or reduction is crucial to the continued functioning of the Centre. Firstly, not only have the innovations introduced by the Centre been successfully disseminated in the educational system, but the "know how" of producing such innovative programmes also became a public asset. In other words we have been very good teachers and taught others all our "professional secrets". Unlike industrial companies, we could not and did not want to keep our secrets. Nowadays, we competitors. They contribute to the good, but what is (or do not want to
are not the only tradesmen in the market. There are are our students, and we are proud of them. Their activities improvement of the quality of curricula in schools. This is not good is that many members of the Centre are not cognizant be cognizant) of these facts.
The second problem relates to growth. We grew by splintering, which meant that a Subject Team could obtain additional staff members on the condition that they agreed to assume responsibility for additional tasks. For example a Subject Team could increase the scope of its work, by starting to prepare computerized instructional materials. To carry out this new task, the team split into two, and consequently the size of the original teams became smaller. Parallel with broadening the scope of the centre's responsibility and increasing the size of the staff, under-staffing became a growing concern. Staff enlargement has never matched the enlargement of the scope of responsibilities. As a result of the process of splintering, the number of "one-person teams" or even part-time teams is gradually increasing in the Centre. This has resulted in decreased contact between people working in the Centre. Some people feel lonely inside the Centre and yearn for those old days when larger teams worked together on narrowly defined tasks. Finally, assuming responsibility for new goals and new tasks required the Centre to appoint staff ~embers with different specializations. This required a shift in criteria to evaluate the work of staff members. At the initial stage clear conceptualization and writing ability were the most needed skills for doing good work. Staff members had an academic orientation. The majority held part-time positions at universities and at other teaching institutes and had acquired the habit of publishing papers in scientific journals. The working ethos was akin to that which prevails in academic circles, and the approval granted by the academic community to work done in the Centre was extremely important to staff members. Many tasks of the Centre require interpersonal rather than academic skills, as quite often the skill of speaking is more relevant than the skill of writing. These shifts went together with changes in the relationship between the academic community and staff members of the Centre. Twenty years ago the
Israel Curriculum Centre
269
experts of the Centre were leaders in producing curricular knowledge of academic value. Today the Centre is gradually becoming a place where staff members use academic knowledge produced elsewhere. The Staff Workshop served as a self-evaluation mechanism. Decisions were made to create large working units through combining serveral Subject Teams into one, to establish closer co-operation with various branches of the Ministry of Education and Culture, and to give priority to monitoring the work of all curriculum development institutions in the country, even if it necessiated reducing the scope of direct curriculum development activities of the Subject Teams. These decisions were incorporated into the official policy of the Centre, and at present continuous effort is being made to implement them. Nevertheless, their realization may require much more time than was originally contemplated.
The State Comptroller's
Report
Since the ICC is a branch of the Ministry of Education and Culture, it is subject to the control of the State Comptroller's office. The office examines the operations of all institutes funder by the State, and it publishes an annual report specifying details of any malfunctioning it may uncover. The State Comptroller's office enjoys a high reputation in the ranks of the state bureaucracy. Its director has a status comparable with that of a High Court judge, though the office itself is not a part of the judiciary system. The publication of the annual report is considered an event of national importance and the daily newspapers give it extensive coverage. Institutes that are criticized are expected to react to the criticism and the forthcoming annual reports of the State Comptroller's office usually acknowledge these reactions. While the previous reports contained some comments about the ICC, a team of the State Comptroller's office thoroughly examined the operational routines of the Centre in 1984. The team spent about two weeks at the Centre, reviewed a variety of documents, publication products, etc. interviewed staff members and the summary of their observations was published in the 1984 annual report (Israel 1985). By the very nature of such reports, the focus was on flaws detected in the operation of the Centre. Those aspects of the ICC's work which were found to be satisfactory were not mentioned. Some of the issues dealt with in the report are reported here; the report touched on issues of priority, planning, organizational structure, contact with schools, and co-ordination of work with other branches of the Ministry of Education and Culture.
Priority. The formal goal statements following goal priorities:
issued
i.
Publishing syllabuses for revising or up-dating them.
all
2.
Developing instructional materials contained in the syllabuses.
3.
The diffusion and and textbooks).
implementation
by
study
of
the
areas
ICC
in
1984
(subjects)
which
correspond
the
instructional
to
specified
and
the
the
continuously
specifications
products
(syllabuses
270
A. Lewy
4.
Guiding organizations, institutes and schools materials as specified in the syllabuses.
The report by all Subject distribution of differed widely effort, without
to
develop
instructional
indicated that the order of priority was not strictly observed Teams. Moreover, no sufficient guidelines were given to the efforts across these four different goals. The Subject Teams from each other with respect to the distribution of their providing justification for their idiosyncratic decisions.
Plannin$. The report complained that planning activities of the Centre were deficient. Subject Teams had to prepare an annual plan of work which was submitted for approval to the directorial board of the Centre. The duration of the development work mostly exceeded the limits of a single year, however, and therefore a long-term work plan encompassing activities for two to three years would be needed.
Organizational
structure.
At the time the survey was carried out thirty Subject Teams operated in the Centre. All of them reported directly to the director of the institute. Direct contact with thirty production teams imposed an insurmountable task upon the director. Since Subject Teams represent a great variety of specialisations in science, art and humanities, a single person cannot be adequately qualified to review all these products. Another organizational problem, according to the report, was the confusion between line and staff operational modes within the Centre. The Centre itself assumed responsibility for preparing instructional materials and at the same time also served as a facilitator for other institutes in developing instructional products. There is a contradiction between these activities, since the ICC had to monitor the development of products which could compete with products developed by the ICC itself.
Contact with schools. The report pointed out that in a substantial proportion of schools teachers had never seen the official syllabuses, and that the single tool they used for planning their school programme was the textbook. The situation was deplorable, certainly. The State Comptroller's report blamed the ICC for not ensuring that all teachers could have access to the syllabuses, and thus be able to familiarize themselves with their content.
Co-ordination with other departments
and institutes.
The report stated that the activities of the ICC and other Ministry of Education departments were not sufficiently co-ordinated. As an example, they referred to the discrepancy between the goals set by in-service teacher training activities in the ICC, the Department of In-service teacher training and the teacher training colleges.
Israel Curriculum Centre
271
The Impact of Impact Studies The studies reported in this article examined the impact of the ICC and its products in the educational system. The three types of impact studies represent what Rossi and Freeman (1982) called methodological heterogeneity. One may add that it also represents heterogeneity of study aims. At least three areas of aims were dealt with, or in other words three kinds of questions were asked: Did the institute or a particular product of the institute have an impact? Which characteristics of an entity of of an action may precipitate impact? Is the impact congruent with the goal priority of a certain client? By no means do these questions constitute a full list of aims for impact studies, but they suggest that it might be useful to construct a taxonomy of relevant questions. The legitimacy of methodological heterogeneity brought up the question whether different methods and research paradigms are equally effective. Is it possible to identify certain circumstances and/or specific study aims, where one method produces better results than another one? In an attempt to answer this question hard-line empiricists collected and summarized data to find out whether evaluation makes a difference (Alkin 1974, 1979). Others approached this problem in an axiomatic way (Zetterberg 1965), or in a reflective, analytical way (Lindblom and Cohen 1979). Alkin developed a mapping sentence to distinguish between types of evaluation utilization (such as dominant influence, one of multiple influences, cumulative influence, etc.) and also specified conditions which may contribute to the impact of evaluation studies. At the ICC no studies were carried out to examine the impact of evaluation activities. Therefore, each statement made about the impact of evaluation studies in the ICC has only tentative validity. Nevertheless, it can be said that, within the framework of the operational procedures, great importance has always been given to the utilization of evaluation results. Subject Teams systematically used formative evaluation results to improve the quality of their products, and the pattern of the programme-diffusion activities. Little experience has been accumulated in using evaluation results which touch on the operation of the whole institution. The reorganization of the ICC is at present in process, and sincere efforts are being made to use evaluation results in the process of decision-making. There are signs, however, that it is more difficult to implement recommendations of studies which deal with a whole institution than recommendations of formative evaluation. Cooperative links between the Evaluation Team and each Subject Team pave the way for the utilization of formative evaluation findings. In contrast, the operational pattern of an entire institution represents a delicate compromise between conflicting interests of institutional branches, and of competing institutions. Under such circumstances political and power considerations gain great influence in the process of decision making and they may weaken the impact of evaluation findings. It is unlikely that conflicting parties will be overwhelmed by altruistic motives, and will disregard their particularistic interests in favour of the overall institutional interests (Shapiro 1984). In an institutional setting, impact is a function of adequate con~munication, persuasion, and above all inter-organizational and intra-organizational politics. It is not enough to be right, one has to be able to convince others and to weaken resistance.
272
A. Lewy
During the twenty years of its operation the Evaluation Unit has succeeded in creating a favourable climate towards evaluation. Since this is a crucial determinant of evaluation utilization, it seems appropriate to describe strategies employed by the Evaluation Unit for creating such a climate. These strategies proved to be effective in enhancing the utilization of formative evaluation results, and one may assume that they will contribute also to the utilization of impact evaluation results.
Professional Visibility The Evaluation Unit, from the very beginning of its operation, made an attempt to maintain visibility for its work in professional circles. Priority was assigned to the information needs of direct cliencts, but the clientele was broadly defined. The Subject Teams, therefore, were never thought of as the sole clients. The management of ICC, other branches of the Ministry of Education and Culture, teachers, as well as the mass media, were all considered clients of the Evaluation Unit. Beyond dealing with the needs of such multiple clientele, it has been the policy of the Evaluation Unit to share information with the community of professional educators. Studies published by the Evaluation Unit became incorporated in the syllabuses of university courses and in teacher training study materials, and they were also frequently published in international journals. The professional visibility had two important consequences. One, it increased the reputation of the Evaluation Unit, and two, it obliged decision makers to pay attention to the available evaluative information. The Evaluation Unit successfully avoided the pitfall of being too theoretical, which could be seen as non-relevant to the practioner or alternatively, of dealing with pragmatic issues in a way which did not lead towards generalizations.
Sharin~ Responsibilit~ The Evaluation Team shared responsibility with its clients in specifying the parameters of evaluation studies, developing instruments, interpreting the findings, and formulating recommendations for action (Lewy 1981). One may say that the practice of evaluation at the ICC reflected the principles of self-evaluation (Duby 1985). The involvement of the development or implementation team was very intensive, and most studies resulted in published papers written in collaboration by persons from both the Evaluation and the Subject Teams. Such cooperation enhanced the validity and credibility of the study results, and facilitated their utilization.
Keeping a Low Profile The Evaluation Unit has not striven to grow or to elevate its hierarchical status within the Centre. It operated as a service unit and was aware of the commonly shared view that the production units should fulfil the dominant roles within the institution. Instead of increasing its own staff, the Evaluation Team continuously strove to increase the Subject Teams' share in carrying out the evaluation studies relevant to their work.
Israel Curriculum Centre
273
Meeting the Test of Timeliness Contemporary evaluation literature mentions timeliness as one of the most important criteria for judging the adequacy of an evaluation study. Stufflebeam (1969) defined timeliness as the availability of the information at the time it is needed for the decision maker. The Evaluation Unit puts great emphasis on meeting this requirement. The time schedule of each particular study was always negotiated between the client and the evaluator, and in most cases deadlines were met. Not only did this enhance the utilization of evaluation results, but it also developed trust in the promises of the Evaluation Unit. The clients knew what kind of information they would receive and when.
Indirect Indicators of Impact As indicated above, no studies have been carried out to assess the impact of evaluation in the ICC. An indirect indicator of impact may be the simple fact of the continuous operation of the Evaluation Unit over twenty years. It was mentioned previously that the overall impact of the ICC was examined by employing three criteria: survival of the organization, its ability to generate new goals, and growth in resources. Employing these three criteria for examining the success of the Evaluation Unit and its impact, one may say that it met the requirements of the first two criteria. It managed to survive. It had operated during the whole life span of the ICC. Its survival was characterized by periodic changes in its working patterns. It generated new goals and it adjusted itself to the changing goal priorities of the ICC. The Evaluation Unit, however failed to meet the requirements of the third criterion, it has not increased its resources. The growth of its budget and of its staffing has not matched that of the whole institution. Paradoxically, its failure to grow, in this particular case, has contributed to its ability to survive. Nobody questioned the legitmacy of operating a service unit, which gradually reduced the proportion of its scope, within the framework of the whole institution, and could still maintain the level of its visibility.
References AKIN, M.C. (1974). Evaluation and Decision Making: The Title VII Experience. Los Angeles, University of California, Center for Study of Education. AKIN, M.C. (1979). Using Evaluation: Does Evaluation Make a Difference? Beverly Hills, Sage. DORR, E. (1985). Annual Report about Sales of Curriculum Materials. Jerusalem, Ministry of Education and Culture, Israel Curriculum Center. (Hebrew, unpublished manuscript). DUBY, A. (1985). Self Formulative Evaluation of Instructional Materials. Dissertation submitted to University of South Africa. EDEN, S, (1985). Curriculum Development Centers, in T. Husen and T.N. Postlethwaite (eds.) International Encyclopedia of Education. Oxford, Pergamon Press. ELBAZ, F., RAZ, R., and PADUA, M. (1987). School Level Curriculum Implementation (Hebrew). HAVELOCK, R.G. (1969). Planning for Innovation Through Dissemination and Utilization of Knowledge. Ann Arbor, University of Michigan, Center for Research on Utilization of Scientific Knowledge.
274
AoLewy
Israel, State of, 1985. The State Comptroller's Annual Report. Jerusalem, Government Printer. ISRAELI, R. (1983). Developing a Model for Measuring Organisational Effectiveness in an Institution of Higher Education. Doctor of Science Thesis submitted to Technion - Israel Institute of Technology. JACOBS, G. (ed.) (1979). The Participant Observer. New York, Braziller LEWY, A. (1980). New Curricula: Their Diffusion in Israeli Schools (Hebrew). Jerusalem, Ministry of Education and Culture, Israel Curriculum Center. LEWY, A. (1981). Sharing Responsibilities for Evaluation in A. Lewy et al. Decision Oriented Evaluation: The CAse of Israel. Philadelphia, International Science Services. LEWY, A. (1984). Survival and Revival of Organisations. Paper presented at the Kiriat Anavim Conference of the Israel Curriculum Center (Hebrew). LINDBLOM, C.E. and COHEN, D. (1979). Usable Knowledge: Social Sciences and Social Problem Solving. New Haven, Yale University Press. ROSSI, P.H. and FREEMAN, E.H. Evaluation: A Systematic A~proach. Beverly Hills, Sage. SHAPIRO, J.Z. (1984). Conceptualizing Evaluation Use in Ross F. Conner (ed.) Evaluation Studies Review Annual, vol. 9, Beverly Hills, Sage. STUFFLEBEAM, D. (1969). Evaluation as Enlightment for Decision Making in W.H. Beatty (ed.) Improvin~ Educational Assessment. Washington D.C., Association for Supervision and Curriculum Development. ZETTERBERG, H. (1965). On Theory and Verification in Sociology. Tottowa, N.J., Bedminster
The Author ARIEH LEWY is a professor at the School of Education of the University of Tel Aviv, and has been involved in many ways in the work of ICC and especially in evaluation.