Student assessment within the context of constructivist educational settings

Studies in Educational Evaluation Studies in Educational Evaluation 28 (2002) 369-390 www.elsevier.com/stueduc

STUDENT ASSESSMENT WITHIN THE CONTEXT OF CONSTRUCTIVIST EDUCATIONAL SETTINGS

Marianne Elshout-Mohr,*

Ron Oostdam** and Marianne OvermaaP

*University of Amsterdam, faculty of Social and Behavioural Sciences, SCO-Kohnstamm institute Graduate School of Teaching and Learning, The Netherlands **University of Amsterdam, Faculty of Social and Behavioural Sciences, SCO-Kohnstamm Institute, The Netherlands

Introduction The concept of constructivism was introduced in the 1980s to improve students’ learning processes (Simons. Van der Linden, & Duffji, 2000). Within this framework selfregulation is seen as a variable of great importance (Shunk & Zimmerman, 1994; Wang, Haertel. & Walberg, 1990). Closely related to constructivism is the notion that powerful learning environments are required to challenge and stimulate students to act as active and self-directed learners (Brown, Collins, & Duguid, 1989; De Corte, 1990). A main characteristic of these so-called powerful learning environments is that students acquire new knowledge and skills in the context of authentic complex task situations. The general idea is that students are more likely to become active and self-directed learners when they can find out for themselves which competencies contribute to the achievement of meaningful goals and which knowledge, skills and attitudes enable participants in meaningful projects to act effectively, expertly and professionally. The better students know from their own experience which learning goals are worthwhile, the more they will be inclined to learn in an active and self-directed manner. A second characteristic of powerful learning environments is that the students get help to realize their learning endeavours. While self-directed learners are scarcely in need of teacherdirected standardized courses, they are in need of teachers who assist them in the realization and regulation of the learning processes. Teachers’ role is to provide coaching and support, to offer brief courses and training sessions, and to refer students to relevant sources of information, such as books, articles and experts. Electronic communication and information technologies are often seen as a third cornerstone of powerful learning environments. Substantial and proficient application of these technologies by both students and teachers is needed to sustain students’ self-directed learning (Kanselaar, De Jong, Andriessen, & 0191-491X/02/$ - see front matter 0 2002 Published PII: SOlSl-491X(02)00044-5

by Elsevier

Science

Ltd

370

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002)

369-390

Goodyear, 2000). Increased interest in establishing powerful learning arrangements can be found at all educational levels (Boekaerts, 1997; Dufrj, & Jonassen, 1992). In the present study we focus on higher vocational education. In this type of education constructivism is often combined with competence orientedness (Birenbaum & Dochy. 1996). Competence orientedness implies that the focal aim of education is that students develop the ability to function effectively in authentic. real-life contexts and acquire the attitude to reflect on their own functioning. In organising a constructivist, competence-oriented learning curriculum, assessment procedures should be selected with great care (Dochy & McDowell, 1997). Students’ attitudes towards learning in an active, self-directed and self-regulated manner are strongly influenced by the assessment system (Hambleton 1996; Sambell, McDowell. & Brown. 1997; Van Hout-Wolters, 2000). The idea that a transition from an instructivist to a constructivist educational system should be accompanied by a transformation of the assessment system is embodied in the principle of alignment (Biggs, 1996). In the case of a constructivist, competence-oriented vocational training curriculum the alignment-principle implies two things. First, the assessment system should be individualized and studentcentred in order to be aligned to learners’ self-directedness. Second, the assessment system should make use of authentic situations for testing in order to be aligned to the learners’ competence orientedness. Although the need for congruence between learning arrangements and assessment methods is easily understood at a theoretical level, it has been found again and again that it is very difficult to bring the principle into practice (Driessen & Van der Vleuten, 2000). The aim of this article is to analyse some of the problems that occur and to suggest possible solutions. In the following sections we first describe an experimental curriculum that offers a good example of an attempt to organize students’ learning in a constructivist, competenceoriented manner. Second, the principle of alignment is elaborated by means of a presentation of three familiar educational configurations, in which optimal alignment is reached. Only one of these configurations appears to be adequate in the context of a constructivist. competence-oriented curriculum. Third, we turn to an evaluation study on the assessment procedures used in the experimental curriculum. Main findings were that a high degree of individualization was reached, but respondents had serious concerns about the lack of standardization of the assessment criteria and about the meagre check on students’ available expert knowledge. Finally, we present some ideas about how to meet these concerns without abandoning the framework of the student- and competence-oriented approach. A distinguishing feature of this approach is that students are given a substantial role in defining, applying and maintaining standards. An Example

of a Constructivist

Competence-Oriented

Curriculum

An example of a constructivist competence-oriented curriculum is the Amsterdam Faculty of Education (EFA). The Dutch minister of Education awarded the Faculty the status of experimental teacher training program in October 1997. The curriculum of the EFA is explicitly inspired by the constructivist movement. Main components of the training sessions and metawork. These curriculum are learning practices, sources, components have been explained in full in publications by EFA faculty members (Snoek &

M. Elshout-Mohr et al. /Studies

in Educational Evaluation 28 (2002) 369-390

371

Wielenga, 1999; Dietze. Snoek, & Wielenga, 2000; for further reference, cf. www.efa.nl). In the present article a brief description should suffice to clarify how competence orientedness and self-directedness are interwoven in this curriculum. Learning practices are learning environments in which student-teachers have to deliver authentic products in collaboration with others, such as teachers, students and community workers. For instance. in one learning practice labelled “Ideologies and religion” . students are invited to collaborate with elementary schools in an endeavour to deliver a product that the school can subsequently use to stimulate pupils’ learning about ideologies and religion. This product might be a computer program that provides information and stimulates pupils to explore different ideologies and religions. It might also be a description of a procedure that pupils might use to interview each other about ideologies and religion or any other product that the learning practice team would consider useful and meaningful for the school and the pupils. Ideally. each learning practice combines two features. First the situation should be representative for the work that student-teachers are being prepared for, i.e., for their future profession. The abovementioned learning practice is adequate in this respect because competence in constructing lesson materials is a component of the teaching profession. Second, the situation must provide rich opportunities for the students to learn. It must enable them to find out which knowledge. skills and attitudes are needed to be competent in constructing interesting lesson materials. Whether the above-mentioned learning practice is adequate in this respect depends to some extent on the composition of the team, the project plan, and the manner in which cooperation and collaboration between team members is organized. By being authentic however the learning practice is probably complex enough to provide relevant information about what it takes to construct lesson materials. Once students have gained first hand information about relevant learning goals, they can start to direct their own learning. Partially, this learning can take place within the actual context of the learning practice, but students are also expected to enrich their learning and to speed it up by adequate use of the facilities, such as sources and training sessions. Steered by the gaps, which they discover in their knowledge and skills in the course of learning practices, students should decide for themselves which sources and training sessions are worthwhile for them to attend and which teachers they should consult. The sources and training sessions are thus conceptualised as means to develop competencies that are relevant for the work in the learning practice and for the future job. In order to help students to organize their own learning and to keep pace, they have to perform metawork. This term refers to a collection of instruments, such as (obligatory) tutoring sessions and a digital portfolio system. Metawork is developed to enable the faculty to keep track of the students’ activities and to enable teachers and tutors to coach students in making the necessary decisions. The main purpose of metawork is, however to foster reflection, which is a cornerstone of self-directed learning (Korthagen, Klaassen, & Russell, 2000). Metawork is deemed highly important. because student teachers who have learned to reflect and to take responsibility for their own learning process will probably become life-longlearners. Moreover, it is assumed that student teachers who have experienced the profits of self-directed learning during their own training will transfer this experience to future pupils. The EFA is constituted by many divisions, such as the division for teachers in French as a foreign language in secondary education, and the division for teachers in

372

M. Elshout-Mohr

et nl. /Studies

in Educntionnl

Evnlucrtion

28 (2002)

369-390

elementary education. In some divisions the development of the new curriculum is at an advanced stage. while in others it is at an early state of construction. The former divisions have developed a sufficient number of learning practices to f’ulfil a central role in the students’ learning. while this number is too restricted yet in the latter divisions. Substantial differences among divisions also exist in regard to the development of new assessment procedures. The general design of the planned assessment system is shared, however. In the new system. assessments are linked to phases of the curriculum. namely the introductor) phase (1 year), a main phase (2 years) and a teacher-assistant phase (I year). At the end 01 each phase an integrative assessment session takes place. which is decisive for the students’ admission to the next phase and for the formative feedback that the student receives. Assessment tools are portfolio presentations and critical situations. These tools and their alignment to the learning arrangements will be discussed belolv.

‘fhree Prototypical

Configurations

for Aligmnent

To be successful, new instructional arrangements must be combined with appropriate assessment procedures (Biggs. 1996; Entwistle. 2000; Lane & Glaser. 1996). In an innovative context it is common practice to speak about traditional versus novel assessment procedures (Lane & Glaser. 1996; O’Neil, 1997). The former are described as representatives of old learning theories, which arc not suited to modern educational demands, while the latter are described as representatives of new and more advanced learning theories. Hager ( 1994) distinguished three approaches to assessment. ranging from old to new. The first is called the behaviourisf approach. This approach is grounded in behaviourist psychology and addresses knowledge and skills that result from the performance of discrete instructional tasks. Hager characterized this approach as atomistic. rcductivc. mechanistic, standardized and routine. Evidently it is not suitable to capture the integrated quality of professional performance. The second approach is called the attribute or general skill approach. It is grounded in cognitive psychology and addresses personal attributes and general skills that result from practice on instructional tasks. Hager characterized this approach as less atomistic than the first but still remote from professional practice. When this approach is used to predict future professional performance. it capitalizes on the uncertain assumption that general skills transfer to future situations. The third approach is called the integrated task and attribute approach. It is grounded in modern learning theories. such as theories about situated learning, and it is underpinned by an understanding of purposeful action in complex real-life situations. Hager characterized this approach as holistic and potentially suitable to assess professional competence. Prerequisites are that the assessment is planned carefully and that portfolios are used to provide evidence of quality of performance over a considerable period of time. Three Prototypes We shall now elaborate on Hager’s three approaches. We take a somewhat different perspective, however, because we believe that it is premature to conclude that just one kind of assessment is suitable to assess professional competence. Table 1 shows three complex

M. Elshout-Mohr

et al. /Studies

in Educational Evaluation 28 (2002) 369-390

373

educational configurations, each of which unfolds into three components, namely learning goals, learning arrangements and assessment procedures. Within each configuration all components are congruent, i.e., stemming from one particular approach to learning and instruction. It is congruent for instance to assess factual, discrete knowledge by means of a knowledge test, and it is also congruent to transmit factual, discrete knowledge in a teacherdirected course and to test the learning results by means of a teacher-marked knowledge test (Configuration 1). Between configurations, components are assumed to be incongruent, i.e., stemming from different approaches to learning and instruction. For instance, it is neither congruent to assess professional competence (a component from Configuration 3) with a teacher-marked knowledge test (Configuration l), nor to organize complex learning arrangement such as described in Configuration 3 with the aim to assess discrete factual and conceptual knowledge (Configuration 1). Table 1:

Alignment

Configuration

of Learning Goals, Learning Arrangement Learning goals

and Assessment

Learning arrangement

Assessment

Teacher directed

Discrete factual and conceptual knowledge, of a declarative or procedural nature.

Teacher-directed courses and guided practice in laboratory settings.

Students do tests and take examinations. The teacher marks these.

Standard oriented

General skills

Strategy training; individual skill practice in a variety of circumstances.

Students demonstrate general skills in performances. These are assessed with fixed evaluation scales (standards).

Competence oriented

Professional competence

Active participation in professional work or simulations of such work; guided work experience; construction of personal working and learning concept.

Students present views and competencies in - portfolio presentations - critical situations

Assessors are also recruited from the professional field.

The configurations (see the first column of Table 1) are labelled teacher directed, standard oriented and competence oriented. These are well-known approaches to learning and instruction (Berliner & Calfee, 1996; Knoers, 1996). Very briefly, one could say that the teacher-directed approach capitalizes on the benefits of direct teaching on the part of teachers and guided practice on the part of students. The standard-oriented approach capitalizes on the benefits of students’ active learning under conditions in which they can decide for themselves how to proceed in order to reach previously-defined standards. Teachers are assumed to fulfil a coaching role. The competence-oriented approach capitalizes on active participation of students in professional environments and learning

374

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

2X (2002)

369-390

communities. Teachers are supposed to be participants in these environments and to act as co-learners rather than all-knowing guides. The learning goals detailed in the first column of Table 1 closely match the learning goals distinguished by Hager (1994). In the first configuration teachers are assumed to address discrete knowledge of a declarative or procedural nature, in the second configuration students are assumed to strive for general skills, and in the third configuration the prominent learning goal is to reach professional competence. We speak of discrete declarative and procedural knowledge when knowledge and skills are taught to students as part of a discipline, such as history. physics or educational psychology. The term discrete refers to the fact that knowledge and skills are not necessarily directly related to professional tasks. We use the term general skills when students learn skills at a crosscurricular level (in different lessons and out of school situations) and are expected to transfer skills to new situations or to particularize skills in new contexts (Alexander, Graham, & Harris, 1998). It is assumed that general skills are complex and that discrctc knowledge and skills are subsumed under general skills. We refer to professional competence when students acquire a combination of attributes, such as knowledge, skills and attitudes, which enable them to perform a role or complex task in an authentic. professional context (Brown, Collins, & Duguid, 1989). In the Dutch context, the term competence is used to refer to attributes that enable a person “to handle complex professional tasks in an appropriate, process- and product-oriented manner” (see for a discussion Westera, 2001). Tasks that should be handled competently by a novice teacher are, for instance, lesson preparation, cooperating with colleagues. and communicating with pupils’ parents (Moltmaker & Meulenkamp, 2000). In comparison to general skills competencies are equally complex. the difference being that general skills arc not associated with one particular domain of application, while competencies are closely associated with particular professions or -jobs (Klarus, 1998). While it is relatively easy to formulate well-defined standards and criteria for general skills, this is more difficult for competencies (Wolf & Cumming, 2000). For instance, the competence required to communicate with the pupils’ parents differs very much in schools that are located in multicultural neighbourhoods from that in schools with a mono-cultural population. The second column in Table 1 addresses learning arrangements that are congruent with the three approaches to learning and instruction. In the row of the teacher-directed approach, learning arrangements are aligned to the assumption that students profit from learning episodes in which teachers take the lead, such as teacher-directed courses and guided practice. In the row of the standard-oriented approach, learning arrangements are aligned to the idea that students should engage actively in learning tasks. Examples are strategy training and exercises, which involve strategy employment under varying circumstances. In the row of the competence-oriented approach, learning arrangements are aligned to the idea that learning is situated and that learners should learn to profit from learning episodes in which they participate as novices in authentic professional work. This includes projects such as the learning practices that we discussed earlier. The third column in Table 1 provides examples of assessment procedures that are congruent with the other entries in each row. In Configuration 1, it is a reasonable option to use tests and examinations that are composed and marked by teachers. In Configuration 2, it is in line with both learning goals and arrangements to employ tests that allow students to

M. Elshout-Mohr et al. /Studies in Educational Evaluation 28 (2002) 369-390

315

perform open tasks in their own manner. Performances on these open tasks may then be assessed by fixed evaluation scales, about which students have been informed beforehand. In Configuration 3, portfolio presentations (Dierick & Dochy, 2001) and critical situations (Sandberg, Anjewierden, Groothuismink, De Hoog, & Giebels, 1999) are suitable instruments for assessment. Such instruments allow students to provide information about the manner in which they construct and implement their vision on professional work and life-long learning. Preferably, representatives of the professional field should partake in the assessment committees (Hofstee, 1999; Uhlenbeck, 2002), because assessors with different backgrounds often employ divergent standards, which may all be relevant for the assessment of professional competence. Special Features

qfthe

Three Configurations

Table 2 calls attention to some details of the three configurations that are of special interest for the present article. The table focuses on equality, and more specifically on equality in regard to learning arrangements (Column 2). the ways of presenting learning outcomes (Column 3), and the standards and assessment criteria (Column 4). Table 2:

Equality for All Students of Learning Arrangement, Presentation of Learning Outcomes and Assessment Criteria in the Three Educational Configurations

Configuration

Learning arrangement

Presentation of learning outcomes

Standards and assessment criteria

Teacher directed

Teacher directed: equal for all students.

Teacher directed: equal for all students.

Established by the teacher: equal for all students.

Standard oriented

Student directed: dz~ferent learning routes.

Teacher directed: equal for all students.

Established by a panel: equal for all students.

Competence oriented

Student directed, affected by personal preferences and opportunities: different learning routes.

Student directed and affected by personal preferences and opportunities: d@erent for students.

Adapted to preferences and opportunities: different for students. Even the composition of the assessment panel might be different.

Table 2 shows three things: first that accommodation to the alignment principle is optimal in Configurations 1 and 3. Where learning arrangements are equal for all students (in Configuration l), they are invited to present their learning outcomes on standardized tests and the assessment criteria are equal too. Where learning arrangements are different (in Configuration 3), students present different learning outcomes, and the standards and criteria vary accordingly. Even the composition of the assessment panel may be adapted to

316

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002) 369-390

the particular learning routes of individual students. As far as equality is concerned, Configuration 2 is a mixed model. On the one hand, students are encouraged to direct their own learning routes. but on the other standards and criteria are fixed. In educational settings it is common practice to handle this apparent contradiction by making the standards known beforehand. Usually the implicit reasoning behind this practice is that students should have the opportunity to internalise these standards and apply them to steer their “self- directed” learning process (Dochy, Segers, & Sluijsmans, 1999). We shall return to this reasoning and its implications below. Second, Table 2 shows that students arejudged by standards that are equal for all in Configurations 1 and 2, while this is not the case in Configuration 3. From the point of view of the “equality principle” that is shared by many educators and students, the situation in Configuration 3 is undesirable. This equality principle states that assessment is only fair when students demonstrate their achievements in identical testing situations and when identical assessment criteria are used for all students. According to this principle it is not fair. for instance, that the assessment procedures allow part of the students to present competencies in teaching pupils in multicultural classes. while other students present competencies in teaching pupils in mono-cultural classes. There is no guarantee that both tasks are equally difficult and thus there is no guarantee that the better student will get the better grades. A third conclusion to be drawn from Table 2 concerns a special feature of Configuration 3. The alignment principle is in conflict with the equality principle. The alignment principle requires that students who have developed competencies by participation in authentic prqjects of their own choosing should be allowed to demonstrate these competencies in appropriate. and thus potentially different, conditions, and to gain due credits for their achievements. Not all students have, for instance, the ambition to become experts in dealing with the problems of multicultural teaching and those who do not should bc judged against other standards than those who do. The equality principle, however, states that individualized assessment procedures are essentially unfair and therefore undesirable in an educational context. This special feature of Configuration 3 may form an obstacle for educational innovators who want to transform a traditional curriculum into a dynamic competence-oriented one. They themselves need to come to terms with the conflicting principles first. Once they have decided to ground their assessment procedures on Configuration 3, they will also have to deal with the doubts and objections of colleagues, teachers and students. In the next section, some empirical findings about these aspects of the innovation process in the EFA are presented and discussed.

Transition

Towards

New Assessment

Procedures

From the start, the faculty involved educational researchers in the process of innovation. The researchers were asked to monitor and evaluate progress, to make existing educational expertise available for staff and students, and to ensure that emerging expertise was shared and recorded. Recently an evaluation was made of the new assessment

M. Elshout-Mohr et al. /Studies

in Educational Evaluation 28 (2002) 369-390

371

procedures used in the years 1999-2000 and 2000-2001 (Overmaat & Otter, 2001; Overmaat, Dietze, Oudkerk Pool, & Wilschut, 2001). This evaluation focused on the assessments that concluded the first study phase according to five divisions. The following materials were collected and analysed: documentation about the organization of integral assessment sessions, information provided to assessors and students in answer to a questionnaire, lists of competencies, descriptions of critical situations, criteria and cutting scores. Also. oral interviews were held with mentors, teacher-trainers, assessors and students. Below we discuss the most relevant findings. One finding was that all the members of all divisions adhered to the idea of organizing one integral assessment at the end of the introductory year. The general set-up was as follows: As a first step, students prepared a portfolio presentation. In order to demonstrate that they were competent to enter the main phase of the curriculum, students had to construct an electronic portfolio throughout the year. In this portfolio students inserted a self-evaluation at the end of the first year, in which they gave a description of their progress in the different competence domains. For that purpose they had at their disposal a list with competencies and the criteria they had to meet in order to be judged main phase accessible. To sustain claims in their self-evaluation, students had to refer to documents in their portfolio, such as papers, self-reflections based on meetings with a tutor, feedback received from teachers. test results, products produced in learning practices, feedback on these products, peer evaluations, and so on. Once completed, the portfolio was checked by one or two teachers who acted as counsellors or pre-assessors. These persons indicated where they found the self-evaluation unclear and where they had doubts about the adequacy or completeness of the presented evidence. The student then got the opportunity to improve on the portfolio before presenting it. in the so-called portfolio presentation. A second component of the integral assessment session consisted of so-called critical situations, which were defined as situations that represent authentic professional work and demand competence, as well as effective use of knowledge and skills. Students were for instance confronted with the task to organize a classroom instruction or a conversation with parents of an unruly class. They worked on these tasks in small groups of two or three students. First they had to make a written plan, in which they showed how they intended to approach the task and from which (theoretical) perspectives they chose to analyse it. Then they had to execute parts of the plan in a real or simulated situation and finally they had to reflect on the execution and the plan. Processes as well as products of the students’ work were subject of evaluation. The third component of the assessment was a so-called roundtable conversation among five students and their assessors. During this conversation the students reacted to each others’ work and to questions and comments of the assessors. To conclude the assessment, the assessors carefully considered all available evidence and evaluated students’ competence to enter the main phase of the study. Final judgments were presented to the student both orally and in writing. The study revealed that the organization was similar in all divisions. However, the stakes were different. In two divisions the final judgements were summative and decisive for students’ admission to the main phase. In three divisions they were formative and meant to guide students towards increased insight and reflection about strengths and weaknesses in their progress. Respondents from all divisions agreed, however, that the new assessment procedures differed substantially from the more traditional ways of testing they were used to. The integrative application of

378

M. Elshout-Mohr et al. /Studies

in Educational Evaluation 28 (2002) 369-390

knowledge and skills in a critical situation was much appreciated. Direct information about the students’ planning, executive activities and reflections in authentic professional situations was seen as a valuable supplement to the information provided in the portfolios and portfolio presentations. The evaluation study had still another focus. The evaluators closely inspected how assessment of expert knowledge, i.e., the didactical, pedagogical and content knowledge that a teacher must have, was handled by the various divisions. The finding was that critical situations and portfolios were indeed used in the integrative assessment sessions, but that all divisions employed a number of standardized tests for measuring student achievements throughout the year. These tests concerned expert knowledge covered in the courses, which were offered by way of “sources” and “training sessions”. Moreover, the evaluation made it quite clear that teacher-trainers considered the results on the standardized tests as far more important than the outcome of the integrative assessment session. Evidence for this was that all divisions still expressed the results of the standardized tests in terms of course credits, just like they used to do before the innovation. The general rule was that students had to accumulate a certain number of credits (2X out of a possible maximum of 40) to gain admission to the integrative assessment session. As we mentioned earlier, three divisions used the assessment exclusively as a diagnostic instrument meant to give formative feedback to students. Students in these divisions received four credit points each for their participation in the assessment, regardless of the competencies displayed in the portfolio presentation and the critical situations. To gain admission to the main phase they had to complete enough courses to collect 40 credits. The two other divisions counted more heavily on the outcome of the assessment. Once admitted (on the basis of having obtained 28 credits or more) students could compensate for “missing” credit points (up to 12 points) by performing well during the integrative assessment session. This implied that students with just 28 credits could gain admittance to the main phase just as well as students with for instance 36 credits, depending on competencies evidenced in the portfolio presentation, critical situations and round table conversations. Oral interviews held in the evaluation study revealed that profound discussions took place among teachers and students about the fairness and representativeness of the final evaluation in the first year. What these teachers and students disapproved of was the way in which students in these divisions were allowed to compensate - by competencies shown during the integrative assessment session - for “normal” credit points associated with courses. One of the reasons was that they considered this unfair to students who worked hard for getting credits in the “regular” manner, i.e., by taking courses and standardized tests. In addition, teachers doubted that the tools used during the integrative assessment session gave a reliable and valid impression of the expert knowledge students acquired in the first year. Most teachers expressed the opinion that assessment tools that are explicitly designed to gain information about professional competence are not suited to produce a clear picture of students’ expert knowledge. Although they did not deny that portfolio presentations and critical situations may provide indications of the presence of such knowledge, they pointed out that other instruments, i.e., conventional tests, are much more adequate to measure whether a student has reached sufficient coverage of the relevant knowledge domain. The third and most important reason why teachers had great difficulties accepting the outcomes of the integrative assessment procedures as decisive for a high-

M. Elshout-Mohr

et al. /Studies

in Educational Evaluation 28 (2002) 369-390

319

stake decision. i.e., the decision about students’ admittance to the next study phase, was that they considered expert knowledge an independent indicator of future performance. They were convinced that this indicator should not be substituted by professional competence. In summary, we drew the following conclusions: A large majority of respondents in this evaluation study gave a positive judgement about an assessment of professional competencies and fully agreed with the implementation of an integrative assessment procedure. Full alignment of the assessment system to the intended constructivist, competence-oriented curriculum was not realized however for two reasons. A first obstacle was that many respondents strongly preferred the use of fixed and well-defined standards that were equal for all students. In absence of such standards it was deemed unjustified to base high-stake, summative decisions on assessment outcomes. A second obstacle was that many respondents were convinced that expert knowledge should be measured directly. Indirect evaluation of expert knowledge, based on information that students provided during portfolio presentations and critical situations was not accepted as substitute. There were no large differences in opinion between students, teachers who function as assessor, and teachers who did not participate in assessment panels. In our opinion these findings are not idiosyncratic for the EFA. Similar reactions, doubts and conflicts have been found elsewhere in innovative faculties (Driessen & Van Vleuten, 2000). In general, it is easily accepted that complex testing procedures with a high degree of authenticity should be used in so called “developmental assessments”, i.e., assessments that have a diagnostic and formative purpose (Gipps, 1994). In contrast, the use of authentic and multiple procedures is much disputed in high-stake assessments, which have a summative purpose and are decisive for acceptance or rejection of the candidate for a desired position (Linn, Baker, & Dunbar 1991; Reckase, 1995). Objections are raised, for instance, against the use of portfolios. One of them is that the role of standards is unclear. In a portfolio students include attestations received from different persons, such as supervisors, co-workers and peers with whom they participated in leaning projects. Each of those persons may have used different standards and may have been biased in his or her judgement (Albanese, 1999; Gitomer, 1993; Schoonen, 1991; Van Gelderen & Oostdam, 1992; Wigdor & Green, 1991). Therefore one may question on which grounds assessors vouch for the reliability of the attestations and how they legitimize conclusions based on the portfolio. The use of critical situations too has been disputed. Research literature suggests that ten to twelve critical situations should be presented to a person in order to gain a reliable measure of just a small number of his or her general skills or competencies (Brennan & Johnson, 1995; Shavelson, Webb, & Rowley, 1989). Evidently such a large number of critical situations cannot be included in an integral assessment procedure. Further Steps to Promote

Congruency

Between

Curriculum

and Assessment

When we presented the three configurations (see Table l), we stated that it might be too early to discard Configurations 1 and 2 in favour of Configuration 3. One could say that this conjecture was confirmed by the findings of the evaluation study. The wish to use fixed standards can be interpreted as a plea for the use of Configuration 2. Within the framework of Configuration 2 it is common practice already to combine a learning arrangement where students are encouraged to take different learning routes with standardized assessment

380

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002) 369-390

procedures. In the same vein, one could interpret the wish to measure the students’ expert knowledge as a plea for the use of Configuration 1. Within the framework of Configuration I it is common practice to test students’ knowledge throughout the year in direct relation to the courses that teach this knowledge. A simple conclusion from the results of the evaluation study would be that the faculty should decide to combine the new integral assessment approach with the two other approaches. In that case one should settle for partial rather than full alignment with the intended curriculum, since there would be substantial friction between the intended curriculum and the assessment system. We want to argue that there may be better ways to reconcile individualized competence-oriented learning arrangements with an acceptable level of standardization. Our proposal, however, demands a further step in the direction of student-orientedness, which may be hard to take. In faculties such as the EFA, a first step from teacher-oriented towards student-oriented assessment procedures has been taken successfully by establishing an integrated assessment procedure at the end of each study phase. The following step, which we want to propose here, concerns the use of standards. Traditionally, standards are defined and imposed by experts and teachers. Although ownership of standards by experts and teachers may be conceived as a necessity, it is nevertheless a source of friction in constructivist competence-oriented curricula. In the foregoing discussion of Configuration 2 (in Tables 1 and 2) we already mentioned that it is somewhat confusing to stimulate students to find individual learning routes when their skills are measured eventually by fixed standards. Loosening the standards is evidently no option, because neither assessors nor students feel comfortable in the absence of fixed standards. An option that may be more promising in the context of self-directed learning is to convert to another way of thinking about the ownership of standards and to switch over to the view that it should be one of the aims of education to reach shared ownership of standards (Sluijsmans, 2002). Shared ownership, in our opinion, implies that standards are made explicit, and that standards are shared and negotiated rather than transferred in one-way communications from experts to students or from one subgroup of experts to another subgroup. It will doubtlessly demand energy, inventiveness and persistence for experts and teachers to share standards among each other and with novices, but recent literature has something to offer (Dochy, Segers, & Sluijsmans, 1999; Harris & Bell, 1990; Struyf, Vandenberghe, & Lens, 2001). In projects that used co-, peer- and self-assessment methods, several procedures have been developed that are potentially useful to realize sharing and negotiation of standards in order to enlarge students’ role in defining, applying and maintaining standards. If faculties would succeed in integrating these procedures in the curriculum, this might lead to a new situation where the friction between individualization and standardization is diminished. No serious friction is to be expected when standards are employed that students experience as their own. Table 3 shows the rough outlines of an integral assessment procedure that could be aligned to a curriculum that is both student oriented and competence oriented. The procedure capitalizes on the idea that students become serious partners in the defining and application of standards. The first column states three goals of self-directed learning: 1) professional competence, 2) proficient use of relevant standards, and 3) expert knowledge. The second column shows the learning arrangements. In contrast to the learning arrangements presented in Table 1, all learning arrangements in Table 3 capitalize on students’ self-directed learning. These arrangements will be discussed presently. The third

M. Elshout-Mohr

et al. /Studies

in Educafional Evaluation 28 (2002) 369-390

381

column outlines forms of assessment that are congruent with the learning goals and arrangements in each row. These assessment procedures, too, will be clarified below. The total exposition in Table 3 differs in several ways from the exposition in Table 1. The main difference is that all three configurations in Table 3 are resonant with the same approach to learning, namely the constructivist approach, where self-directed inquiry is the leading principle, while in Table 1, they are resonant with different approaches to learning, namely behaviourism, direct instruction and self-directed learning. Because of their founding in a common approach, the three configurations in Table 3 can be conceptualised as components of one culture of assessment, a culture that is more fully aligned to a constructivist competence-oriented curriculum than a composite of the configurations presented in Table 1. A second point is that the three learning goals are highly interrelated. They form aspects of professional competence rather than separate learning goals. They are presented separately here for the mere reason that they need separate attention from teachers, students and assessors. Table 3:

Components Assessment

of a Proposal

Learning goals of studentsin a constructivist competenceoriented cllrriculum Professional competence

for Reconciliation

Learning arrangements

of Individualized

and Standardized

Integral assessment at the end of each study phase:

Active participation in professional work or simulations of such work; guided work experience; construction of personal working and learning concept.

Students demonstrate views on the profession and their own competencies in a portfolio, a portfolio presentation and critical situation(s). Professional competence is discussed and judged by the assessment panel. Critical situations are used to facilitate communication and formative feedback.

Proficient use of relevant standards

Students participate in co-, peer- and self-assessment integrated in the curriculum. Professional standards are shared and negotiated and students are trained in defining, applying and maintaining standards.

Students show, explain and justify their (reflective) use of professional standards in their portfolio, portfolio presentation and critical situation(s). Their use of standards is discussed and judged by the assessment panel. Critical situations are used to facilitate communication and formative feedback.

Expert knowledge

Participation in the learning practices (communities of learners) and training sessions. Consultation of sources, such as experts, handbooks, study books, and internet.

Students take a cumulative progress test before each integrative assessment. Students report on the outcomes of this test and demonstrate their application of expert knowledge in their portfolio, portfoliopresentation and critical situation(s). Expert knowledge is discussed and judged by the assessment panel. Critical situations are used to facilitate communication and formative feedback.

382

M. Elshout-Mohr

et al. /Studies

Elabot-ation

in Educational

Evaluation

28 (2002)

369-390

of the First Row in Table 3

The first row in Table 3 does not substantially differ from the third row in Table 1 but now it is flanked by aligned configurations in rows 2 and 3. That is to say that the portfolio assessment and the critical situations are now used in combination with other instruments that stem from a constructivist, competence-oriented approach to learning. By combining the three configurations the burden posed on each of them is diminished. Earlier we discussed some problems regarding the use of portfolios and critical situations for the purpose of summative assessment. One problem concerned the multiple standards on which the attestations and self-assessments in a portfolio are based. Although this problem will not be solved easily, the design offers at least an idea about how to handle it, namely by making the standards explicit and by making explication and justification of standards into a subject of discussion and training. The more students understand the professional standards, their definition and use, the more they will succeed in gathering reliable attestations and in estimating the reliability of evidence that they present in their portfolios and presentations. In regard to the use of critical situations, a major objection has been that it is impossible to gain reliable insight in a student’s expert knowledge or competencies by means of two or three critical situations. In the design in Table 3 the critical situations are still incorporated because they are, in general, much appreciated as components of integral assessments. However, the critical situations are given a new function because the checking on expert knowledge is organized in new ways (as indicated in row 3) and the students are better equipped to self-assess their competencies (as indicated in row 2). Rather than employing the small number of critical situations as an (inappropriate) tool to measure expert knowledge and competencies, we propose to use them as an (appropriate) tool in the communication between assessors and candidates. The candidates’ reactions to the critical situations (their planning, actions, and reflections) can be referred to by both assessors and candidates. Used in this manner, the critical situations provide anchors for the communication between students and assessors, who each have their own perspective on competencies, expert knowledge and so on. Uhlenbeck (2002) provides an example in which assessors and a candidate discover, in reference to a critical situation, that they have totally different views on what “constructive feedback” entails. Concrete anchoring is needed also to deal with diverging standards. Students’ self-evaluations tend, for instance, to be correlated with their competence, in the sense that good students tend to underrate themselves and weaker students tend to overrate themselves (Boud & Falchikov, 1989; Sprangers, 1989). Reference to concrete actions and utterances may reveal these misratings and therefore may prevent undesirable consequences. Elaboration

of the Second Row in Table 3

The second row of Table 3 concerns implementation of the idea to give students a substantive role in defining, applying and maintaining standards. Traditionally, teachers prefer to simply inform students about the standards that will be used in order to help them to develop the “right” competencies and skills instead of competencies and skills that merely satisfy students’ own (poorly informed) standards. Moreover. confrontation with

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002) 369-390

383

standards of other persons, such as experts in the field, is often postponed in order to prevent students from getting confused. As we have seen before, there are some problems with using fixed standards which stem from just one source, in this case, the teachers. Once the standards are defined and fixed, it is difficult for teachers to give credit to students’ unique, unforeseen achievements. It is also difficult to delegate parts of the assessment task to relevant others who may have divergent standards. Most important, however, is that the use of fixed standards is not aligned to a constructivist curriculum, which aims at selfdirected learning. The second row in Table 3 proposes that students should be given adequate help in getting familiar with standards and criteria used by others, in learning to explicate their own standards, and in gaining insight in the processes of defining and applying standards (Dochy, et al., 1999). This probably requires complicated learning arrangements, which may however be essential for achieving high-level professional competence (Harris & Bell, 1990; Struyf, et al., 2001). As suggested in Table 3, teachers and students should view standards and criteria as issues that require study and discussion in authentic situations throughout the curriculum, such as learning practices and courses (sources and training sessions). Co-assessment (assessment by student and teacher), peer-assessment (assessment by peers) and self-assessment may be used to organize these study- and discussion-sessions (Sommervell, 1993). Once the students have gained insight in the multiple standards of the professional community they intend to enter, teachers need no longer impose their standards in a one-sided way. Instead, teachers, co-workers in learning practices and students can jointly evaluate students’ performances and competencies, each with a contribution of their own. The general idea, expressed in Table 3, is that students should show, explain and justify their knowledge and (critical) use of standards in various sections of their portfolio, in their portfolio presentation and in their handling of critical situations. Thus, they enable assessors to judge their proficiency in regard to professional standards.

Elaboration of the Third Row in Table 3 One aspect of professional competence consists of expert knowledge. Within the framework of self-directed learning students should determine for themselves where and how to master this knowledge. They consult sources and participate in training sessions of their own choice in their own time, within the limits of the common understanding that each student’s relevant expert knowledge should grow at a certain rate. Although it is much applauded that students should attend courses and training sessions in direct connection to an authentic project or learning practice, there is also a strong need to check and control their progress. Are the students accumulating enough expert knowledge when they are directing their own learning? Although it is of course possible to act along the lines of Configuration 1 in Table 1, this may interfere with the intended curriculum. Students should not be hindered by test taking during periods when they are engaged in projects, which reveal the need to extend expert knowledge and urge them to consult sources. A better arrangement might be to make it obligatory for students to take a progress test, at least once a year, or more frequently if

384

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002) 369-390

they wish. On the nature of a progress test we cite Driessen evaluated the test as used in the setting of problem-based learning:

and Van Vleuten,

who

A progress test is basically a conventional knowledge test using multiplechoice questions of the true/false type, being formatted and used in such a way that it prevents test-directed studying, and reinforces longitudinal functional knowledge. A progress test can best be conceived of as a final examination. It is a comprehensive examination representing the ultimate objectives of a curriculum. It is made up of questions from all the disciplines involved. The test is given at intervals, i.e. three times a year. It is administered to all the students enrolled in the curriculum. The questions could also either be answered or may be completed with a “don’t know option”. (. .). In the course of the curriculum, the student would “progress” towards the final level of knowledge. (...). Eventually, if a student studied regularly. essentially following his own or the group’s learning objectives, he would find the content reflected in the test and would show progress. (2000, p. 238). A progress test is congruent with self-directed learning, even though it is a traditional test. It provides students with valuable feedback about the growth of their expert knowledge. In order to enable students to present their knowledge growth in more authentic ways than through multiple choice items, Table 1 suggests that students may also present their knowledge in forms of their own choice (in addition to the progress test). The following example of a critical situation that enables a student to demonstrate his or her expert knowledge in an authentic way, concerns student-teachers from the history division of the EFA. Over the years these student-teachers have to accumulate a considerable amount of expert knowledge in the field of history. They need this knowledge to perform professional tasks such as giving feedback to pupils’ written papers. A student-teacher who does not yet possess a rich store of historical knowledge needs a lot of time to evaluate the assignments, especially if the pupils were free to select a subject for their paper. He or she has to consult study books, the encyclopaedia or the internet to judge the correctness, relevance and completeness of the pupil’s work. To give authentic evidence of increasing expert knowledge, a student-teacher might refer to a situation where he or she proved to be proficient in evaluating papers. Such a situation might occur more or less incidentally in the course of a learning practice or could be sought actively by a student who wishes to collect this type of evidence for his or her portfolio presentation.

Summary

of the Proposal

in Table 3

The three configurations presented in the rows of Table 3 constitute components of a recommendation to faculties that strive to reconcile individualization of assessment with its standardization. First we recommend to keep hold of the achieved integral assessment sessions, in which professional competence is assessed on the basis of portfolios, portfolio presentations and critical situations. The achievement is in line with the intended curriculum, although it may still be possible to improve on the procedures of the integral assessment. One improvement might be to use critical situations explicitly for

M. Elshout-Mohr et al. /Studies

in Educational Evaluation 28 (2002) 369-390

385

communicational purposes. Secondly, we argue that faculties should reconceptualize the issue of standardization. While it is common practice to view teachers as the proper persons to define and apply standards and to inform students and relevant others about the criteria and cutting scores, the new view would be that the ownership of standards is shared. Also it should be taken for a fact that students and relevant others, such as co-workers in prqjects and learning practices, share responsibility in maintaining professional standards. A reconceptualization of this nature might be implemented by the creation of learning arrangements, which aim explicitly at skills involved in the defining, sharing and applying of professional standards. Thirdly, we suggest to remove one obstacle on the way to acceptance of a student-oriented curriculum, namely the concern about students’ expert knowledge. This may be done by using a progress test, which enables students to gain and report valid information about the growth of their expert knowledge. It should be noted that Table 3 presents no configuration related to general skills, while Tables 1 and 2 did present such a configuration. The reason for this omission is that we want to argue that separate training and testing of general skills doesn’t fit well into a competence-oriented curriculum. Of course it is important that students’ competencies are generic, in the sense that they have transfer value for new situations, but in our opinion this aspect should be assessed within the context of the integral assessment of the students professional competence. Transferability of achieved competencies should be an issue throughout students’ portfolios, their self-evaluations and the portfolio assessment.

Conclusion The issue of combining individualized assessment with an acceptable degree of standardization is a complicated one (Elshout-Mohr & Oostdam, 2001). The views that we presented have been influenced by the context in which we encountered and studied this problem, namely the constructivist, competence-oriented curriculum of an Educational Faculty in the Netherlands. It remains to be seen whether the recommendations can be brought into practice. Implementation has not yet taken place. Nevertheless we wanted to offer our ideas for inspection by other faculties, who are involved in similar innovation projects. Our position in this article has been that any concern about standardization of teachers, students and other people involved, should be taken very seriously. Innovation in the direction of a truly student-oriented curriculum is not sufficiently served by taking on individualized assessment procedures for formative purposes only. The fact is that assessments procedures employed for summative purposes have a much stronger influence on the attitudes and behaviour of students and teachers, especially when important decisions are at stake (cf. Hambleton, 1996; Sambell, et al., 1997; Van Hout-Wolters, 2000). It is premature to claim that joint implementation of the three components presented in Table 3 will suffice to demonstrate that it is indeed possible to maintain standards without abandoning the framework of a constructivist, competence-oriented curriculum. It is promising however that all three recommendations fit very well into a curriculum in which students are expected to take responsibility for their own learning. The three configurations can be combined without violation of either the alignment principle or the

386

M.

Elshout-Mohr et al. /Studies in Educational Evaluation 28 (2002) 369-390

equality principle. The former principle is ensured by the fact that formative and summative assessment procedures are aligned to each other as well as to the learning arrangements, whereas the latter principle is ensured by giving students a justified share in defining, applying and maintaining standards. It cannot be denied that the proposed assessment procedures may be experienced as complex and time consuming. In this respect our approach does not differ from other approaches to assessment in innovative curricula. In general it is recommended to employ various qualitative and quantitative instruments in conjunction and to base the final decision on converging evidence (Birenbaum & Dochy, 1996; Shulman, Haertel, & Bird, 1988: Uhlenbeck, 2002). One redeeming quality of an integral assessment at the end of each study phase might be that the amount of time spent on assessment proper would be extremely limited in colnparison to the time that is traditionally spent on test taking. Of course, students have to work on their portfolios, self-evaluation and so on throughout the year, and they have to consult teachers and other persons to receive formative feedback (which can be used as evidence in the portfolios), but they will be allowed to organize these activities in natural ways, which demand minimal organization and cause minimal stress. Once students and teachers begin to experience the advantages of flexible linking between formative feedback (throughout the year) and summative feedback (at the end of each study phase) this might well become one of the strongest arguments to find effective ways to switch from teacher-directed use of standards to an assessment culture where standards are detined, applied and maintained by a wider group of persons, including the students. Acknowledgement We wish

to thank

the reviewers

and editor

of.CEE

for the helpful

comments

on an earlier

version

of

the manuscript.

References Albanese.

M. (I 999).

Ac.udemic Medicine, Alexander, and prospects.

P.A.,

D.C..

& Schuster Biggs,

educational

quality:

factors

in the erosion

of professional

standards.

Graham,

S.. & Harris,

K.R.

(1998).

A perspective

on strategy

research:

progress

Educational Psychology Review. IO (2). 129- 154.

Berliner. Simon

Rating

74 (6), 652-658.

J.B.

& Calfee.

R.C.

(Eds.)

(I 996). /fundhook of edttcationcd psychnlo~~. New

York:

Macmillan.

(I 996).

Enhancing

learning

through

constructive

alignment.

Higher Educution, 32, 347.

364. Birenbaum,

trnd &o&edge.

Boekaerts. makers,

M., & Dochy,

Boston:

educators,

M.

F. ( 1996). A1ternative.r in us.wssment ~~‘achievements. leurningproce.~ses

Kluwer

Academic.

(1997).

Self-regulated

teachers,

and students.

learning:

Lecrrning

a new

concept

embraced

and btstruction. 7 (2), I6 I - 186.

by

researchers.

policy

381

M. Elshout-Mohr et al. /Studies in Educational Evaluation 28 (2002) 369-390 Boud, D.. & Falchikov, N. (1989). Quantitative studies on self-assessment critical analysis for findings. Higher Education, 18 (5). 529-549. Brennan, R. L., & Johnson, E. G. (I 995). Generalizability Measurement: Issues and Practice, 15. 9- 12, 21.

in higher education: a

of performance assessments.

Brown, J.S., Collins. A., & Duguid, P. (1989). Situated cognition Educational Rese;rrcher, 18 (I), 32-42.

Educational

and the culture of learning.

De Corte, E. (I 990). A state-&he-urt of research on learning und teaching. Keynote lecture presented at the first European Conference in Higher Education, Aalborg University, Denmark, 23-25 April 1990. Dierick, S., & Dochy, F. (2001). New lines in edumatrics: New forms of assessment assessment criteria. Studies in Educational Evuluation, 27, 307-329.

lead to new

Dietze, A., Snoek, M., & Wielenga, D. (2000). Exploring the future: A Dutch experimental teacher education programme. In G.M. Willems, J.H.J. Stakenborg & W. Veugelers (Eds.), Trends in Dutch teacher education (pp. 49-60). Leuven: Garant. Dochy, F., & McDowell, Evuluation. 23 (4). 279-298.

L. (1997). Assessment

as a tool for learning. Studies in Educational

Dochy, F., Segers, M., & Sluijsmans, D. (I 999). The use of self-, peer and co-assessment education: A review. Studies in Higher Education, 23 (3) 33 I-350.

in higher

Driessen, E., & Van der Vleuten. C. (2000). Matching student assessment to problem-based learning: lessons from experience in a law faculty. Studies in Continuing Education, 22, 235-249. Duffy, T.M., & Jonassen, Hillsdale, NJ: Erlbaum.

D.H. (Eds.) (1992). Constructivism

and the technology of instruction.

Elshout-Mohr, M., & Oostdam. R. (2001). Assessment van competenties curriculum [Assessment of competencies in a dynamic curriculum]. Amsterdam: SCO.

in een dynamisch

Entwistle, N. (2000). Constructive alignment to improve the quality of learning in higher education. Paper abstracted in the Proceedings of‘the Onderwijs Research Dagen [Educational Research Days] (pp. 34). Leiden: ICLON/Universiteit Leiden. Gipps, C.V. (1994). Beyond testing: towards a theory ofeducational assessment. London: Falmer. Gitomer, D.H. (1993). Performance assessment and educational measurement. In R.E. Bennett & W.C. Ward (Eds.), Construction versus choice in cognitive measurement; issues in constructed response, perfirmance testing, undportfilio assessment (pp. 241-264). Hillsdale, NJ: Erlbaum. Hager, P. (1994). Is there a cogent Australian Journal qfEducation, 38 (I), 3-18.

philosophical

argument

against

competency

standards?

M. Elshout-Mohr

388

Hambleton, &

R.C.

R.K.

Calfee

(Eds.).

Harris.

D.. &

et al. /Studies

(1996).

Advances

Hundhook

in Educational in assessment

of’educutionul

Evaluation

models,

ps_vcholqv

28 (2002) 369-390

methods

and practices.

(pp. 899-926).

New

York:

In D.D. Simon

Berliner

& Schuster

MacMillan. Bell.

( 1990). Elwluuting

C.

ur7d

a.s.se.s.sing ,fiw leurning.

London/New

York:

Kogan

Page-Nochols. Hofstee,

W.K.B.

Princi1x.v

(1999).

own heoordelen

[Principles

of assessment].

Lisse:

Swets

&

Zeitlinger. Kanselaar, J. & T. Duffy

G., De Jong. T.. Andriessen, New learning

(Eds.),

Klarus,

C’ompetenties

R. (1998).

heoordeliq

models

for course-independent

P. (2000).

Dordrecht/Boston/London:

een

erkennen.

/~~~n,egonq/hunk~l~jk~ and procedures

J., & Goodyear,

(pp. 55-83).

studie

he~oepscom~,etenties

vm7

assessment

In R-J. Simons,

Van der Linden.

Kluwer.

nuar

modellen

[Accreditation of professional

en procedures

of competencies: competencies].

voo~

A study

on

‘s-Hertogenbosch:

CINOP. Knoers.

A. (I 996).

Weinert

(Eds.7,

Internationul

Oxford:

Pergamon. Korthagen.

Simons,

Van

der

Theories

and models

F.. Klaassen, Linden,

of learning

in educational

of developmentalund

encyclopordiu

C.. &

Russell.

Duffy

(Eds.).

Ne~c learning

(pp.

Assessment

in the service

of learning.

J. & T.

T. (2000).

New

settings.

instructional

learning

In E. de Corte

& F.E.

ps,vc’holoR)~ (pp. 3 17-32

in teacher

243-261).

education.

In

I ).

R-J.

DordrechtiBostonlLondon:

Kl uwer. Lane, (Eds.),

S.. & Glaser.

International

R. (19.96).

encylopuedia

uf’develr~pmentul

and inrtructionul

&

(1991).

In E. de Carte

p.syholo~3,

& F.E. Weinert

(pp. SOS-SOS). Oxford:

Pergamon. Linn. cspectations

R.L.,

Baker,

and validation

Moltmaker,

tijdschrifi

T.. &

H.F. (Ed.)

Overmaat.

Faculty

Overmaat.

M., Dietze,

VELON,

Tijdschr$ Reckase,

A4wszcremen/.

vow M.D.

performance-based

beoordeeld

assessment:

[Assessed

Velon:

competently].

Portfolio

I.vsues and Pructice.

Pool,

[Integrative

1. I2-

of teachers

I., & Wilschut,

experimentc>le [Evaluation and students]. A. (2001).

A step into

lerar~nuplriding. study

Amsterdam:

lntegratief

deel B.

of the experimental SCO.

beoordelen:

compentence-oriented

Een stap education].

13-20.

assessment:

14.

onderzork

assessment:

22 (2).

Erlbaum

le onderzoeksjuur

and perspectives

Lerurenop1eider.r.

Mahwah:

EZmkerrnd

en stztdenfen,

A., Oudkerk

opleiding

(1995).

Complex,

20 (8), I S-2 I.

Bekwaam

readiness.

M. (2000).

part B: Attitudes

naar competentiegericht

H. (2000).

Workforce

van docenten

of Education,

S.B.

2 1 (3 ), 30-3 5,

(I 997).

M., & Otter,

en percept&

Dunbar.

Edzrctrtionul Reseurcher,

Meulenkamp.

vow lerurenopleider.~. O’Neil,

utti/udes

E.L.. criteria.

a theoretical

estimate

of score reliability.

Educationul

M. Elshout-Mohr

Sambell. perceptions

et al. /Studies

K., McDowell,

in Educationat

L., & Brown,

of the consequential

validity

S. (1997).

Evaluation

“But

is it fair?“:

Studies

of assessments.

389

28 (2002j 369-390

An exploratory

in Educutional

study

Evaluation,

of student

23 (4), 349-

371. Sandberg. CREDIT

J.,

project:

Edrtcationul

Anjewierden.

Towards

Technology,

Schoonen. instruments

31 (4)

Shulman.

L., Haertel.

Shavelson.

~wrk

J.. De

framework

for assessment

R.. &

Giebels,

and accreditation.

VN,I s~hrjifi,ua~digheid.ssmetingen

evuluurie

E.,

Ph.D. thesis.

& Bird.

University:

R.J.,

Hoog,

M.

(1999).

The

.Journal

&if&h

of

Webb.

(1988).

T.

School N.M..

Amsterdam:

[Evaluation

of measurment

SCO.

Tr~~cn-c/ c&e,~~tlivc

rrsse.ssmen/ qftcachit~g:

A sport

of

of Education.

&

Rowley.

G.L.

(1989).

Generalizability

Americun

theory.

-I-l. X2-932.

Shulman,

of

/It

Unpublished

hawk in p~~+ycs.c. Stanford

P.sychologi.st,

Groothuismink.

28 l-298.

(I 991).

R.

for writing].

A.,

an international

I..%,

Haertel.

E., & Bird,

T. (1988).

in J~rogres.s

(Report

4-20-88).

Teacher

Toward

al/ernutive

Assessment

a.s.te.s.smen~s ofteaching

Prqject.

School

of

A report

Education:

Stanford

University. Shunk. md

D.H.

educutionu/ Simons,

P.R.J.,

Sluijsmans, Maastricht,

M..

conference, Sommerveil.

Sprangers.

Struyf,

S/z&n/

im~hww~t Maastricht.

D. ( I999).

(I 993).

H.

Issues

assessment. M.

( I989).

und perfixwuncr:

N CH’ learning.

T. (Eds.) (2000) in

c~v~.~.vmer7~~

Issue.s

~‘~~~.LIIY/~Y II more

Dordrecht:

Kluwer.

ofpeeru.s.se.s.sme~~~ .skills.

The lvuining

d,~wtrn~ic cwrhhn.

in assessment.

cntcrprise

and higher

~l.v.~essr~?er~/ trr?t/ E~~~/ut~tim in /fi,&r

K~.\/xIF~.\L’ .\hif/

I?.. Vandenberghe.

for students.

I! hlenbcck.

Paper

education:

Eductrtion,

presented

at the

The

case for self-,

1K (3). 221 -233.

trml /he r~~‘/r.o.cp~/ir~~ ~FC/CY/. linpublished

R.. & I,cns.

Ph.D.

thesis.

The

A.M.

T/w

(2002).

Van

Geldcren.

language Van

lessons].

Hout-Walters.

ban der Linden

A.,

&

in de lessen

& T. Duff)

Lelmde

tlndopn~cr7/

01

m

Ph.D. thesis. R.J. [The

7itlct7. 469.

(2000).

Ycu, Letming

124

m.w.ss~~m

Leiden:

I let

( 1992).

Oostdam, Nederlands

B.H.A.M. (Eds.).

W. (2001 ). ‘l’he evaluation

practice

of teachers

as a learning

E~YI/ULII~O~I. 27, 2 I 5-238.

S/udie.s irk Wuca/iomrl

(1.s(I for-ripz Itrngutr,y~. Unpublished

spreekvaardigheid Dutch

of‘leurning

SVO.

opportunity

E&i.sh

J.L., & Duffy,

Datawyse

& Wielenga.

Self-~egzlltrtiorz

NJ: Erlbaum.

Houston.

peer and collaborative

Hague:

Van der Linden,

D. (2992).

( 1994)

B.J. (Eds).

Hillsdale,

The Netherlands:

Snoek. AECT

& Zimmerman,

upplications.

pr~ocrtlzwe

ICI .ON Graduate gebruib

use of assessment

van

schemata

,for

heginning

School

teucher

beoordelingsschema’s for oral

in

of Education.

communication

voor in

132.

Assessing

active

(pp. 83. IO

self-directed

I ). Dordrecht:

learning. Kluwcr.

In P.R.J.

Simons.

J.L.

390

M. Elshout-Mohr

et al. /Studies

in Educational

Evaluation

28 (2002) 369-390

Wang, M.C., Haertel, G.D., & Wahlberg, H.J. (1990). What influences learning? A content analysis of review literature. Journal Westera, W. (2001). Studies.

33

ofEducational

Research,

Competencies

84, 30-44.

in education:

A confusion

of tongues. .Journa/

qf Curriculttm

(I ), 75-88.

Wigdor, A.K., Washington,

& Green, B.F. (Eds.) (1991)

Perfiwmunce

a,ssessment

The inside story:

The reality

i?z the workplace

(Volume

I).

DC: National Academy Press.

Wolf,

A, & Cumming,

instrument. Studies

in Educutional

J.J. (2000). Evaluufion.

of developing

an assessment

26. 21 l-229.

The Authors MARIANNE ELSHOUT-MOHR is a senior researcher in the SCO-Kohnstamm Institute for Educational Research. Faculty of Social and Behavioral Sciences, University of Amsterdam. She also participates in the research program of the Graduate School of Teaching and learning of the Universiteit van Amsterdam. Her research focuses on learning processes in secondary and higher education. RON OOSTDAM is a senior researcher in the SCO-Kohnstamm Institute for Educational Research. Faculty of Social and Behavioral Sciences, University of Amsterdam. His research interests include instructional objectives and learning processes for language education, measurement of language skills, literary response, and dynamic assessment. MARIANNE Educational Amsterdam. programs for Correspondence:

OVERMAAT is a senior researcher in the SCO-Kohnstamm Institute for Research. Faculty of Social and Behavioral Sciences, University of Her research interests include the design and evaluation of instructional L 1 and 1,2.