- Email: [email protected]
An evaluation network for educational change
Studies in Educational Evaluation PERGAMON
Studies in Educational
Evaluation
29 (2003)
43-56 www.elsevier.com/stueduc
AN EVALUATION
NETWORK
Alvaro Marchesi*, Alejandro
l
***Faculty
FOR EDUCATIONAL
CHANGE
Elena Martin**, Rosario Martinez Arias*, Tiana***, and Amparo Moreno**
*Faculty of Psychology, Complutense University of Madrid, Spain * Faculty of Psychology, Universidad Autdnoma de Madrid, Spain of EducatioqSpanish National Distance Teaching University, Madrid,
Spain
It is through the development of a strategy capable of encompassing the most relevant conditions of educational change that such change is best supported. This article describes the experience of an evaluation network being put into practice in Spain to encourage this type of change. First, the most relevant conditions needed for educational change are briefly outlined. Second, the experience of a school evaluation network which includes all the previously described conditions is more extensively developed. Conditions for Educational Change Innovations that adopt a multi-dimensional perspective have a higher chance of success. Moreover, experience reveals that educational changes must be adapted to the reality of the specific school, and therefore general reforms and assessments have to be sensitive to existing differences at the same time. Nevertheless, teachers perceive a great distance between the reform proposals from those in charge of education and their opportunities for putting them into practice. They also feel that their opinions are hardly ever taken into account, and this affects their active commitment to these changes. The same is true for educational research: Results are often considered to be far removed from the specific educational reality and with few suggestions to help teachers reorganise their teaching practices. The teacher, who has little confidence in external support systems and few available options for professional development, is limited to carrying out his or her work in the school and classroom environment without ihe opportunity to assess the possibilities of new forms of co-operation. 0191-491X/03/$ - see front PII: SOlSl-491X(03)00006-3
matter
0 2003
Published
by Elsevier
Science
Ltd.
44
A. Mc~rche.si
rt ul. /Studies
in Educational
Evaluation
29 (2OO.r)
4.jL.76
These reflections underline the importance of the following five conditions needed in order to develop good educational innovation: (1) a multi-dimensional and contextual outlook; (2) continuous assessment of how each individual school is being run; (3) inter-relations between schools: (4) connections between research and the reality of schools, and (5) channels for teachers to express their opinions. In the following pages each one of these is briefly analysed. Subsequently and more extensively, a “door to change” (Joyce, 1991) is described, i.e., the School Assessment Network, which incorporates the previously mentioned factors and which attempts to develop new prospects for educational change. Educational Change Must Adopt a Multi-Dimensional
and Contextual Outlook
Effective and lasting changes are not usually the result of partial and isolated initiatives. They demand global, systemic, interactive and contextualised approaches. Research on effective schools (Sammons, Hillman, & Mortimore, 1995, Teddlie, Stringfield, & Reynolds, 2000) has underlined the importance of a multi-dimensional outlook for understanding the reality of schools. Such an approach must be capable of analysing the school’s procedures, the impact and recognition of the head teacher’s job, coordination between teachers, classroom teaching, students’ educational progress, and parents, teachers and students’ assessment of how their school operates. Another dimension, however, which profoundly conditions educational action, is the socio-cultural context of the school. Its influence on students’ academic results has been extensively outlined (Sammons, Thomas, Mortimore, Owen, & Pennell, 1994, Teddlie, et al., 2000). As was directly and impressively shown by Mortimore and Whitty (1999), the single most important factor distinguishing the most academically successful schools is that only a few of their students come from socially disadvantaged homes. Consequently, school assessment which does not take into account the prior attainment of students and their background factors offers information which bears no justice to the reality of the particular school. Even if these factors are taken into account and the “value added” by the school (Goldstein & Myers, 1996; Goldstein & Thomas, 1996) is studied, the risk of a limited vision of school reality continues to exist. The socio-cultural context not only affects pupils’ academic results but also the school culture, teacher expectations, relationships with families and students, the style of leadership needed and the necessary effort to promote programmes of change (St011 & Fink, 1996; Fink & Stoll, 1998). School Evaluation Reveals the School’s Functioning and Guides the Processes of Change Educational change demands permanent assessment of each school’s circumstances. Few teachers doubt the importance and usefnlness of such monitoring. However, many do not trust the way in which assessment is put into practice. The existing tensions in evaluation encourage mistrust. Prioritising accountability and the tension resulting from publication of from obtained results, increase reservations about external evaluation. Making the results public is mostly regarded as being imposed from above, the main aim being that of control and revelation of student results, but it runs the risk of creating an unbalanced image of education. In contrast, internal evaluation avoids these problems and
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
45
is firmly oriented towards school improvement. The problem is that it cannot incorporate any of the positive traits of the external evaluation: its greater objectivity and the ability to offer external points of reference to the schools for better assessment of results. It is difficult to promote educational change without knowing what happens in the specific school. However, it is also difficult to do so when the information gained is neither well received by the educational community nor regarded as an aid towards change. Contextualised and multi-dimensional assessment programmes must be developed, offering relevant information and with sufficient external points of reference to be correctly interpreted by the schools. This type of information is likely to be accepted by the educational community and used as an instrument of change and improvement, contributing to the general debate and over time, revealing the decisions each school adopts to resolve detected inadequacies. Under these conditions, assessment may become a powerful instrument of change. Change is Easier if Undertaken in Several Schools Traditionally, schools have done their jobs on an individual basis, separately from one another. Each school was considered to be self-sufficient and capable of carrying out its functions with its own resources. However, school work have changed and have been extended over the last few decades, which has made it necessary to seek new partnerships which will work together to obtain projected aims. The incorporation of parent associations, volunteers, former students, companies and sponsors has been of considerable help for many schools resulting in an extension of their activities and an improvement in what they offer. The search for new partnerships appears to have led schools to believe in the importance of relating to each other and seeking mutual support. This is no simple task when development and success are defined within a competitive context, leading to initial suspicion and reluctance to participate in joint projects. However, in spite of these difficulties, the communication and support networks between schools and teachers are developing and becoming consolidated (Lieberman & Grolnick, 1998). The ever-increasing use of the Internet is aiding connection between participants and therefore its extension and consolidation. Notwithstanding, it is no simple task for school networks to persist over time because of the difficulty of connecting among experiences, expectations and cultures. Research into how networks operate outlines how certain factors help to reinforce them. The first of these is the existence of a common project offering an interesting alternative to a specific problem. Meaning and cohesion is thus afforded to the organisation of joint activities. The second is mutual trust and conviction that working together will prove beneficial to all participants. Discretion and respect are necessary attitudes for maintaining the network. The third is the existence of a leader to put the network into practice and help resolve difficulties. This leadership may be undertaken by one of the schools or groups of teachers participating in the network, or become the responsibility of an external team which presents an initial proposal which the school adopts. The fourth is willingness and equality between participants. When the school network possesses all these traits the possibilities of continuity and satisfaction among participants are likely to be enhanced.
Schools Should Feel Invohw~ in Educationrrl Reseurc~h The participation of teachers in research referring to their own actions has been considered a powerful strategy of change and an improvement in teaching quality. At the same time it allows the immediate promoters of change to play a leading role. The proposal for action research in schools encourages and supports this orientation. Its obvious advantages should not obscure its weaknesses: wider frames of reference may be lost and it is difficult to know, from this orientation, about the general factors conditioning educational change. Moreover, the effort the teachers have to invest in developing this type of research project and the need for external advice often limits the duration and extent of these initiatives. How can schools be involved in research about how they operate when this research is connected to their own educational activities? The convergence. occurring between research on school effectiveness and school improvement practice (Reynolds, Hopkins, & Stoll, 1993; Stoll & Fink, 1996) may open up opportunities. This process. however, needs to go a step further and incorporate the positive elements involved in action research for schools. The schools must participate in the research design, get to know the results obtained, use them to the extent possible in their processes of change and evaluate the results afterwards. In this way, schools are likely to accept a greater role and involvement in the research taking place and have a higher appreciation of the results obtained since they themselves have taken part in the research. Action research with several participating schools becomes a powerful instrument of knowledge and change. Participation
and Expression Supports Commitment to Change
One of the problems that arise in reform processes is disconnection between the educational authority, i.e., the body responsible for the running of education, and the schools and teachers. The teachers feel that their opinion is hardly ever taken into account by those in charge of education, and even less so by the leaders of national politics. Often they also believe that no internal mechanisms exist in the school to express their opinions and evaluations. This all leads to distancing, to loss of a sense of change and to a feeling that the decisions adopted by those responsible for education are not in touch with the real problems in the schools. There is therefore a need for channels of communication and information guaranteeing an easy-going relationship between the education authority and the teachers, and helping to overcome any distance existing between them. The schools and teachers would thus feel more like participants and prime movers in the process of change.
The Aims and Activities of the Evaluation Network The five factors that have just been outlined appear to be valid focal points for achieving more continuous change processes with greater guarantees of success. The School Evaluation Network which has been developed over the last five years in Spain
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
47
integrates these five factors into a common strategy. It is a network which involves multidimensional and external assessment of schools, providing them with contextualised, comparable and confidential information about how they work. Its single aim is that of promoting the processes of change and improvement, for the purpose of internal evaluation so that constructive dialogue may exist internally and externally (Nevo, 2001). This aim also includes facilitating connection and exchange of experiences and initiatives between the schools and to analyse the results obtained so that the schools become aware of the factors associated with good results. Since the Network is separate from the government, proposals come directly from the opinions collected in the participating schools. The progressive increase in the number of participating schools - 30 for the 1997-98 academic year and around 150 for the 200-2001 academic year - and the greater involvement of some government bodies and local authorities has promoted the achievement of these aims over this last academic year. An Evaluation Network for Schools Becoming Aware of their Educational Situation The school evaluation network is the product of the voluntary participation of secondary schools. The same tests and questionnaires are applied to each school. The schools receive two complete reports with details of the results obtained, the first in the month of November and the second in the month of June. The information collected has several characteristics: Contextualised. The initial performance of students is controlled when they begin secondary education at the age of 12 as is the socio-economic level of each school. Each school is put into one of the four contexts which were defined (high, medium-high, medium-low and low) from a questionnaire completed by the students about the profession of their parents and their employment situation, data concerning where they live, the books in their house, reading newspapers, etc. Broad bused. The information obtained does not only refer to students’ academic results but also to their attitudes and learning strategies, to school and classroom processes and to parents, teachers and students’ opinion of the school (see Tablel). It is thus possible to put forward the, not always identical, points of view of different sectors of the educational community. Comparable. Each school receives the results comparison with the average of results obtained and all the schools participating in the network Figures 1 and 2 illustrate how the information mathematics test and to school procedures.
obtained in all dimensions studied in by the schools in the same social context to aid better comparative interpretation. is supplied to a school referring to the
Table 1: Model ofthe Four Levels for School Evaluation . . =
Entrance level Contexl level
Initial student results Socio-cultural context Size of school
. School processes m Classroom processes
Procedure level
1 1 . . 9
Curriculum areas: mathematics, language, social sciences,natural sciences, technology Attitudes and values Learning strategies Metacognitive skills School assessment
0 Parents
9
School assessment
0
.
School assessment
0 Students Result level
Teachers
Confidential. Only the school itself has access to the information concerning it. Schools are informed about the number of schools which are in its own socio-cultural context and the total number of participant schools. Objective. The information supplied comes from the questionnaires and standardised tests applied in the school, all of which have previously been analysed and have reached a high level of reliability and validity. Interpreted by the khools aAd teachers themselves. The information analysed and assessed by each school. An external consultant supports this task. The schools themselves are in a better position to comprehend and to explain them since they are aware of their history, culture, the experienced and the decisions they have adopted.
supplied should be them in undertaking the results obtained problems they have
Formative. The role of evaluation is to work together with the schools so that the latter can become more aware of themselves and may design strategies of change. Control or supervision are provided by other educational agents. Continuous. The proposal made to the schools is that they continue in the Network for at least four consecutive years. This enables a continuous follow-up of the same students. It is possible to analyse change measuring the same subjects at least three times and studying the added value of each school. Elucidating extent that information dimensions
the factors it is applied concerning constituting
that influence the quality of learning. Finally, assessment, to the to a representative number of schools, may provide more scientific the influence of the assessment relative to the different levels and it.
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
49
43-56
Figure 1: Example of a Graph Providing a School with the Results Obtained by its Pupils in the First Year of Secondary School Mathematics Test
100 90
1
80 70 60 .50 40 :30 20 IO 0 HEAD
PARTICIPATION
STUDENT EXPECTATIONS RELATION
5 SCHOOL
0 CONTEXT
WORKING DEPARTMENTS AWARENESS RELATIONS 4
WORKING CONDITIONS
n TOTAL
Figure 2: Example of a Graph Supplied to a School About its Educational Processes
The aim ofco-operating with each school to become familiar with and improve the way in which it functions is completed with research into factors more directly responsible for school efficacy. Participation in the assessment network of a large number of‘ schools, the application of standardised tests’ common to all of them and the follow-up of the same groups of students over time means that statistical models of several levels may be applied and the influence of the different variables in obtained results may be researched. Use of qualitative analysis, at the same time, on both school and classroom procedures completes the information obtained through the quantitative study. From the second year onwards, and at least once, two repeated measurements are available about the same students allowing analysis of the data obtained. The conclusions are published in specialised journals (Marchesi, Martin, Martinez Arias. Tiana, & Moreno, 1999) but are also sent in a simplified and summarised version to the participating schools so they may become aware of the results of the research in which they are participating. In this way the school may compare general conclusions with their reality and with the details of their own assessment and distinguish the influence of the factors which depend on their responsibility with those others which are more stable and more structural, such as the socio-cultural context of the school and its size. This process of analysis helps to create a dynamic situation of reflection among the teachers and encourages the development of more essential and realistic programmes of change. We shall provide two examples of results obtained. The first refers to the most significant predictable variables regarding student performance. The second to the interaction between the socio-cultural context and school size. 1. The three main factors influencing students’ academic performance are previous performance, metacognitive skills and socio-cultural context. The best foresight into the academic performance of students in mathematics, language and social sciences is previous performance in these same areas, which emphasises the importance of controlling this variable when school quality is being studied (see Table 2). The absence of this control and the direct analysis of students’ final results offers a distorted image of the effort that each school makes to improve student learning. This fact reinforces the importance of added value models taking repeated measurements of the results of the same students. Occasionally and when faced with the absence of the initial student results, measurements of their socio-cultural context have been used to control their influence and offer more balanced comparisons from the obtained results. Data obtained indicate that this method is better than comparing direct scores but is clearly insufficient The influence of students’ metacognitive skills in the results obtained in the three curricular areas, particularly the ability to verify results and metacomprehension, appears to underline the importance of these skills in learning. In the light of these data, metacognitive development should not be considered a marginal activity in the learning process but should be one of its priority focal points. The explicit incorporation of this aim in the learning of different curricular areas contributes to the improvement of learning. There is also a positive relationship between student awareness that they have extensive command of learning strategies and the results of their learning in the three areas studied.
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
51
43-56
In contrast, the negative relationship between the use of rote learning as a strategy and the results obtained, appears to demonstrate the ineffectiveness of this strategy and the need to guide the students towards those activities related to effective learning. Table 2: Results of Multiple Linear Regression Analysis (stepwise) in Mathematics, Language and Social Sciences for 2nd year Secondary Students Subject
Predictive
2” ES0
variables
Standarized
weights
Mathematics
N= 1184 F ,y117~,=171,46 *** R= 0,7l I R’adjus+ed=0,502 Std. errot=
9,64
Language N = 1205 F~y,,<~8=167,58 *** R = 0.676 R? ad,psted= 0,454 Std. error = 9,26
Mathematics 1” Verification of results Sociocultural context Awareness of strategies Awareness of knowledge Metacomprehension Rote learning Gender
0,491 *** 0,221*** 0,090*** 0,073*** 0,081***
Language lo Verification of results
0,469*** 0,140***
Rote learning
-0,127***
Metacomprehension
Awareness of knowledge
0,082*** 0,078*** .0,065**
Social sciences
1’
0,391***
of results
0,157***
Awarenessof strategies
0,083*** -0,054* 0,048*
Social sciences N = 1094 F,> ,(,ab=l02,69 R=0,631 R’
***
0,394 Std. error = 12,33 adJWed=
Verification
Metacomprehension
b,145***
Sociocultural
context
0,l lo***
Awareness
of strategies
0,112***
Awareness
of knowledge
0,098***
Note:*:p<.05;**:p<.Ol;***:p<.OOl.
2. There is a clear interaction between the socio-cultural context of the school and its size in relation to the assessment of students about the order of the school, their teachers and tutors (see Figure 3). This same interaction is found in parent assessment. The parents of small schools in the lowest socio-cultural levels give higher scores to the tinctioning of the school, attention received, participation, order, school activities and opinion about the school (see Figure 4).
Context context 1
E 8 I
m
context 2
&jgJ
context 3
50
contea 4 Teachers L
Order Large
Tutors L
Order Small
Teachers S
S. = small
schools
Tubis s
L. = large schools
Figure 3: Student Satisfaction According to Context and School Size
Context 60
Context 1 m
Context 2 Context 3 contenl4
Attention Large
I‘articipation
Opinion L. Opinion S.
Attention Small S. = small
schools
L = large
L. Participation S.
schools
Figure 4: Parental Satisfaction According to Context and School Size
A. Marchesi
et al. /Studies
in Educational
Evaluation
An Evaluation Network for Exchanging Information
29 (2003)
43-56
53
and Improvement
The main aim of the Network is to link the schools in a joint assessment project so that they receive relevant information which allows them to become aware of their situation and promote processes of change. Assessment, as already mentioned, has a truly formative end. The results students obtain in each of the curricular areas and in each of its dimensions constitute relevant information for teachers of the departments to exchange information and analyse the reasons for the data obtained. The assessment of teachers themselves on how the school functions, as well as parents’ and students’ assessments contribute complementary, sometimes contradictory, information which demands detailed analysis by the school staff. The head teacher has to co-ordinate and promote the process of reflection plus the most appropriate initiatives for resolving problems detected. School assessment in the following years ensures that the teachers and the educational community check the consequences of the initiatives adopted. Over the three years of the Network’s existence, some head teachers and teachers took an interest in establishing contact with other participating schools to exchange experiences and become familiar with the proposals or methods which would have had success in other places. Quite a few teachers considered that the discussion with colleagues participating in the Network could be a good training instrument. Meetings with head teachers have consequently been organized to discuss the main results, the appropiate strategies for change and the conditions which support or hinder it.
An Evaluation Network to Collect Teachers, Parents and Students’ Opinions As has been previously stated, there is a risk that in the course of processes of educational change, the opinion of teachers, parents and students is hardly ever taken into account. When teachers, by contrast, are certain that their opinions are communicated to the educational authorities and to public opinion, they feel glad to be participating and they feel,.acknowledged. The evaluation Network also tries to channel teachers’, parents’ and students’ opinions over subjects of general interest. This channelling is three-dimensional: to participating schools, sending them the summary of their assessments as compared with the general trends of the other schools; to educational authorities, and to public opinion. The procedure used is that of incorporating some specific questions into the questionnaires every year which the teachers, students or parents must complete and which respond to existing social concerns. Order and discipline in schools, school failure or inmigrants in schools are some of the topics studied. All those completing the questionnaires are previously informed that their answers will be communicated to those in charge of education and also to the public. The relevance of the chosen subjects ensures that the media pay special attention to the information that is given to them. The participating schools therefore strengthen their sense of belonging to the same net and feel that they are having a greater effect in the process of educational change.
54
A. Marchesi et al. /Studies in Educational Evaluation 2Y (200.7) 43-56
Note For the awareness tests the psychometric model of the Theory of item response with 3 parameters was used. In all cases the single dimensional nature was examined through the factorial analysis of tetracoric correlation nuances, using the TESTFACT programme (Wilson, Wood, Kandola, & Gibbons, 1991). In all tests a primary factor with an associated value at least five times larger than the second was found which is why they are considered to be of a single dimensional nature and appropriate for the proposed model of item Response Theory, according to Lord’s (1980) criteria. This psychometric model permits vertical comparison of scores obtained in tests of different school levels. To obtain this comparison, a body of common items was used for anchorage between the tests of the first year and second year and those of the second and fourth year. Analysis of the items and their adjustment to the model was made using the BILOG 3 programme (Mislevy & Bock, 1990), eliminating those items with poor adjustments from the tests. A Likert type format was used for construction of the questionnaires, with seven or five levels, and factorial analysis of the items were made through analysis of the main components with varimax rotation.
Acknowledgement Leoncio FenBndez, Isabel Monguilot, Vicente Rivikre, Enrique Rota, Eva Ma Ptrez and Ricardo Lucena collaborated in the development of the School Evaluation Network
References Fink, D., & Stall, L. (1998). Educarional change: Easier sai$ than done. In A. Hargreaves, 6. Lieberman, M. Fullan, & D. Hopkins (Eds.), International handbook ofeducational change. Dordrecht, the Netherlands: Kluwer. Goldstein, H., & Myers, K (1996). performance indicators. Research Intelligence: 12-16.
Freedom of information: Towards a code of ethic for British Educational Research Association Newsletter, 57,
Goldstein, H ., & Thomas, S. (1996). Using examination results as indicators of school and college performance. Journal of the Royal Statistical Society, 159, 149-l 63. Joyce, B. (1991). The doors to school improvement. Educational Leadership, 48 (S), 59-62 Lieberman, A., & Grolnick, M. (1998). Educational reform network: Changes in the forms of reform. In A. Hargreaves, A. Lieberman, M.Fullan, & D. Hopkins (Eds.), International handbook of educational change. Dordrecht, the Netherlands: Kluwer. Lord, F.M. (1980). Applications
of
item response theory to practical
testingproblems.
Hillsdale, NJ:
Erlbaum. Marchesi, A. (2000). Controversius en la educacidn espafiolu [Controversies in Spanish education]. Madrid: Alianza.
A. Marchesi Marchesi, centros
de educacidn
mo&rels.
Mislevy, Chicago:
Hayton
(Ed.),
et al. /Studies
in Educational
A., Martin,
E., Martinez
Arias,
secundaria
[A secondary
school assessment
R., Tiana,
R.J., & Bock, D.R. (1990). BILOG. SSI-Scientific Software International.
Mortimore,
P., & Whitty,
Dissafection
Nevo,
& Social
D. (2001).
School
G. (1999).
School
Exclusion.
London:
evaluation:
internal
Evaluation
29 (2003)
A. , & Moreno, network]. Item
analysis
Una red de evaluation
Aprendizaje,
and
improvement: Kogan
A. (1999).
Infanciay
55
43-56
test
A remedy
scoring
for
de
85,59-73
with
social
binary
logistic
exclusion?
In A.
Page.
or external?
Studies
in Educational
Evaluation.
27, 9%
106
Reynolds, improvement
D., Hopkins,
practice:
Sammons, Instrtute
Towards P., Hillman,
of Education
D. , & Stall, a synergy.
L. (I 993).
School
J., & Mortimore,
P. (1995).
school
effectiveness
and School Key
characteristics
4 (I),
and school 37-58.
of effective schools. London:
D. (I 996). Changing
P., Owen, performance
our schools.
Ch. and Pennell, H. (I 994). Assessing in context. London: Institute of Education.
Buckingham,
UK: Open University
Teddlie, Ch., & Reynolds, D. (Eds.) (2000). The international handbook research. London: Falmer. Teddlie, Ch., Stringfield, S., & Reynolds, D. (2000). school effectiveness research. In Ch. Teddlie & D. Reynolds (Eds.), The international effectiveness research. London: Falmer.
statistics,
knowledge
Improvement,
& Ofsted.
Sammons. P., Thomas, 5., Mortimore, effectiveness. Developing measures to put school Stall, L. & Fink,
Linking
Effectiveness
Wilson, D.T., Wood, R., Kandola-Downs, P., & Gibbons, R. (1991). and item,factor analysis. Chicago: SSI-Scientific Software.
school
Press.
of school
TESTFACT.
effectiveness Context issues within handbook ofschool
Test scoring,
item
The Authors ALVARO MARCHES1 is the director of Department of Developmental and Educational Psychology in the Complutense University of Madrid, Spain. He has carried our research in inclusive school and school evalaution. He worked at the Ministry of Education from 1986 to 1996 as General Director and Secretary of State for Education coordinating the Spanish educational reform. ELENA MARTIN is a professor at the Universidad Autonoma de Madrid, Spain. She worked at the Ministry of Education from 1986 to 1996 coordinating the design of a new curriculum. Her research area is related to school evaluation and curriculum. In the last few years she has been carrying out research on secondary school evaluation and incidence and intervention on bullying. ROSARIO University
MARTINEZ ARIAS is full professor of Psychometrics at the Complutense of Madrid in the department of Methodology of Behavioral Sciences. Her
56
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
research lines are educational measurement, psychological assessment, large scale assessment, program evaluation, applied multivariate analysis, and risk perception ALEJANDRO TIANA is a professor of Education and vice-rector for Innovation and Evaluation at the Spanish National Distance Teaching University (UNED). He currently is chair of IEA and has been director of the Spanish National Institute for Quality and Evaluation (INCE). AMPARO MORENO is a professor at the Universidad Autonoma de Madrid, Spain. Her research area is related to Cognitive Development. During the last few years she has been carrying out research on secondary students’ metacognitive development. She is also devoted to study the relationships between cognitive and linguistic development in infancy. Correspondence:
Studies in Educational
Evaluation
29 (2003)
43-56 www.elsevier.com/stueduc
AN EVALUATION
NETWORK
Alvaro Marchesi*, Alejandro
l
***Faculty
FOR EDUCATIONAL
CHANGE
Elena Martin**, Rosario Martinez Arias*, Tiana***, and Amparo Moreno**
*Faculty of Psychology, Complutense University of Madrid, Spain * Faculty of Psychology, Universidad Autdnoma de Madrid, Spain of EducatioqSpanish National Distance Teaching University, Madrid,
Spain
It is through the development of a strategy capable of encompassing the most relevant conditions of educational change that such change is best supported. This article describes the experience of an evaluation network being put into practice in Spain to encourage this type of change. First, the most relevant conditions needed for educational change are briefly outlined. Second, the experience of a school evaluation network which includes all the previously described conditions is more extensively developed. Conditions for Educational Change Innovations that adopt a multi-dimensional perspective have a higher chance of success. Moreover, experience reveals that educational changes must be adapted to the reality of the specific school, and therefore general reforms and assessments have to be sensitive to existing differences at the same time. Nevertheless, teachers perceive a great distance between the reform proposals from those in charge of education and their opportunities for putting them into practice. They also feel that their opinions are hardly ever taken into account, and this affects their active commitment to these changes. The same is true for educational research: Results are often considered to be far removed from the specific educational reality and with few suggestions to help teachers reorganise their teaching practices. The teacher, who has little confidence in external support systems and few available options for professional development, is limited to carrying out his or her work in the school and classroom environment without ihe opportunity to assess the possibilities of new forms of co-operation. 0191-491X/03/$ - see front PII: SOlSl-491X(03)00006-3
matter
0 2003
Published
by Elsevier
Science
Ltd.
44
A. Mc~rche.si
rt ul. /Studies
in Educational
Evaluation
29 (2OO.r)
4.jL.76
These reflections underline the importance of the following five conditions needed in order to develop good educational innovation: (1) a multi-dimensional and contextual outlook; (2) continuous assessment of how each individual school is being run; (3) inter-relations between schools: (4) connections between research and the reality of schools, and (5) channels for teachers to express their opinions. In the following pages each one of these is briefly analysed. Subsequently and more extensively, a “door to change” (Joyce, 1991) is described, i.e., the School Assessment Network, which incorporates the previously mentioned factors and which attempts to develop new prospects for educational change. Educational Change Must Adopt a Multi-Dimensional
and Contextual Outlook
Effective and lasting changes are not usually the result of partial and isolated initiatives. They demand global, systemic, interactive and contextualised approaches. Research on effective schools (Sammons, Hillman, & Mortimore, 1995, Teddlie, Stringfield, & Reynolds, 2000) has underlined the importance of a multi-dimensional outlook for understanding the reality of schools. Such an approach must be capable of analysing the school’s procedures, the impact and recognition of the head teacher’s job, coordination between teachers, classroom teaching, students’ educational progress, and parents, teachers and students’ assessment of how their school operates. Another dimension, however, which profoundly conditions educational action, is the socio-cultural context of the school. Its influence on students’ academic results has been extensively outlined (Sammons, Thomas, Mortimore, Owen, & Pennell, 1994, Teddlie, et al., 2000). As was directly and impressively shown by Mortimore and Whitty (1999), the single most important factor distinguishing the most academically successful schools is that only a few of their students come from socially disadvantaged homes. Consequently, school assessment which does not take into account the prior attainment of students and their background factors offers information which bears no justice to the reality of the particular school. Even if these factors are taken into account and the “value added” by the school (Goldstein & Myers, 1996; Goldstein & Thomas, 1996) is studied, the risk of a limited vision of school reality continues to exist. The socio-cultural context not only affects pupils’ academic results but also the school culture, teacher expectations, relationships with families and students, the style of leadership needed and the necessary effort to promote programmes of change (St011 & Fink, 1996; Fink & Stoll, 1998). School Evaluation Reveals the School’s Functioning and Guides the Processes of Change Educational change demands permanent assessment of each school’s circumstances. Few teachers doubt the importance and usefnlness of such monitoring. However, many do not trust the way in which assessment is put into practice. The existing tensions in evaluation encourage mistrust. Prioritising accountability and the tension resulting from publication of from obtained results, increase reservations about external evaluation. Making the results public is mostly regarded as being imposed from above, the main aim being that of control and revelation of student results, but it runs the risk of creating an unbalanced image of education. In contrast, internal evaluation avoids these problems and
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
45
is firmly oriented towards school improvement. The problem is that it cannot incorporate any of the positive traits of the external evaluation: its greater objectivity and the ability to offer external points of reference to the schools for better assessment of results. It is difficult to promote educational change without knowing what happens in the specific school. However, it is also difficult to do so when the information gained is neither well received by the educational community nor regarded as an aid towards change. Contextualised and multi-dimensional assessment programmes must be developed, offering relevant information and with sufficient external points of reference to be correctly interpreted by the schools. This type of information is likely to be accepted by the educational community and used as an instrument of change and improvement, contributing to the general debate and over time, revealing the decisions each school adopts to resolve detected inadequacies. Under these conditions, assessment may become a powerful instrument of change. Change is Easier if Undertaken in Several Schools Traditionally, schools have done their jobs on an individual basis, separately from one another. Each school was considered to be self-sufficient and capable of carrying out its functions with its own resources. However, school work have changed and have been extended over the last few decades, which has made it necessary to seek new partnerships which will work together to obtain projected aims. The incorporation of parent associations, volunteers, former students, companies and sponsors has been of considerable help for many schools resulting in an extension of their activities and an improvement in what they offer. The search for new partnerships appears to have led schools to believe in the importance of relating to each other and seeking mutual support. This is no simple task when development and success are defined within a competitive context, leading to initial suspicion and reluctance to participate in joint projects. However, in spite of these difficulties, the communication and support networks between schools and teachers are developing and becoming consolidated (Lieberman & Grolnick, 1998). The ever-increasing use of the Internet is aiding connection between participants and therefore its extension and consolidation. Notwithstanding, it is no simple task for school networks to persist over time because of the difficulty of connecting among experiences, expectations and cultures. Research into how networks operate outlines how certain factors help to reinforce them. The first of these is the existence of a common project offering an interesting alternative to a specific problem. Meaning and cohesion is thus afforded to the organisation of joint activities. The second is mutual trust and conviction that working together will prove beneficial to all participants. Discretion and respect are necessary attitudes for maintaining the network. The third is the existence of a leader to put the network into practice and help resolve difficulties. This leadership may be undertaken by one of the schools or groups of teachers participating in the network, or become the responsibility of an external team which presents an initial proposal which the school adopts. The fourth is willingness and equality between participants. When the school network possesses all these traits the possibilities of continuity and satisfaction among participants are likely to be enhanced.
Schools Should Feel Invohw~ in Educationrrl Reseurc~h The participation of teachers in research referring to their own actions has been considered a powerful strategy of change and an improvement in teaching quality. At the same time it allows the immediate promoters of change to play a leading role. The proposal for action research in schools encourages and supports this orientation. Its obvious advantages should not obscure its weaknesses: wider frames of reference may be lost and it is difficult to know, from this orientation, about the general factors conditioning educational change. Moreover, the effort the teachers have to invest in developing this type of research project and the need for external advice often limits the duration and extent of these initiatives. How can schools be involved in research about how they operate when this research is connected to their own educational activities? The convergence. occurring between research on school effectiveness and school improvement practice (Reynolds, Hopkins, & Stoll, 1993; Stoll & Fink, 1996) may open up opportunities. This process. however, needs to go a step further and incorporate the positive elements involved in action research for schools. The schools must participate in the research design, get to know the results obtained, use them to the extent possible in their processes of change and evaluate the results afterwards. In this way, schools are likely to accept a greater role and involvement in the research taking place and have a higher appreciation of the results obtained since they themselves have taken part in the research. Action research with several participating schools becomes a powerful instrument of knowledge and change. Participation
and Expression Supports Commitment to Change
One of the problems that arise in reform processes is disconnection between the educational authority, i.e., the body responsible for the running of education, and the schools and teachers. The teachers feel that their opinion is hardly ever taken into account by those in charge of education, and even less so by the leaders of national politics. Often they also believe that no internal mechanisms exist in the school to express their opinions and evaluations. This all leads to distancing, to loss of a sense of change and to a feeling that the decisions adopted by those responsible for education are not in touch with the real problems in the schools. There is therefore a need for channels of communication and information guaranteeing an easy-going relationship between the education authority and the teachers, and helping to overcome any distance existing between them. The schools and teachers would thus feel more like participants and prime movers in the process of change.
The Aims and Activities of the Evaluation Network The five factors that have just been outlined appear to be valid focal points for achieving more continuous change processes with greater guarantees of success. The School Evaluation Network which has been developed over the last five years in Spain
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
47
integrates these five factors into a common strategy. It is a network which involves multidimensional and external assessment of schools, providing them with contextualised, comparable and confidential information about how they work. Its single aim is that of promoting the processes of change and improvement, for the purpose of internal evaluation so that constructive dialogue may exist internally and externally (Nevo, 2001). This aim also includes facilitating connection and exchange of experiences and initiatives between the schools and to analyse the results obtained so that the schools become aware of the factors associated with good results. Since the Network is separate from the government, proposals come directly from the opinions collected in the participating schools. The progressive increase in the number of participating schools - 30 for the 1997-98 academic year and around 150 for the 200-2001 academic year - and the greater involvement of some government bodies and local authorities has promoted the achievement of these aims over this last academic year. An Evaluation Network for Schools Becoming Aware of their Educational Situation The school evaluation network is the product of the voluntary participation of secondary schools. The same tests and questionnaires are applied to each school. The schools receive two complete reports with details of the results obtained, the first in the month of November and the second in the month of June. The information collected has several characteristics: Contextualised. The initial performance of students is controlled when they begin secondary education at the age of 12 as is the socio-economic level of each school. Each school is put into one of the four contexts which were defined (high, medium-high, medium-low and low) from a questionnaire completed by the students about the profession of their parents and their employment situation, data concerning where they live, the books in their house, reading newspapers, etc. Broad bused. The information obtained does not only refer to students’ academic results but also to their attitudes and learning strategies, to school and classroom processes and to parents, teachers and students’ opinion of the school (see Tablel). It is thus possible to put forward the, not always identical, points of view of different sectors of the educational community. Comparable. Each school receives the results comparison with the average of results obtained and all the schools participating in the network Figures 1 and 2 illustrate how the information mathematics test and to school procedures.
obtained in all dimensions studied in by the schools in the same social context to aid better comparative interpretation. is supplied to a school referring to the
Table 1: Model ofthe Four Levels for School Evaluation . . =
Entrance level Contexl level
Initial student results Socio-cultural context Size of school
. School processes m Classroom processes
Procedure level
1 1 . . 9
Curriculum areas: mathematics, language, social sciences,natural sciences, technology Attitudes and values Learning strategies Metacognitive skills School assessment
0 Parents
9
School assessment
0
.
School assessment
0 Students Result level
Teachers
Confidential. Only the school itself has access to the information concerning it. Schools are informed about the number of schools which are in its own socio-cultural context and the total number of participant schools. Objective. The information supplied comes from the questionnaires and standardised tests applied in the school, all of which have previously been analysed and have reached a high level of reliability and validity. Interpreted by the khools aAd teachers themselves. The information analysed and assessed by each school. An external consultant supports this task. The schools themselves are in a better position to comprehend and to explain them since they are aware of their history, culture, the experienced and the decisions they have adopted.
supplied should be them in undertaking the results obtained problems they have
Formative. The role of evaluation is to work together with the schools so that the latter can become more aware of themselves and may design strategies of change. Control or supervision are provided by other educational agents. Continuous. The proposal made to the schools is that they continue in the Network for at least four consecutive years. This enables a continuous follow-up of the same students. It is possible to analyse change measuring the same subjects at least three times and studying the added value of each school. Elucidating extent that information dimensions
the factors it is applied concerning constituting
that influence the quality of learning. Finally, assessment, to the to a representative number of schools, may provide more scientific the influence of the assessment relative to the different levels and it.
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
49
43-56
Figure 1: Example of a Graph Providing a School with the Results Obtained by its Pupils in the First Year of Secondary School Mathematics Test
100 90
1
80 70 60 .50 40 :30 20 IO 0 HEAD
PARTICIPATION
STUDENT EXPECTATIONS RELATION
5 SCHOOL
0 CONTEXT
WORKING DEPARTMENTS AWARENESS RELATIONS 4
WORKING CONDITIONS
n TOTAL
Figure 2: Example of a Graph Supplied to a School About its Educational Processes
The aim ofco-operating with each school to become familiar with and improve the way in which it functions is completed with research into factors more directly responsible for school efficacy. Participation in the assessment network of a large number of‘ schools, the application of standardised tests’ common to all of them and the follow-up of the same groups of students over time means that statistical models of several levels may be applied and the influence of the different variables in obtained results may be researched. Use of qualitative analysis, at the same time, on both school and classroom procedures completes the information obtained through the quantitative study. From the second year onwards, and at least once, two repeated measurements are available about the same students allowing analysis of the data obtained. The conclusions are published in specialised journals (Marchesi, Martin, Martinez Arias. Tiana, & Moreno, 1999) but are also sent in a simplified and summarised version to the participating schools so they may become aware of the results of the research in which they are participating. In this way the school may compare general conclusions with their reality and with the details of their own assessment and distinguish the influence of the factors which depend on their responsibility with those others which are more stable and more structural, such as the socio-cultural context of the school and its size. This process of analysis helps to create a dynamic situation of reflection among the teachers and encourages the development of more essential and realistic programmes of change. We shall provide two examples of results obtained. The first refers to the most significant predictable variables regarding student performance. The second to the interaction between the socio-cultural context and school size. 1. The three main factors influencing students’ academic performance are previous performance, metacognitive skills and socio-cultural context. The best foresight into the academic performance of students in mathematics, language and social sciences is previous performance in these same areas, which emphasises the importance of controlling this variable when school quality is being studied (see Table 2). The absence of this control and the direct analysis of students’ final results offers a distorted image of the effort that each school makes to improve student learning. This fact reinforces the importance of added value models taking repeated measurements of the results of the same students. Occasionally and when faced with the absence of the initial student results, measurements of their socio-cultural context have been used to control their influence and offer more balanced comparisons from the obtained results. Data obtained indicate that this method is better than comparing direct scores but is clearly insufficient The influence of students’ metacognitive skills in the results obtained in the three curricular areas, particularly the ability to verify results and metacomprehension, appears to underline the importance of these skills in learning. In the light of these data, metacognitive development should not be considered a marginal activity in the learning process but should be one of its priority focal points. The explicit incorporation of this aim in the learning of different curricular areas contributes to the improvement of learning. There is also a positive relationship between student awareness that they have extensive command of learning strategies and the results of their learning in the three areas studied.
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
51
43-56
In contrast, the negative relationship between the use of rote learning as a strategy and the results obtained, appears to demonstrate the ineffectiveness of this strategy and the need to guide the students towards those activities related to effective learning. Table 2: Results of Multiple Linear Regression Analysis (stepwise) in Mathematics, Language and Social Sciences for 2nd year Secondary Students Subject
Predictive
2” ES0
variables
Standarized
weights
Mathematics
N= 1184 F ,y117~,=171,46 *** R= 0,7l I R’adjus+ed=0,502 Std. errot=
9,64
Language N = 1205 F~y,,<~8=167,58 *** R = 0.676 R? ad,psted= 0,454 Std. error = 9,26
Mathematics 1” Verification of results Sociocultural context Awareness of strategies Awareness of knowledge Metacomprehension Rote learning Gender
0,491 *** 0,221*** 0,090*** 0,073*** 0,081***
Language lo Verification of results
0,469*** 0,140***
Rote learning
-0,127***
Metacomprehension
Awareness of knowledge
0,082*** 0,078*** .0,065**
Social sciences
1’
0,391***
of results
0,157***
Awarenessof strategies
0,083*** -0,054* 0,048*
Social sciences N = 1094 F,> ,(,ab=l02,69 R=0,631 R’
***
0,394 Std. error = 12,33 adJWed=
Verification
Metacomprehension
b,145***
Sociocultural
context
0,l lo***
Awareness
of strategies
0,112***
Awareness
of knowledge
0,098***
Note:*:p<.05;**:p<.Ol;***:p<.OOl.
2. There is a clear interaction between the socio-cultural context of the school and its size in relation to the assessment of students about the order of the school, their teachers and tutors (see Figure 3). This same interaction is found in parent assessment. The parents of small schools in the lowest socio-cultural levels give higher scores to the tinctioning of the school, attention received, participation, order, school activities and opinion about the school (see Figure 4).
Context context 1
E 8 I
m
context 2
&jgJ
context 3
50
contea 4 Teachers L
Order Large
Tutors L
Order Small
Teachers S
S. = small
schools
Tubis s
L. = large schools
Figure 3: Student Satisfaction According to Context and School Size
Context 60
Context 1 m
Context 2 Context 3 contenl4
Attention Large
I‘articipation
Opinion L. Opinion S.
Attention Small S. = small
schools
L = large
L. Participation S.
schools
Figure 4: Parental Satisfaction According to Context and School Size
A. Marchesi
et al. /Studies
in Educational
Evaluation
An Evaluation Network for Exchanging Information
29 (2003)
43-56
53
and Improvement
The main aim of the Network is to link the schools in a joint assessment project so that they receive relevant information which allows them to become aware of their situation and promote processes of change. Assessment, as already mentioned, has a truly formative end. The results students obtain in each of the curricular areas and in each of its dimensions constitute relevant information for teachers of the departments to exchange information and analyse the reasons for the data obtained. The assessment of teachers themselves on how the school functions, as well as parents’ and students’ assessments contribute complementary, sometimes contradictory, information which demands detailed analysis by the school staff. The head teacher has to co-ordinate and promote the process of reflection plus the most appropriate initiatives for resolving problems detected. School assessment in the following years ensures that the teachers and the educational community check the consequences of the initiatives adopted. Over the three years of the Network’s existence, some head teachers and teachers took an interest in establishing contact with other participating schools to exchange experiences and become familiar with the proposals or methods which would have had success in other places. Quite a few teachers considered that the discussion with colleagues participating in the Network could be a good training instrument. Meetings with head teachers have consequently been organized to discuss the main results, the appropiate strategies for change and the conditions which support or hinder it.
An Evaluation Network to Collect Teachers, Parents and Students’ Opinions As has been previously stated, there is a risk that in the course of processes of educational change, the opinion of teachers, parents and students is hardly ever taken into account. When teachers, by contrast, are certain that their opinions are communicated to the educational authorities and to public opinion, they feel glad to be participating and they feel,.acknowledged. The evaluation Network also tries to channel teachers’, parents’ and students’ opinions over subjects of general interest. This channelling is three-dimensional: to participating schools, sending them the summary of their assessments as compared with the general trends of the other schools; to educational authorities, and to public opinion. The procedure used is that of incorporating some specific questions into the questionnaires every year which the teachers, students or parents must complete and which respond to existing social concerns. Order and discipline in schools, school failure or inmigrants in schools are some of the topics studied. All those completing the questionnaires are previously informed that their answers will be communicated to those in charge of education and also to the public. The relevance of the chosen subjects ensures that the media pay special attention to the information that is given to them. The participating schools therefore strengthen their sense of belonging to the same net and feel that they are having a greater effect in the process of educational change.
54
A. Marchesi et al. /Studies in Educational Evaluation 2Y (200.7) 43-56
Note For the awareness tests the psychometric model of the Theory of item response with 3 parameters was used. In all cases the single dimensional nature was examined through the factorial analysis of tetracoric correlation nuances, using the TESTFACT programme (Wilson, Wood, Kandola, & Gibbons, 1991). In all tests a primary factor with an associated value at least five times larger than the second was found which is why they are considered to be of a single dimensional nature and appropriate for the proposed model of item Response Theory, according to Lord’s (1980) criteria. This psychometric model permits vertical comparison of scores obtained in tests of different school levels. To obtain this comparison, a body of common items was used for anchorage between the tests of the first year and second year and those of the second and fourth year. Analysis of the items and their adjustment to the model was made using the BILOG 3 programme (Mislevy & Bock, 1990), eliminating those items with poor adjustments from the tests. A Likert type format was used for construction of the questionnaires, with seven or five levels, and factorial analysis of the items were made through analysis of the main components with varimax rotation.
Acknowledgement Leoncio FenBndez, Isabel Monguilot, Vicente Rivikre, Enrique Rota, Eva Ma Ptrez and Ricardo Lucena collaborated in the development of the School Evaluation Network
References Fink, D., & Stall, L. (1998). Educarional change: Easier sai$ than done. In A. Hargreaves, 6. Lieberman, M. Fullan, & D. Hopkins (Eds.), International handbook ofeducational change. Dordrecht, the Netherlands: Kluwer. Goldstein, H., & Myers, K (1996). performance indicators. Research Intelligence: 12-16.
Freedom of information: Towards a code of ethic for British Educational Research Association Newsletter, 57,
Goldstein, H ., & Thomas, S. (1996). Using examination results as indicators of school and college performance. Journal of the Royal Statistical Society, 159, 149-l 63. Joyce, B. (1991). The doors to school improvement. Educational Leadership, 48 (S), 59-62 Lieberman, A., & Grolnick, M. (1998). Educational reform network: Changes in the forms of reform. In A. Hargreaves, A. Lieberman, M.Fullan, & D. Hopkins (Eds.), International handbook of educational change. Dordrecht, the Netherlands: Kluwer. Lord, F.M. (1980). Applications
of
item response theory to practical
testingproblems.
Hillsdale, NJ:
Erlbaum. Marchesi, A. (2000). Controversius en la educacidn espafiolu [Controversies in Spanish education]. Madrid: Alianza.
A. Marchesi Marchesi, centros
de educacidn
mo&rels.
Mislevy, Chicago:
Hayton
(Ed.),
et al. /Studies
in Educational
A., Martin,
E., Martinez
Arias,
secundaria
[A secondary
school assessment
R., Tiana,
R.J., & Bock, D.R. (1990). BILOG. SSI-Scientific Software International.
Mortimore,
P., & Whitty,
Dissafection
Nevo,
& Social
D. (2001).
School
G. (1999).
School
Exclusion.
London:
evaluation:
internal
Evaluation
29 (2003)
A. , & Moreno, network]. Item
analysis
Una red de evaluation
Aprendizaje,
and
improvement: Kogan
A. (1999).
Infanciay
55
43-56
test
A remedy
scoring
for
de
85,59-73
with
social
binary
logistic
exclusion?
In A.
Page.
or external?
Studies
in Educational
Evaluation.
27, 9%
106
Reynolds, improvement
D., Hopkins,
practice:
Sammons, Instrtute
Towards P., Hillman,
of Education
D. , & Stall, a synergy.
L. (I 993).
School
J., & Mortimore,
P. (1995).
school
effectiveness
and School Key
characteristics
4 (I),
and school 37-58.
of effective schools. London:
D. (I 996). Changing
P., Owen, performance
our schools.
Ch. and Pennell, H. (I 994). Assessing in context. London: Institute of Education.
Buckingham,
UK: Open University
Teddlie, Ch., & Reynolds, D. (Eds.) (2000). The international handbook research. London: Falmer. Teddlie, Ch., Stringfield, S., & Reynolds, D. (2000). school effectiveness research. In Ch. Teddlie & D. Reynolds (Eds.), The international effectiveness research. London: Falmer.
statistics,
knowledge
Improvement,
& Ofsted.
Sammons. P., Thomas, 5., Mortimore, effectiveness. Developing measures to put school Stall, L. & Fink,
Linking
Effectiveness
Wilson, D.T., Wood, R., Kandola-Downs, P., & Gibbons, R. (1991). and item,factor analysis. Chicago: SSI-Scientific Software.
school
Press.
of school
TESTFACT.
effectiveness Context issues within handbook ofschool
Test scoring,
item
The Authors ALVARO MARCHES1 is the director of Department of Developmental and Educational Psychology in the Complutense University of Madrid, Spain. He has carried our research in inclusive school and school evalaution. He worked at the Ministry of Education from 1986 to 1996 as General Director and Secretary of State for Education coordinating the Spanish educational reform. ELENA MARTIN is a professor at the Universidad Autonoma de Madrid, Spain. She worked at the Ministry of Education from 1986 to 1996 coordinating the design of a new curriculum. Her research area is related to school evaluation and curriculum. In the last few years she has been carrying out research on secondary school evaluation and incidence and intervention on bullying. ROSARIO University
MARTINEZ ARIAS is full professor of Psychometrics at the Complutense of Madrid in the department of Methodology of Behavioral Sciences. Her
56
A. Marchesi
et al. /Studies
in Educational
Evaluation
29 (2003)
43-56
research lines are educational measurement, psychological assessment, large scale assessment, program evaluation, applied multivariate analysis, and risk perception ALEJANDRO TIANA is a professor of Education and vice-rector for Innovation and Evaluation at the Spanish National Distance Teaching University (UNED). He currently is chair of IEA and has been director of the Spanish National Institute for Quality and Evaluation (INCE). AMPARO MORENO is a professor at the Universidad Autonoma de Madrid, Spain. Her research area is related to Cognitive Development. During the last few years she has been carrying out research on secondary students’ metacognitive development. She is also devoted to study the relationships between cognitive and linguistic development in infancy. Correspondence: