Teachers as evaluators: Results from a systematic literature review

Studies in Educational Evaluation 64 (2020) 100830

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/locate/stueduc

Teachers as evaluators: Results from a systematic literature review a

b,

Amanda McFadden , Kate E. Williams * a b

T

Faculty of Education, Queensland University of Technology, Lady Gowrie QLD, Brisbane, Australia Faculty of Education, Queensland University of Technology, Brisbane, Australia

ARTICLE INFO

ABSTRACT

Keywords: Teacher education Evaluation Teacher research Systematic literature review

Teacher professional standards globally now include a stronger emphasis on teacher research and evaluation skills. Yet, little is known about how to build these capacities through teacher education. This paper reports on a systematic literature review conducted to synthesize the literature about approaches to improving research and evaluative skills and attitudes in teachers. A total of 19 studies were included. The field is largely dominated by small qualitative studies that focus on teacher action research as a facilitator of professional development. There is a substantial and highly problematic gap in the research specifically in relation to building evaluation skills in teachers. Recommendations include a stronger investment in teacher education focussed on evaluation and evaluative thinking, which holds significant potential to positively impact the professional practice of teachers and student learning outcomes.

1. Introduction Internationally, education systems have been focussed for some time on improving student achievement outcomes (Australia. House of Representatives Standing Committee on Education & Vocational Training, 2007), with quality teaching practice the chief mechanism through which this is expected to occur (Gore, Barron, Holmes, & Smith, 2016; Hattie, 2003). Within this focus on teaching practice, what happens at the classroom level has been considered central to student achievement. However, it is increasingly recognized that what teachers do outside of their classroom, particularly through their own research and evaluation of educational approaches, is significant in terms of quality teaching and impact for learners. In recent years there have been several national calls in both the United Kingdom (BERA-RSA, 2014), and Australia (White et al., 2018) for an increased focus on capacity building for the teaching profession in relation to research. In the United States of America, the National Board for Professional Teaching Standards outlines key areas for accomplished teaching including references to teachers being consumers of research, embedding research findings into their teaching practice and conducting and sharing research (National Board for Professional Teaching Standards, 2016). Further to this focus on teacher capacity building around research, the school improvement agenda has been gathering steam internationally (Ainscow, Dyson, Goldrick, & West, 2016; Bryk, 2010;

Hajisoteriou, Karousiou, & Angelides, 2018; Harris et al., 2018). This has brought an increasingly evaluative lens to the school level, while acknowledging the teacher level as a critical factor (Feldhoff, Radisch, & Bischof, 2016). Despite a large and long-standing body of literature on teachers as researchers (Cochran-Smith & Lytle, 1992, 1999; Hatch, 2006; Meier & Henderson, 2007; Stremmel, 2002) and reflective practice to improve teaching approaches (Calderhead, 1989; Freese, 2006; Pollard, 2002; Schon, 1983; Sparks-Langer & Colton, 1991; Walkington, 2005; Zeichner & Liston, 2014), little is known about how teachers specifically employ evaluative capabilities to understand the effectiveness of their teaching and learning programs. Evaluation refers to the systematic investigation of the effectiveness, worth, or merit of a program, practice or policy (Patton, 2008). Evaluation can be thought of as a particular subset of research, which applies the same systematic and rigorous principles across all aspects of research conceptualization and implementation, but with a unique lens that aims ultimately to result in a judgement about the value of a particular approach (Schwandt, 2015). This review seeks to identify and synthesise what is known about approaches to improving research and evaluative skills and attitudes in teachers (both pre-service and in-service). It adds to the recent work of Willegems, Consuegra, Struyven, and Engels (2017) who reviewed research on one key approach to building research capacity in the profession, collaborative teacher research. The current review takes a broad approach including any approach to building research OR evaluative capacity in teachers for two reasons. First, evaluation, as a

⁎ Corresponding author at: School of Early Childhood and Inclusive Education, Faculty of Education, Queensland University of Technology, Victoria Park Road, Kelvin Grove, QLD, 4059, Australia. E-mail address: [email protected] (K.E. Williams).

https://doi.org/10.1016/j.stueduc.2019.100830 Received 28 March 2019; Received in revised form 27 November 2019; Accepted 29 November 2019 0191-491X/ © 2019 Elsevier Ltd. All rights reserved.

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

subset of research, requires research skills as well as those specific to evaluation, which will be further discussed below. Second, there have been no prior broad reviews on approaches to research or evaluation capacity building in teachers. The value of this review exercise is twofold. First, any gaps in the field will become evident and will provide the basis for a future program of research. Second, it can inform the building of a framework of evidence-based principles and practices that will support pre-service curriculum reform, and the development of professional development strategies.

1.2. Evaluation research and teachers While seemingly sounding like research, there is a point of departure between research and evaluation that is nuanced and not always understood. While teacher research is concerned with systematic collection of data to solve a problem or understand a phenomena, evaluation is “the systematic process of delineating, obtaining, reporting, and applying descriptive and judgmental information about some object’s merit, worth, probity, feasibility, safety, significance and/ or equity” (Stufflebeam & Coryn, 2014, p.14). One of the defining aspects of evaluation is the judgement element which requires the identification, application, and communication of a set of defensible values against which the object is to be judged (Stufflebeam & Coryn, 2014). Values will influence the final judgements made, but also what evaluation questions are asked and how constructs are operationalized. For example, is the merit of a new homework policy in a school to be judged on students’ level of engagement, students’ academic achievement, or parental or teacher perspectives on the approach? Is merit even the most important focus of this evaluation or is feasibility for teachers, parents, and students of most interest? Which stakeholders get to contribute to defining these values for this evaluation — school leaders, teachers, parents, and students are likely to have different priorities? The systematic process steps noted in the evaluation definition above include many skills from general research, but also some specific to evaluation. For example, delineating requires communication skills and stakeholder engagement to select focus evaluation questions and design the evaluation. This may include the drafting of a program logic model or theory of change that is able to guide the evaluation design and documents the existing, commonly understood theory underlying the program or practice to be evaluated (Funnell & Rogers, 2011). Obtaining and reporting involves typical technical skills relevant to all research including data collection, analysis, and clear reporting. Evaluations are not confined to one specific research methodology, but should apply the same rigour to design and data collection approaches as is expected in all research. Data collected may not always be from primary sources and might instead include a synthesis of existing research, meaning that high levels of research literacy and skills as a research consumer are required. Reporting for evaluation involves a clear description of activities or approaches and data collected, as well as the synthesis of judgements made through the analysis of the data and in line with the values applied. Applying is an important aspect of evaluation, as the findings should be applied to improve or build on existing practice (Stufflebeam & Coryn, 2014). Along with evaluation actions, there has been recognition of the role of a particular mindset as underpinning evaluation capabilities, recently termed Evaluative Thinking (Buckley, Thomas, Hargraves, & Trochim, 2015; Schwandt, 2015). One definition of Evaluative Thinking is (see Earl & Timperley, 2015 for a more detailed definition):

1.1. Teachers as researchers The concept of the teacher as researcher is now part of the educational landscape. Since the 1980s there has been a heightened focus on teacher research, with teachers being encouraged to be active researchers within their own classrooms (Cochran-Smith & Lytle, 1999). Research within the classroom is defined as systematic data gathering to research a problem and to improve professional practice (Hatch, Greer, & Bailey, 2006). Undertaking research as a teacher uses an insider perspective to improve professional practice through understanding, addressing a research problem and potentially using results in the reconceptualization of teaching approaches and philosophies (Stremmel, 2002). The positive outcomes for learners from teachers researching their own practice have been identified (Somekh & Zeichner, 2009). Despite teachers as researchers being a widely acknowledged part of teachers’ practice, there seems to be a disconnect around embedding research into teachers’ practice (Willegems et al., 2017). Ideas about why there is not traction in the field in this area have included a focus on teacher education programs that do not prepare pre-service teachers with the research literacy needed to identify as teacher researchers (Willegems et al., 2017). Building research literacies at all levels of education to ensure critical engagement with research in the field is key to advancing the profession of teaching (White et al., 2018). Research literacy refers to teachers as “knowledge producers” who can engage with a wide variety of research methods, interpret research findings and draw connections to their everyday teaching practice (BERA-RSA, 2014, p. 40). There have been calls for more synergy across research institutions and the field of education, which will build connections between pre-service and in-service engagement in research (White et al., 2018). Within teacher education preparation programs globally, teacher as researcher units of study are common place and have established success in their implementation (Harrison, Dunn, & Coombe, 2006). Courses on research have been shown to increase feelings of self-efficacy around engaging in research and raising positive attitudes to research for pre-service teachers, however this does not seem to translate to confidence in rating themselves as likely to engage in research as teachers (van der Linden, Bakx, Ros, Beijaard, & Vermeulen, 2012). Preservice teachers who have been engaged in collaborative teacher research in the field and supported through their higher education institution cite benefits for their developing professionalism (Willegems et al., 2017). As an example, a recent desktop review of Australian tertiary education institutions revealed that research units in education courses are embedded across initial teacher education courses. However, of note is that these research units are found in only half of education courses across Australia (see Appendix A). Within these research units, there is inequity across the cohorts who have access to research units within their education degree programs. While most common in early childhood education degree programs, research units are also found in primary education but are limited in secondary education programs. Given the Australian teacher education context typically reflects that of the US and UK, it is likely that a similar situation exists in these countries (Mayer, 2014). Despite calls for more systematic engagement in research, it appears that teacher education courses are not universal in their commitment to building research and evaluation capabilities (Tatto, 2015).

critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action (Buckley et al., 2015, p.4). Key authors agree that evaluation capabilities, including Evaluative Thinking, require specific training (Buckley et al., 2015; Schwandt, 2015; Stufflebeam & Coryn, 2014) reflected in the growing field of evaluative capacity building (ECB; Wandersman, 2014). While evaluation is not a new concept within education (see this journal for example), the terms evaluation and evaluation capacity building, have not been prevalent to date in teacher research literature even though there have been clear indications that evaluative practice is expected of teachers. For example, a UK inquiry argued that teachers should be “equipped to conduct their own research, individually and collectively, to investigate the impact of particular interventions or to explore the positive and negative effects of educational practice” (italics 2

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

2. Method

added to indicate the evaluative focus, BERA-RSA, 2014, p.11). Similarly in Australia, a seminal report on teacher education (Teacher Education Ministerial Advisory Group (TEMAG), 2014) noted “teachers need to analyse and evaluate their impact on learning” (p. 19), and that “assessment should be used to inform decision-making…including… evaluating the effectiveness of teaching practice” (p.20). Further, the report stresses that “innovations and practices that have a demonstrable impact on student learning need to be identified and shared nationally” (p.49). Globally, the adoption of professional standards for teachers has been growing. This growth reflects the interest in high quality teaching and its relationships to economic progress (Call, 2018). In the United Kingdom, the standards advise on demonstrated values, attitudes and practices with evaluation mentioned in the preamble, but only once in reference to evaluating teaching approaches (Department of Education, 2011). In the United States, evaluation is not present in the high level descriptors, however evaluation does permeate the expanded description of the standards, including references to evaluating curriculum materials, pedagogical methods and professional judgment (National Board for Professional Teaching Standards, 2016). To illustrate how evaluation has been embedded within professional teaching standards in Australia, the Australian Professional Standards for Teachers (APST) require graduate teachers to continually evaluate the effectiveness of their teaching approaches and the impact this has for student learning (Australian Institute of Teaching & School Leadership [AITSL], 2012). As graduate teachers move through the professional standards career continuum from graduate to proficient, to highly accomplished, to lead teachers, the demands around research and evaluation increase. A defining difference between the levels of the standards across the career continuum is that lead teachers are ultimately responsible for the evaluation of teaching programs, accountability systems, their colleagues’ professional learning opportunities and the overall effectiveness of their teaching environments (Australian Institute of Teaching & School Leadership [AITSL], 2012). Despite the concept of teacher as a researcher permeating the education field, what is not so well established or understood is how principles of evaluation are taught to both pre-service and in-service teachers. The desktop review of the Australian example (Appendix A), highlights the invisibility of the term evaluation in education research units across education degree programs. With the United Kingdom and United States professional standards also referring to evaluation as an area teachers demonstrate their meeting of professional standards, it is problematic to note there is a need for this knowledge to be demonstrated in practice, but little mechanisms for explicit teaching of evaluation across education degrees. To evidence a demonstrable impact for education, it is clear that evaluative skills will be required of teachers and school leaders, begging the question as to how these skills can be built.

2.1. Search strategy Systematic reviews provide a methodological approach to mapping the available research evidence on a specified topic, appraising that information, and synthesizing the results (Grant & Booth, 2009). The current study is considered a scoping review (Grant & Booth, 2009) which aims to provide a preliminary assessment of the size and scope of available literature in the area of interest within given time constraints. The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses Statement (PRISMA), which consists of an evidence-based set of items guiding the conducting and reporting of systematic reviews (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009). Three academic databases (ERIC, Education Source, and Psychinfo) were searched using the terms Evaluat* or research AND Teacher* or educator* AND skill* or attitude* or competenc* or practic* or approach*. The search was restricted to the date range of January 2000 to July 2016, reflecting a period in which a focus on school improvement and research capacity building in the profession heightened, with only English language peer reviewed papers included. Screening of papers for inclusion was conducted by both authors of this paper, with only primary empirical literature included. 2.2. Inclusion and exclusion criteria The initial search yielded a total of 655 studies (Fig. 1). The first stage of screening involved a review of titles only by the second author, to remove additional duplicates not removed through the database search system, and to exclude papers that clearly did not meet the inclusion criteria. For example, titles such as ADHD-specific knowledge and attitudes of teachers (ASKAT): Development and validation of a new research instrument (Mulholland, 2016), and many others were clearly papers publishing specific research findings related to teachers, rather than examining approaches to building research or evaluation capacities in teachers. The second stage of screening involved careful review of paper abstracts again conducted by the second author with reference to the inclusion criteria. In both of these stages a conservative approach was taken such that if there was any uncertainty about whether a particular paper should be included in either Stage 1 or 2 of screening it was retained until Stage 3. Stage 3 was conducted collaboratively by both authors together to reach consensus and included a review of the full text of 103 articles. Inclusion criteria were: a) described a specific activity designed to stimulate change in research and/or evaluation skills or attitudes in teachers or PSTs; b) had a clearly documented methodology. Papers were excluded if: a) there was no specific approach documented to build research and evaluation capacity but rather the study was an observational investigation of existing attitudes or practices; b) papers had no clear methodology or empirical approach to data collection and analysis; or c) papers were opinion or review pieces rather than primary studies.

1.3. Aims of the current study

2.3. Data extraction

The guiding research question for this review is: What is known about approaches to improving research and evaluative skills and attitudes in preservice and in-service teachers? We include both research and evaluation as the focus of the review because: a) evaluation appears to be a relatively new term in this field of inquiry and it is unlikely that a focus on this term alone will yield useful information; b) evaluation capacities include standard research skills as a necessary component, as discussed above, and so approaches to building research skills, with or without the evaluative focus, are highly useful to understand; and c) given there has not been a review of this kind in relation to either research or evaluation capacity building approaches for teachers, it is valid at this point to maintain a broad focus on both. This broad focus is reflected in the search terms that include evaluation OR research.

A total of 19 journal papers between 2003 and 2015 met the inclusion criteria and are included in this review (see Table 1). Papers were divided into two groups, with one group assigned to each of the authors of the current paper. Data extraction was conducted by the assigned author using a predetermined data extraction form. The form completed for each paper included details for: author; year; country; research focus/aim; research design / methodological approach; participants; data collection approaches; data analysis approach; theoretical / conceptual framework; approach to building research/evaluation capacity; and key findings. A sub-sample of papers (three from each of the current paper authors’ groups) were then cross-checked for data extraction by the other author. This process provided confidence that 3

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Fig. 1. Record of literature review processes.

sufficient and detailed data was similarly extracted by each author.

malleable through the capacity building approach taken (BradleyLevine, Smith, & Carr, 2009; Campbell & Jacques, 2004; Myers & Dillard, 2013; Ross & Bruce, 2012; van der Linden et al., 2012; van der Linden, Bakx, Ros, Beijaard, & van den Bergh, 2015; Vrijnsen-de Corte et al., 2013). Five more recent studies focussed on the extent to which the capacity building approaches influenced knowledge, confidence, and skills specifically in research processes (Lu, Ward, Overton, & Shin, 2014; Niemi & Nevgi, 2014; Tosun, 2014; van der Linden et al., 2015; van Ingen & Ariew, 2015). Beyond this focus on the proximal outcomes of research capacity building approaches, it was very common for studies to have a stronger or even sole focus on more distal outcomes of research capacity building including those related to enhanced practice or professional skills and knowledge. Seven studies positioned research capacity building as a means to broader teacher professional development including knowledge, personal development, and reflective practice skills (Colucci-Gray, Sharmistha, Gray, Robson, & Spratt, 2013; Goodnough, 2010; Hagevik, Aydeniz, & Rowell, 2012; Jaipal & Figg, 2011; Medwell & Wray, 2014; Myers & Dillard, 2013) and six of the studies were interested in the extent to which research capacity building yielded impacts on teacher practice (Bradley-Levine et al., 2009; Campbell &

2.4. Sample and study participants Included studies were primarily from the United States (7), with others from Canada (4), the United Kingdom (3), the Netherlands (3), Finland (1), and Turkey (1). The participants were primarily in-service teachers (in 11 studies), with seven studies focussing on pre-service teachers, and one study including both pre- and in-service teachers (Vrijnsen-de Corte, den Brok, Kamp, & Bergen, 2013). In-service teacher participants were largely primary and secondary school teachers, with one study specifically focussing on early childhood teachers (Nimmo & Park, 2009) and another a case study with a grade 1 teacher (Goodnough, 2010). 2.5. The focus of the studies The way that research or evaluation capacity building was positioned within the included studies varied greatly in relation to the desired or studied outcome (Table 1). Seven studies focussed on the extent to which positive perceptions or attitudes toward teacher research were 4

Country

USA

UK

UK (Scotland)

Canada

Canada

USA

Canada

USA

UK

USA

Author, year

Bradley-Levine et al. (2009)

Campbell and Jacques (2004)

Colucci-Gray et al. (2013)

Goodnough (2010)

Goodnough (2011)

Hagevik et al. (2012)

5

Jaipal and Figg (2011)

Lu et al. (2014)

Medwell and Wray (2014)

Miranda and Damico (2013)

Impact of a 6 week research experience program with experienced program mentors on teacher beliefs

Reflection on supporting the research skills of PSTs conducting research in schools and impact on their personal development.

Evaluation of a new approach to research education in relation to student confidence and skills in research.

Engagement/ practices of teachers in collaborative action research as professional development

Impact of action research in promoting reflective thinking and practice; perceptions about the action research process as related to teaching.

Impact of collaborative action research on selfknowledge; practice, broader education context.

Impact of collaborative action research on teacher knowledge and learning.

Impact of teacher action research programme on teachers’ knowledge, practice, and professional community.

Perceptions about the role of teacher research as professional development and as a way to impact practice.

Experiences in an action research course – perceptions of research and practice.

Study focus

Table 1 Details of papers included in this review.

Coursework -Graduate university program -Action research (AR) course -AR project undertaken -Professional learning community Teachers undertaking research projects -Mentoring by academics - Some focus on evaluation teaching and learning Teachers undertaking research projects -Action research -Mentoring by academics -Dialogical conversations with colleagues - Some focus on evaluation of teaching and learning Teachers undertaking research projects -Action research -Structured collaborative inquiry community Teachers undertaking research projects -Action research -Structured collaborative inquiry community Coursework -University pre-service program -Action research - AR project undertaken -Collaborative inquiry communities Teachers undertaking research projects -Action research -Semi-structured collaborative inquiry community Coursework -Voluntary graduate weekend workshops on the research process Preservice teachers undertaking research projects -One lecture -Group discussions -Tutor guidance Teachers undertaking research projects

Approach to building research capacity

Qualitative: discussion board posts, journal entries

Qualitative: video and text reflections

Quantitative: pre-post survey

(continued on next page)

In-service high school science teachers (n = 14). Variation in race, gender, education experience,

Teachers (n = 40) as graduate students, 76% female, mean 4 years teaching experience, mean year in graduate program just under 1 year, 95% Hispanic. PSTs (n = 8)

8 teacher researcher teams (3 to 5 teachers in each). Teams were at one school or across multiple schools.

Pre-service teachers (PSTs) middle grade teachers (n = 20); 21 to 40 years – mostly social study majors.

Qualitative: multiple case method and cross-case comparison: research reports and presentations, reflections, journals, conversations and class discussions, qualitative survey

Qualitative: interviews, observations, reflections, emails, research reports, field notes

Teachers (n = 10)

Grade 1 teacher (n = 1) conducting an individual AR project but working collaboratively with inquiry groups & facilitator (author).

Qualitative: single case study including follow-up; field notes, journals, lesson plan, student materials, interviews Qualitative: phenomenology: interviews

Teachers (n = 13; primary and secondary).

Teachers (n = 19) working across a range of schools and subject areas where funding had been supplied for small research projects.

Teachers (n = 12) in a graduate degree cohort; elementary and secondary

Participants

Qualitative exploratory study: appreciative inquiry, research report text,interviews

Mixed method small scale project (part of a larger project): survey, diary and field notes, group interviews

Qualitative: case study of AR courses: observations, email text, interviews

Research design & methodology

A. McFadden and K.E. Williams

Studies in Educational Evaluation 64 (2020) 100830

6

USA

Netherlands

Vrijnsen-de Corte et al. (2013)

Turkey

Tosun (2014)

van Ingen and Ariew (2015)

Canada

Ross and Bruce (2012)

Netherlands

USA

Nimmo and Park (2009)

van der Linden et al. (2012)

Finland

Niemi and Nevgi (2014)

Netherlands

USA

Myers and Dillard (2013)

van der Linden et al.(2015)

Country

Author, year

Table 1 (continued)

Perceptions about contextual input, personal input, process and outcomes of practice-based research schools.

Impact of course on skills in research skills of: articulating a classroom-based research problem, creating research-guiding questions, and designing and conducting effective search strategies.

Impact of an introductory course in research student teachers’ research knowledge and beliefs and attitudes towards research. Impact of an introductory course in research student teachers’ research knowledge and beliefs and attitudes towards research.

Impact of the research course on research knowledge and skills

Impact of undertaking action research on attitudes to research and professional confidence.

Impact of participation in a research mentorship team on early childhood teachers’ professional identities.

Impact of authentic researcher experiences as part of pre-service education on knowledge creation skills.

Coursework - Research processes - some focus on evaluating research literature Coursework - Research literature searching & selecting - Some focus on evaluating research literature Research-focussed whole-ofschool approach - Some focus on evaluating research project quality

-Laboratory science research -Mentorship by research scientist Teachers undertaking research projects -Action research -Semi-structured collaborative inquiry community Coursework - within whole-of-course focus on research - literature synthesis or action research projects undertaken - acknowledges that evaluating practice is a competency required by teachers Research mentoring team to inservice teachers. - Action research Teachers undertaking research projects -Collaborative action research with academics Coursework - Scientific research methods - Action research Coursework - Research processes

about pedagogy with aim of improving science pedagogy.

Impact of participation in an Action Research Group on teachers’ leadership attitudes and perceptions about involvement in a research project.

Approach to building research capacity

Study focus

Quantitative: survey

Quasi-experimental: research literature search portfolio - marked

Quantitative: survey

Teachers (n = 63) and PSTs (n = 39) from 50 schools.

PSTs (n = 96)

PSTs (n = 81)

PSTs (n = 96)

PSTs (n = 27)

Action research: survey with only open-ended questions Quantitative: pre-post quantitative survey: pre-post concept maps by students – quality rating

Teachers in two linked studies (n = 80 and 105)

Early childhood teachers (n = 7)

PSTs (n = 287) - two Finnish Universities; primary and secondary programs

Teachers (n = 24)

years of experience and diversity in terms of science discipline taught

Participants

Quantitative evaluation: pre-post survey

Qualitative: phenomenology: interviews, observations, meeting recordings, journals, documents

Quantitative: survey

Mixed methods (part of a longitudinal project): survey with quantitative and qualitative questions

Research design & methodology

A. McFadden and K.E. Williams

Studies in Educational Evaluation 64 (2020) 100830

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Jacques, 2004; Colucci-Gray et al., 2013; Goodnough, 2011; Hagevik et al., 2012; Miranda & Damico, 2013). Studies also focussed on the way that engaging in research capacity building activities influenced professional identities for teachers (Nimmo & Park, 2009), professional confidence (Ross & Bruce, 2012), had impacts on the professional community (Colucci-Gray et al., 2013), and influenced the broader educational context (Goodnough, 2011). Regarding evaluation specifically, there was limited representation of this construct across the review papers. In only one study teachers were described as using research skills within action research projects to evaluate programs or practices, with guidance of a mentor (ColucciGray et al., 2013). An earlier paper that also described teachers undertaking research projects with mentors acknowledged that teacher research may include evaluation of teaching and learning, but this was not clearly demonstrated or articulated in the description of the types of research teachers engaged in (Campbell & Jacques, 2004). Authors describing pre-service coursework related to research acknowledged that a key professional competency for teachers is to be able to evaluate their own practice (Niemi & Nevgi, 2014), but there appeared to be no specific attention paid to teaching the evaluative skills required within the course. Several studies describing pre-service coursework had a specific focus on the skills required to evaluate existing research literature and evidence (an important component of evaluative capacities), but did not appear to have a focus on evaluation of teaching and learning (Tosun, 2014; van der Linden et al., 2012; van Ingen & Ariew, 2015).

qualitative in nature (n = 9), with approaches including phenomenology, case study, appreciate inquiry, reflective work, and document analysis (Table 1). This means that findings presented in the following section provide very limited evidence of effectiveness for the approaches. However, the synthesis and discussion of these findings to date across a range of study designs is a useful exercise in that it can provide the basis for the development of future research agendas in this area. 3. Results 3.1. Documented approaches to building teacher research capacity The most widely used approach to building teacher research capacity across the studies was planned and enacted support of teachers undertaking small research projects, with nine studies focussed on this method (eight with in-service teachers and one with pre-service teachers; Table 1). Support tended to include the formation of collaborative inquiry communities with other teachers undertaking research, with regular meetings, workshops where information on planning and undertaking research was delivered by university academics, and ongoing mentoring by university academics. Of these nine studies, six identified the research undertaken by participants as action research if situated in classrooms (Colucci-Gray et al., 2013; Goodnough, 2010, 2011; Jaipal & Figg, 2011; Myers & Dillard, 2013; Ross & Bruce, 2012). Pre-service (n = 6) and in-service (n = 2) teacher coursework approaches were the next most common approach to building research capacity. These courses focussed on: literature searching and selection through two workshops co-designed with librarians for pre-service teachers (van Ingen & Ariew, 2015); learning about the research process without undertaking research in the field through semester long university units for undergraduates (Tosun, 2014; van der Linden et al., 2012; 2015) or voluntary weekend workshops on seven research process topics for postgraduate students (Lu et al., 2014); and learning about research through coursework with the inclusion of a research project for undergraduate pre-service teachers (Hagevik et al., 2012; Niemi & Nevgi, 2014) and postgraduate teachers (Bradley-Levine et al., 2009). Again, in this group of studies, there was a relatively consistent identification of teacher research as action research (Bradley-Levine et al., 2009; Hagevik et al., 2012; Niemi & Nevgi, 2014; Tosun, 2014). While one coursework approach focussed on the skills required to select and evaluate research literature (van der Linden et al., 2012, 2015; van Ingen & Ariew, 2015), there were no documented coursework elements related specifically to evaluation. Further to these approaches, one study had a specific focus on research mentorship in which monthly meetings of teachers were facilitated by two teacher educators with a focus on action research approaches (Nimmo & Park, 2009). The first year of meetings focussed on exploration and exchanging ideas about the meaning of research and the collection of informal data. This exchange led to a natural process in which teachers became collaborative researchers with the academics in a project conducted in the second year. Finally, one Dutch study focussed on a particular type of school (Professional Development School [PDS]) in which there is a commitment to research as a central part of the school organisation (Vrijnsen-de Corte et al., 2013). Teachers in PDSs conduct research in conjunction with their classroom teaching role. They collaborate with other school teachers and with teacher education institutions to do so (Darling-Hammond, 2005). PDSs were developed to enable partnerships to focus on continual improvement of practicing teachers, improved outcomes for students, and improved teacher education in integrated and interconnected ways (Levine and Trachtman, 1997).

2.6. Organisation of review study findings and strength of evidence provided by study designs In the following results section, first the different approaches to building teacher capacity that were the subject of the research paper are documented. Next, findings presented in each of the review studies were categorized as either (1) outcome results related to specific teacher attitudes, knowledges, skills or behaviours influenced by the capacity building approach (the what), or (2) process related findings that explained the mechanisms including enablers and barriers to implementing the capacity building approach and stimulating the desired changes in participants (the how; see Table 2, discussed in a following section). Outcome results are presented first and are grouped according to four categories of outcomes documented in the studies, each addressed in turn. As is common in systematic reviews within education, we present a narrative synthesis here of the included studies. We use well established hierarchies of evidence in relation to research designs, to provide a descriptive account of the strength of evidence produced by the studies reviewed. That is, experimental studies with random assignment are able to make the strongest claims of effectiveness, followed by quasi-experimental studies that include comparison groups, pre- and post- comparison studies, cross-sectional studies, action research, and finally qualitative studies (Pawson, 2006). As will be documented below, there were no experimental designs represented in the review studies and only one quasi-experimental study represented which examined the impact of coursework on pre-service teachers’ skills in searching for and selecting quality research literature in relation to a research problem (van Ingen & Ariew, 2015). Three studies used a quantitative pre-post design (Lu et al., 2014; Ross & Bruce, 2012; van der Linden et al., 2015). These pre-post design studies examined the impact of coursework for pre-service (van der Linden et al., 2015) and in-service teachers (Lu et al., 2014) in terms of research knowledge, beliefs, and attitudes, and the impact of undertaking action research projects on teacher attitudes toward research (Ross & Bruce, 2012). Three studies used a cross-sectional quantitative design (Niemi & Nevgi, 2014; van der Linden et al., 2012; Vrijnsen-de Corte et al., 2013), two studies used a mixed method design (Campbell & Jacques, 2004; Myers & Dillard, 2013), and one study used an action research design (Tosun, 2014). Most of the review studies were 7

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Table 2 Outcome and process findings presented in each of the review studies. Author, year

Approach to building research capacity

Key outcome findings (what)

Key process findings (how)

Bradley-Levine et al. (2009)

Coursework -Graduate university program -Action research (AR) course -AR project undertaken -Professional learning community

Professional learning community supported teachers in AR understanding and implementation Collaborative nature assisted reflection on inquiry

Campbell and Jacques (2004)

Teachers undertaking research projects -Mentoring by academics

Teachers reported: - increased confidence in and ability to carry out own research, including evaluating existing research findings -relevance to own context -deep understanding and reflection on practice, and changes in practice - AR research contributed to (1) deep understanding and reflection on practice, and (2) changes in practice Teachers reported: - increased knowledge of what was being taught - more pupil centred planning - better focus on evaluation of teaching and learning

Colucci-Gray et al. (2013)

Teachers undertaking research projects -Action research -Mentoring by academics -Dialogical conversations with colleagues Teachers undertaking research projects -Action research -Structured collaborative inquiry community

Reflection on AR allowed a critical and proactive stance to practice.

Goodnough (2011)

Teachers undertaking research projects -Action research -Structured collaborative inquiry community

Hagevik et al. (2012)

Coursework -University pre-service program -Action research - AR project undertaken -Collaborative inquiry communities

Jaipal and Figg (2011)

Teachers undertaking research projects -Action research -Semi-structured collaborative inquiry community

Lu et al. (2014)

Coursework -Voluntary graduate weekend workshops on the research process

Teacher identity changed through: - increasing confidence in teaching science - viewing “teacher-as-critical-learner” including taking risks, collaborating, reviewing literature and not accepting things at face value, thinking creatively. - viewing students differently re diverse learning and changing approaches Professional practice changed through: - ongoing use of approaches explored through AR - increasing use of research literature - taking a more critical and reflective stance - having more systematic approaches to practice and reflection Teachers reported their desire to engage in AR again and valued it as PD. School & broader educational context changed through: - strengthened collegial relationships and dialogues - informal dissemination Teaching/research goals were: related to the teachers themselves, improving student learning, communicating with parents, and ways of teaching subject matter PSTs mostly moved to dialogic level of reflection (e.g., understanding multiple ways of teaching for all students, listening to and incorporating student ideas) AR facilitated discussions about theory and practice links. Completing lit reviews and hearing other presentations increased knowledge of the field and alternative practices. AR gave opportunity to challenge and extend existing personal beliefs about student learning and their practice. PSTs learnt to collaborate and discuss problems with others. Three collaborative approaches identified: - classroom practice within one school - classroom practice within multiple schools - school-wide issues within on school Teachers found AR more beneficial to their learning than board-mandated PD, but noted the workload was above that of usual reflective practice. Significant gains in confidence and skills. Students highly satisfied with workshop features.

Goodnough (2010)

Positive aspects of AR for teachers included: - ability to “create their own knowledge” - incorporating reflection, trying out different things, interrogating literature into teacher practice - more willing to take risks and discuss them. Broad dissemination of knowledge gained through the AR project did not occur

Regular contact between research mentors and teachers Effective mentoring requires communication, transparency and role clarity. Main support mentors gave included refining research proposals, using ICT, critical analysis of data, and dissemination of findings. Teachers motivated to engage in AR when related to practice and aimed at developing professional knowledge Some teachers may be drawn to AR by a personal issue.

- Considerable time needed for planning - teachers may struggle to narrow research focus and formulate a specific question Inquiry groups used to help choose a research focus, support planning; and reflect and make changes Traditional dissemination avenues are time consuming (e.g., journals, conferences, etc.). AR inquiry groups seen as under-utilized PD opportunity for teachers. Recommended approaches include: - teacher agency - collaboration - reflection

Recommend a: - reflection rubric for self-assessment early in course −12-month time period - partnership between PSTs, mentoring teachers, faculty, uni supervisors, and AR professor

Sustained collaboration and engagement for novice researchers needs to be supported Release time is valuable

(continued on next page)

8

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Table 2 (continued) Author, year

Approach to building research capacity

Key outcome findings (what)

Key process findings (how)

Medwell and Wray (2014)

Pre-service teachers undertaking research projects -One lecture -Group discussions -Tutor guidance

Time was a barrier Having other members of the research group readily available was important Tutors supporting the research rarely mentioned as part of success factors in the process

Miranda and Damico (2013)

Teachers undertaking research projects -Laboratory science research -Mentorship by research scientist

Myers and Dillard (2013)

Teachers undertaking research projects -Action research -Semi-structured collaborative inquiry community

Niemi and Nevgi (2014)

PSTs undertaking research projects

Nimmo and Park (2009)

Research mentoring team to in-service teachers. - Action research focus

Ross and Bruce (2012)

Teachers undertaking research projects -Action research -Mentorship

Tosun (2014)

Coursework - Scientific research methods - Action research

van der Linden et al. (2015)

Coursework - Research processes

van der Linden et al. (2012)

Coursework - Research processes

van Ingen and Ariew (2015)

Coursework - Research literature searching & selecting

The research process was discussed as a way of discovering how much they could affect children’s progress by working with others and the nature of the thinking which underpinned this. Each pre-service teacher emphasised different aspects of personal development with main categories: - growth of feelings of self-efficacy, particularly in making decisions, discussed by seven of the eight; - new insights into working with other adults; - a better understanding of the many issues which come between any idea and outcomes for learners. Teachers: - assisted with cognitive skills that contributed to successfully conducting of research (critical thinking and reasoning, data analysis) - considered important for science teacher education in providing opportunities for students to practice and develop these skills - stated the apprenticeship nature of the program enabled reflection on what it feels like to be a learner Fifty-four percent of the participants changed their perspective on action research. 4 participants noted the collaborative approach contributed to their pedagogy. 3 participants noted impact that AR had on their pedagogical practices. The process contributed to feelings of empowerment with regards to their own pedagogy. Research studies in TE can build research skills in critical thinking, reviewing research literature, learning of research methods and independent inquiry PST consider learning research skills as important for their professional development and for the advancement of knowledge and growth of the teaching profession Research study assisted PSTs with collaboration and in classroom teaching Teachers broadened view of research as an accessible and relevant part of everyday practice Collaborative and reflective culture is important for teachers to feel ideas were “safely” presented Teachers demonstrated willingness to engage in deep discussions about their practice Teachers moved from being research “outsiders/ consumers” to research contributors Study 1 Participation in AR contributed to: teacher beliefs about their ability to, and confidence in, engaging students in classroom activities Study 2 Pre–post improvement on teacher attitudes toward educational research and on teacher efficacy. PSTs valued undertaking the research course but found the concepts difficult at times The course contributed to a decrease in anxiety around doing research The course contributed to raising self-efficacy beliefs. The course contributed to PSTs gaining skills in the areas of article writing, article review, and literature review. PSTs concept maps improved over the course and included: (1) research topics, (2) goals and benefits, (3) research process; (4) quality of research, (5) research methods, and (6) participants in research. Course focused on development of research knowledge and practices combined with a focus on the development of a positive approach to research. Study reports a positive attitude towards research in the areas of value of research and own capacity to carry out research. Students had increase in research knowledge and skills Class that included collaboration had significantly higher achievement that the class without collaboration.

Factors teachers believed helped them be successful in conducting research - science content knowledge - laboratory experiences -cognitive skills - mentor dispositions.

Collaboration is important for sharing of research process and support through research.

Research studies must be integrated with other parts of teacher education. PSTs’ comments show that becoming a teacher who creates knowledge to improve a school is a holistic process Support is required from supervisors as well as a TE curriculum design Active learning experiences reinforced positive effect. Stimulating intellectual environments for teachers contribute to supportive networks for teachers engaged in research Creation of reflective and collaborative cultures with mutual support contribute to positive feelings associated with research Self-efficacy benefits were greater for teachers: involved in a more rigorous collaborative AR processes resulting in conceptual understanding; worked at a school supporting teacher learning; and had prior research experience.

Students experienced challenges in: - Understanding verbal concepts lecturers presented - Deciding on a research topic - Conducting the research.

PSTs found working in pairs or groups on task useful. Working with ‘realistic tasks’ and with examples of research from practice was valued by PSTs.

Workshops included: Development of researchable research questions through modelling Academic support to review and refine plans practice searching with academic support

(continued on next page) 9

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Table 2 (continued) Author, year

Approach to building research capacity

Key outcome findings (what)

Key process findings (how)

Vrijnsen-de Corte et al. (2013)

Research-focussed wholeof-school approach

Questionnaire on Teacher Research shown to be a useful tool for assessing teachers’ and PSTs’ perceptions of their practice-based research efforts in secondary education schools. Highest scores were related to research motives and the outcomes of practice-based research. PSTs perceived the ‘research infrastructure’ less positively than experienced teachers. Experienced teachers rated more positively than PSTs on the ‘evaluating and reporting research’ scale Little differences reported between school types.

When schools are implementing activities associated with practice-based research there should be a focus on: Research motives for performing the research Selecting those interested in research

3.2. Study findings related to outcomes: what outcomes are documented in relation to approaches to building teacher research and evaluation capacity and what is the level of evidence available for these?

• •

(Goodnough, 2011, p.80). 3.2.2. Attitudes, knowledge, skills, and confidence in research processes Eleven of the 19 review papers documented impact findings related to teachers’ attitudes, knowledge, skills, and confidence in research processes. The strongest evidence was found in two studies of coursework approaches (Lu et al., 2014; van der Linden et al., 2015) and one of teachers engaging in research (Ross & Bruce, 2012) each with prepost quantitative designs. van der Linden et al. (2015) collected preand post-data from 96 second-year undergraduate pre-service teachers undertaking an 11 session introductory research course. Self-reported positive attitudes toward research significantly increased and negative beliefs about research decreased over the course. In a unique measure of research knowledge (at least across the review studies included here), students were asked at the beginning and end of the course to complete concept maps reflecting their ideas and knowledge about research by primary school teachers. These concept maps were analysed and evaluated for the number and relevance of the concepts present, with substantial improvements shown across the course for most students. Findings indicate that the development of pre-service teachers’ research knowledge and positive changes in beliefs and attitudes towards research can be achieved in their second year of study (van der Linden et al., 2015). Similar findings have been documented with teachers undertaking postgraduate studies reporting increased confidence in reading research articles, and in their level of research knowledge measured through pre and post surveys across a series of eight workshops (Lu et al., 2014). Ross and Bruce (2012) investigated the impact of teachers undertaking collaborative action research projects, in their own settings with academics, and found self-reported perceptions about the usefulness and relevance of educational research to improve for teachers over time. This improvement is significant as it is known that positive attitudes towards research create the conditions for core elements of teacher practice including interpreting student data, setting learning goals, and monitoring the outcomes of student achievement (Harris, Chapman, Muijs, Russ, & Stoll, 2006). These findings on the role of coursework in supporting research skills and confidence are also reflected in qualitative studies (BradleyLevine et al., 2009; Tosun, 2014). A case study of 12 graduate teachers using data sources including email exchange, interviews, and observations reported an increase in teacher confidence across the course of their study, an increased familiarisation with research tools and the presentation of findings, and increased reflection and the taking of pedagogical risks (Bradley-Levine et al., 2009). While research undertaken with 27 pre-service teachers through a questionnaire relating to research coursework reported anxiety levels around conducting research reduced, and despite reporting difficulties within coursework at

In this section five main categories of outcomes represented in the review studies are described: skills and participation in sourcing and evaluating research findings (research literacy); attitudes, knowledge, skills, and confidence in research processes; professional competences or reflective practice skills; teacher identity and self-efficacy; and collaboration, professional conversations, and collegial relationships. These categories arose from the findings reported across the studies and although evaluative thinking and specific evaluation competencies were searched for within the included studies, these were not specifically located. Within each of the following sections, the findings are synthesized according to the hierarchy of evidence described above (Pawson, 2006). 3.2.1. Skills and participation in sourcing and evaluating research findings (research literacy) Five studies documented findings indicating that capacity building approaches had stimulated increased skills in sourcing quality research evidence (Tosun, 2014; van Ingen & Ariew, 2015), skills in evaluating the sources and evidence presented (Niemi & Nevgi, 2014; Tosun, 2014; van der Linden et al., 2012; van Ingen & Ariew, 2015) and that following the capacity building exercise teachers more often sourced and evaluated research findings (Goodnough, 2011). The strongest evidence for the effectiveness of capacity building in this outcome area was found in the only quasi-experimental study included in the review. van Ingen and Ariew (2015) designed and implemented two workshops for pre-service teachers in collaboration with librarians. The workshops focussed on constructing a narrow, answerable research question, and research searching and retrieval. One group of students were exposed to these workshops (n = 28) while another group of students were the comparison group and undertook the same tasks independently (n = 24). Submitted research portfolios documented the research question, search strategy, and selected article for review. A marking rubric was used to assess the quality of the research portfolios, with students receiving the workshops scoring significantly better across all three areas. In two other undergraduate coursework approaches, pre-service teachers reported through surveys that skills in research literature reviewing were important skills gained through participation in the course (Tosun, 2014; Niemi & Nevgi, 2014). Teachers involved in undertaking action research projects with the support of a structured collaborating inquiry community also reported through interviews that this experience had given them a “greater appreciation of… research literature and how it can be used to inform professional practice”

10

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

times, teachers felt an increase in self-efficacy, combined with a recognition of the value of the skills they learned (Tosun, 2014). The undertaking of research projects has also been found to improve research knowledge and skills among teachers (Myers & Dillard, 2013), and pre-service teachers (Niemi & Nevgi, 2014). Survey responses indicated that teachers gained a greater understanding of action research processes, and some had gained a sense of empowerment (Myers & Dillard, 2013). Pre-service teachers reported they had gained knowledge in critical research literacy, and in their understanding of the value and role of research within the profession (Niemi & Nevgi, 2014).

abilities (Ross & Bruce, 2012). In the only quantitative study in this group, Ross and Bruce (2012) investigated the impact of teachers undertaking collaborative action research projects in their own settings with academics and found increases in self-reported teaching efficacy in the areas of student engagement in learning activities, instruction skills, and classroom management. Findings about identity changes were also reflected in qualitative studies of teachers (Goodnough, 2010, 2011) and pre-service teachers (Medwell & Wray, 2014) undertaking research. It was reported that an increase in self-knowledge and confidence in their discipline contributed to teachers’ views of themselves as critical learners (Goodnough, 2011). Similarly, eight pre-service teachers participating in a UK small scale handwriting intervention program, identified research into their practice as facilitating growth in their professional development in the areas of decision making and self-efficacy, expanded understandings of outcomes for learners, and improved understanding of working with others (Medwell & Wray, 2014). Coursework undertaken by both teachers (Bradley-Levine, 2014), and preservice teachers (Tosun, 2014) has also been associated with increased feelings of confidence in the ways research could be applied to professional work and changes in teacher self-conceptions (Bradley-Levine et al., 2009; Tosun, 2014). One of the review studies had a specific focus on the use of collaborative research mentorship for the purposes of impacting on early childhood teacher identity (Nimmo & Park, 2009). By tying research activities to what occurs organically as part of early childhood teachers’ practice, collaborative research can create an expanded view of pedagogical approaches, documentation approaches, inquiry processes and lead to a culture of inquiry and reflection for teachers and children (Nimmo & Park, 2009). The significance of this is early childhood teachers can build their research practices on skills already present in their professional repertories, while enhancing their professional identity.

3.2.3. Professional competencies or reflective practice skills Seven of the 19 review studies reported findings related to impacts on the ways teachers or pre-service teachers used systematic researchbased procedures, or critical thinking learnt through research capacity building exercises, to ultimately improve or influence their practice. For example, analysis of survey data provided by pre-service teachers who undertook coursework in Finland, in which research learning was embedded, found that the course focus on critical research literacy skills predicted the extent to which students felt prepared for designing instruction and maintaining their professional development (Niemi & Nevgi, 2014). Qualitative studies on the impact of coursework (Bradley-Levine et al., 2009; Hagevik et al., 2012) show an increase in teacher confidence to make changes based on research in their own classrooms and increased reflection on their teaching and positive changes in their pedagogical decision making processes (Bradley-Levine et al., 2009). A study of 20 pre-service teachers in a multiple case method spanning a 12-month period reported on the effectiveness of engagement with action research in unpacking personal beliefs, critically reflecting on their classroom experiences, and analysis of classroom data (Hagevik et al., 2012). Qualitative studies on the impacts of teachers undertaking research projects also note these impacts on reflective practice as a positive outcome (Colucci-Gray et al., 2013; Goodnough, 2010, 2011; Miranda & Damico, 2013). Colucci-Gray et al. (2013) investigated action research participation with teachers reporting an increase in their approach to systematic approaches to reflection and increased evaluation of their teaching practice. Of note, these teachers already considered themselves to be reflective practitioners, however their engagement in the action research enabled improvements in the ways teachers engaged in reflective practice (Colucci-Gray et al., 2013). A phenomenological study on the lived experiences of 10 teachers engaged in action research reported changes to professional practice in the areas of confidence in content knowledge, viewing students from a more holistic viewpoint, expanding their reflective practice to examine their professional practice in explicit ways and an increased focus on inquiry orientations (Goodnough, 2011). Practicing teachers engaged in research intensive experiences in a science discipline have reported a range of cognitive skills such as critical thinking, reasoning and analysis as enabling them to be successful in research engagement (Miranda & Damico, 2013).

3.2.5. Collaboration, professional conversations, and collegial relationships Five of the 19 review studies reported impacts related to collaboration, collegial relationships and the role of research capacity building in stimulating useful professional conversations. Qualitative data from a range of studies documenting the experience of teachers and pre-service teachers undertaking research in the field suggest that this experience impacted positively on their collaborative skills and increased professional networks (Myers & Dillard, 2013), strengthened collegial relationships and dialogues (Goodnough, 2011), and helped teachers to develop new insights into working with other adults (Medwell & Wray, 2014). Pre-service coursework, which involved undertaking an action research project, has also been linked with increased collaborative skills in participants (Hagevik et al., 2012). 3.3. Study findings related to process: what are the enablers and barriers to building teacher and pre-service teacher research and evaluation capacity? The most common identified enabler of building capacity for research in teachers was collaborative work through semi-structured collaborative inquiry communities (Bradley-Levine et al., 2009; Goodnough, 2010, 2011; Hagevik et al., 2012; Jaipal & Figg, 2011; Myers & Dillard, 2013; Ross & Bruce, 2012) and collegial discussions (Colucci-Gray et al., 2013; Medwell & Wray, 2014). Qualitative studies on the role of teachers undertaking projects in improving attitudes toward research have documented findings reporting on the success of collaborative action research approaches that build confidence and improve pedagogical outcomes for teachers (Jaipal & Figg, 2011; Myers & Dillard, 2013). Findings from these studies also highlighted

3.2.4. Teacher identity and self-efficacy Eight of the 19 review studies reported findings related to teacher identity impacts as a result of engagement in the research capacity building approach. Ideas about teacher identity were reflected in language about teaching confidence, empowerment, self-efficacy, risktaking, ethical practice, teachers as learners, and importantly, teachers as knowledge creators, as opposed to only knowledge-receivers. Stimulating, rigorous and supportive environments enable teachers’ self-efficacy to grow and positively impact teachers’ professional

11

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Fig. 2. Draft logic model of research and evaluation capacity building approaches for teachers and documented outcome chains represented across the review studies.

professional community included the nurturing of a collaborative and reflective culture. This culture, grown though dialogic and respectful interactions was described in the findings to be a contributing factor in participants expressing an expanded view of research paradigms, a reexamination of research intent and a connection to their own practice (Nimmo & Park, 2009). Findings indicated engagement in rich dialogue and a supportive mentoring environment enabled teachers to broaden their view of research processes and the alignment with their professional practice, engage in reflective and critical conversations, and shift their position of consumer of research to research contributors (Nimmo & Park, 2009). Other studies reported that positive mentor dispositions (Miranda & Damico, 2013), and regular contact with mentors were important success factors (Campbell & Jacques, 2004). Communication, transparency and clarity in roles was critical for effective mentoring with common support provided by mentors including refining research questions and supporting critical analysis of data collection, and dissemination (Campbell & Jacques, 2004). A number of other key considerations for teachers were identified. These included the importance of providing opportunities for teachers to adopt research topics in line with their own interests and needs, as opposed to pre-determined professional learning outcomes (Jaipal & Figg, 2011). This can be linked to ideas that teacher motives are an important consideration, with authors suggesting that only those teachers with a strong interest in research successfully embed this as part of their practice (Vrijnsen-de Corte et al., 2013). However, this is problematic given research and evaluation knowledge and skills are an expected competency for all teachers. Few studies discussed barriers or challenges associated with the approaches taken in any depth. Key barriers that were noted included the considerable time needed for planning and participating in research projects and capacity building activities (Goodnough, 2010; Medwell & Wray, 2014), the difficulty in narrowing a research focus to a specific researchable question (Goodnough, 2010; Tosun, 2014), and the time-

collaborative action research communities that support novice teachers through practice-based inquiry are a successful approache (Jaipal & Figg, 2011). Enablers of collaborative practice for in-service teachers include ways of minimising work load issues, planned release time and regular collaborative interactions (Jaipal & Figg, 2011). An environment that fosters a culture of professional learning and encourages deep engagement with research, contributing to deep understanding of subject and practice knowledge, supports in-service teachers to engage with research (Ross & Bruce, 2012). Collaborative engagement across universities and within the field has been found to increase confidence in implementing research in practice, using research tools and presenting findings to colleagues (Bradley-Levine et al., 2009). For pre-service teachers studying research units as part of their coursework, collaborating as pairs or groups, using examples from practice and performing realistic tasks has been reported as a contributor to success in the area of attitude, knowledge and skills (van der Linden et al., 2012). Active learning experiences have been found to be important in engaging pre-service teachers’ capacities to engage in research and subsequently, to be able to teach their students skills such as critical thinking and independent inquiry which are important for successful research (Niemi & Nevgi, 2014). To contribute to professional learning, the quality of the supervisors, the quality of research studies and the integration of research studies with other elements of teacher education are important in pre-service teachers developing research capacity (Niemi & Nevgi, 2014). Mentorship by academics was another key process linked to success across a number of studies (Campbell & Jacques, 2004; Colucci-Gray et al., 2013; Miranda & Damico, 2013; Nimmo & Park, 2009; Ross & Bruce, 2012). This is reflected in the mentoring-focussed approach taken by Nimmo and Park (2009) which involved monthly research mentorship team meetings of two teacher educators and seven early childhood teachers in the United States. The creation of this supportive,

12

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

consuming nature of traditional research dissemination avenues (Goodnough, 2010). Overall, barriers appear to be an under-researched area and worthy of future investigation.

benefits beyond that of a single classroom, school, or even region (Hattie, 2015). Currently most high quality evaluation evidence for particular educational approaches is produced by academics and distributed through repositories run by organisations such as Evidence for Learning by Social Ventures Australia (2018a), 2018b, 2015), the Education Endowment Foundation (UK; n.d.f), and the Institute of Education Sciences (USA; n.d.f). While these are useful repositories, strong co-investigation models that bring together teachers and educational leaders, with or without academics are not always evident. A greater focus on building evaluative capacities and confidence in preservice teacher education will support teachers to engage in these evaluative practices, building expertise from within and establishing a culture of evaluation. “Schools need to become incubators of programs, evaluators of impact and experts at interpreting the effects of teachers and teaching on all students. In short, we need to develop an evaluation climate in our education system” (Hattie, 2015, p.15). A core underlying skill to creating this evaluative culture is Evaluative Thinking which, if applied widely, can ensure evaluation efforts focus not only on impact but also on process (Buckley et al., 2015). Evaluative Thinking is “critical thinking applied to contexts of evaluation” (Buckley et al., 2015, p. 376) and has been identified as a skill that does not occur naturally and requires intentional teaching (Buckley et al., 2015). While calls for an increasing evaluation focus in education have primarily focussed on student learning impact, Evaluative Thinking and evaluation methodologies can be applied to any aspect of the educational context. An evaluative focus on not only learning outcomes, but the wide array of approaches to teacher education and support, legislative and reporting requirements, and school leadership, will contribute to a strengthened teaching professional. For example, the international school improvement agenda has bought a range of measures and review processes to schools and systems that are likely to have a number of both intended and unintended consequences on teacher and leadership identity and practice. A carefully designed and conducted process evaluation could illuminate these mechanisms creating increased understanding of the ways these processes shape the work of teachers. These complex mechanisms often have downstream effects on key issues such as teacher retention, and student learning and the use of evaluation to zoom in on such mechanisms may be highly beneficial. The studies revealed there was a clear and predominant focus on teacher-conducted action research projects with the specific aim of encouraging greater reflective practice in educators. There are two related and potentially problematic issues with this. First, the research has not focussed on building research capacity per se, but rather positions action research as a means to a different, yet important, end – increased reflective practice skills. Second, although action research might be considered one methodological choice from several diverse educational research methodologies (Rowell, Yu Polush, Riel, & Bruewer, 2015), a number of the studies appear to adopt a definition of action research that suggests that it is the ONLY type of research conducted by teachers with the corollary that ALL research conducted by teachers is action research. For example: “the process by which practitioners study problems in a systematic manner for self-improvement and to increase their knowledge of the curriculum, teaching, and learning is called action research” (Jaipal & Figg, 2011, p. 60) and “action research is any systematic inquiry conducted by teachers… and stakeholders in teaching / learning to gather information about how schools operate, how they teach, and how well their students learn” (Hagevik et al., 2012, p. 675 quoting Mills, 2007, p. 5). Definitional confusion around action research has been recently noted by leaders in the field (Miranda & Damico, 2013) and this review suggests that this definitional ambiguity might also extend to teacher research, which is

4. Discussion This systematic scoping review aimed to document what is known about approaches to improving research and evaluative skills and attitudes in teachers. Given that research and evaluation capabilities are no longer optional extras for the professional work of teachers, but are required skills, it is essential to better understand how to support these capacities within the profession. While a number of studies with useful findings were included in this scoping review, overall this area is somewhat under-researched. Taken together, the studies across this review do not provide strong evidence for the effectiveness of the approaches taken due primarily to the study designs. Several key issues arising from the review of studies are discussed below. Across the review studies, there were various approaches to building research and evaluation capacity represented with the most common including professional learning communities, mentoring, teachers conducting research projects, and pre-service teacher coursework. The expectation is that building research skills will ultimately contribute to quality teaching practice which in turn will contribute to improved student achievement outcomes (Gore et al., 2016; Hattie, 2003). However, what is apparent across the review findings is that this mechanism is unlikely to be direct. Rather, a number of indirect pathways to these aims are likely. Fig. 2 presents a draft logic model that summarizes the capacity-building activities and their documented outcomes across the review studies. These activities are likely to stimulate short term change in attitudes, knowledge, and discrete skill sets that might then contribute to confidence, identity, and behaviour changes in educators over the medium term, and system level changes in practice and outcomes in the longer term. Of note are the unshaded boxes in which very little or no evidence was available across the review studies despite these aspects being considered as important elements in teacher standards and educational policy documents (BERARSA, 2014; National Board for Professional Teaching Standards, 2016; White et al., 2018) A number of these areas are discussed in the following sections. Further, there was very limited investigation or discussion across the studies about the numerous barriers that teachers face in learning and implementing research and evaluation skills including workload, support, and systemic barriers. If these barriers are to be addressed, more needs to be known about when and how they influence teacher capacity building in the context of research and evaluation. Across the studies there was almost no focus on specific evaluation skills for teachers. This is a critical gap given teachers and the education system as a whole are being increasingly expected to evaluate their practices and document impact on learners (BERA-RSA, 2014; OECD, 2013). This emphasis on evaluation in teaching professional standards (Australian Institute of Teaching & School Leadership [AITSL], 2012; National Board for Professional Teaching Standards, 2016) is considered two-fold: as a lever for improving teaching practice; and, for ultimately improving student learning outcomes. In order to evidence these demonstrable impacts for education, it is clear that evaluative skills will be required of teachers and leaders at the classroom, school, district, and policy level. However, it appears that it is not yet fully recognized what evaluation takes, and that this particular skillset will need focussed development within the profession. Evaluative research, if appropriately shared and translated for all stakeholders has the potential to contribute to a practicable evidence base created by and for teachers and school leaders, with wider spread

13

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

being considered as synonymous with action research, narrowing the methodological research opportunities available to teachers.

This would see ideas about evaluative thinking embedded across the whole course, as well as taught in specific units. Future studies will also need to evaluate this approach. In parallel to this pre-service strategy, a framework for building capacity in the in-service teaching profession, and in educational leaders is needed. The review has shown that several elements of research in the field are enhanced by: collaborative communities; topics of research that are connected to practice and solve real world challenges; intellectual growth supported by collaborative cultures; and when explicit partnerships with external agencies such as universities support teachers in their research. The review also highlighted very real concerns for in-service teachers that need to be mitigated to enhance a focus on research, including time constraints, self-efficacy, and work load issues. Finally, the research to date that has evaluated capacity building approaches has not documented longitudinal evidence for the full range of downstream effects hypothesized given the synthesis of findings from this review (Fig. 2). Further and stronger evidence for the effectiveness of approaches through more rigorous research designs involving comparison groups is still needed. Longitudinal mixed method evaluations of approaches that begin in pre-service education and aim to systematically test the draft logic model presented here will help to move this field forward. The review has illuminated a gap in teacher education in terms of an explicit focus on evaluative skills and attitudes. While the studies reviewed had a focus on ways to improve general research capacities in teachers, there is a silence in the literature around how to build evaluative skills and attitudes for pre-service teachers and largely with inservice teachers. This is despite the need for teachers to be engaged in evaluation of their own practices and, throughout their career continuum, largely responsible for the evaluation of broad programs and mechanisms to show accountability of their, and their colleagues’, practice. A deeper focus in initial teacher program design could enhance the evaluative skills and attitudes for pre-service teachers. A recommendation of this review is for initial teacher education courses that include research units in their programs to embed evaluation in their course teaching. To build in-service teachers evaluative capacity, frameworks need to be developed within the field and then evaluated for effectiveness to ascertain whether this approach contributes in this area.

4.1. Limitations There are a number of limitations of this review to consider. First, the search was completed in 2016 with only papers dated 2003–2015 included. It is likely that since that time several other relevant studies have been published. Second, the search terms were not exhaustive and grey literature was not included. It may be that even broader terms beyond research and evaluation, and a detailed search of the grey literature would have yielded additional relevant studies. Third, a formal quality appraisal process was not engaged in but rather the studies were described and synthesized in terms of a hierarchy of evidence. Overall, data extraction from the studies suggest that the rigour and quality of research in the field varies greatly, and reporting often does not follow established protocol that would allow for rigorous quality appraisal. Future research efforts should aim to address this issue of clear and transparent reporting. Finally, despite multiple nations calling for increased research and evaluation skills in teachers (BERA-RSA, 2014; National Board for Professional Teaching Standards, 2016; White et al., 2018), most of the studies included in this review were from North America. This means that it is unknown to what extent the findings reported here can be generalized to other contexts. 4.2. Implications The findings of this review have a number of implications for practice, research and a priority on building evaluative capacities for teachers. Initially, more equitable access to research and evaluation focused units of study within pre-service teacher courses should be addressed. There is some evidence that pre-service teacher coursework may promote both research attitudes and skills, and professional identity and reflective practice skills (Hagevik et al., 2012; Medwell & Wray, 2014; Niemi & Nevgi, 2014; Tosun, 2014; van der Linden et al., 2012, 2015). However, there is currently inequitable access to preservice teacher courses in research, at least in Australia (see Appendix A), and limited focus on evaluation within these courses. The benefits of specific units of study are likely to be enhanced by a pre-service wholeof-course design that adopts a holistic approach to teacher as researcher and evaluator with a connected and integrated strategy across courses.

Appendix A. Desktop review of research / evaluation units within Australian preservice teacher education courses See Tables A1 and A2.

Table A1 Identification of research / evaluation units of study by course type. Course Types

Fraction of Courses with Research Units Offered

Percentage

Early childhood Early childhood/ Primary combined Primary Primary/ Secondary combined Secondary TOTAL COURSES

15/25 6/11

60 % 54.5 %

16/34 2/6 14/33 52/109

47 % 33.3 % 42.4 % 47.7%

14

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Table A2 Universities reviewed and details of research / evaluation unit and course where found. All early childhood, primary, and secondary courses at these institutions were reviewed with only those where a relevant research / evaluation was found are listed below. University

Course/s

Research Unit (Yes/No)

Research Evaluation Unit (Yes/No)

Queensland Queensland University of Technology

Early Childhood

Yes (Early Childhood Research and Evaluation – EUB402)

Secondary

Yes (Teachers as Researchers – EDUC4700) – – Yes (Policy and Research in Early Childhood – EDEC11029) – focusing on policy analysis – Yes (Teacher as Researcher – EDU415) Yes (Teacher as Researcher – EDU415) Yes (Teacher as Researcher – EDU415) –

Yes (Early Childhood Research and Evaluation – EUB402) No

University of Queensland Griffith University University of Southern Queensland Central Queensland University James Cook University University of the Sunshine Coast Australian Catholic University New South Wales University of New South Wales University of Sydney

Early Childhood Early Childhood Primary Secondary

Early Childhood Primary Secondary

University of Technology Sydney

Primary Secondary K-12

Western Sydney University University of Wollongong

University of Newcastle Southern Cross University Macquarie University Charles Sturt University University of New England Avondale College of Higher Education Alphacrucis College Victoria Monash University (All Bachelor of Education courses include honours)

Early Childhood (birth-5) Primary Early Childhood and Primary

Primary Early Childhood and Primary (Honours) Primary (Honours) Primary and Secondary (Honours) Secondary

University of Melbourne (ONLY OFFERS PATHWAYS TO STUDY A MASTER OF TEACHING)

Victoria University RMIT University Swinburne University of Technology La Trobe University

Deakin University Federation University Australia Box Hill Institute

Master of Teaching (Early Childhood) – coursework progression Master of Teaching (Primary) – coursework progression Master of Teaching (Secondary) – coursework progression Early Childhood Secondary Early Childhood and Primary Primary Secondary Early Childhood Primary Early Childhood and Primary Early Childhood (birth – 5) Secondary

– Yes (Reading and Applying Educational Research – EDUF4044; Research with Young Children – EDUF3136 (OPTIONAL); Internship Research Project – EDEC4006) Yes (Reading and Applying Educational Research – EDUF4044; Research with Young Children – EDUF3136 (OPTIONAL)) Yes (Reading and Applying Educational Research – EDUF4044; Research with Young Children – EDUF3136 (OPTIONAL)) Yes (Professional Experience 5: Teacher as Researcher – 028234) Yes (Professional Experience 5: Teacher as Researcher – 028234) Yes (Professional Experience 5: Teacher as Researcher – 028234) – Yes (Educators as Researchers – EYER301) Yes (Professional Development: The Teacher as Researcher – EDPD301) Yes (Research as Practice – EDUC4182) – – – – – Yes (Introduction to Academic Writing and Research – RES101) Yes (Researching teaching and learning - EDF4100; Research project in education – EDF4101) Yes (Researching teaching and learning - EDF4100; Research project in education – EDF4101) Yes (Researching teaching and learning - EDF4100; Research project in education – EDF4101) Yes (Researching teaching and learning - EDF4100; Research project in education – EDF4101) Yes (Researching Education Practice (EC/EC&P) – EDUC90748; Education Research Project (EC/EC&P) – EDUC90749) Yes (Researching Education Practice (Primary) – EDUC90833; Education Research Project (Primary) – EDUC90769) Yes (Researching Education Practice (SEC) – EDUC90758; Education Research Project (SEC) – EDUC90759) – Yes (Early Childhood Teacher as Researcher – TCHE2541) Yes (Research Methods for Teachers – EDU30031) Yes (Researching Evidence of Impact (Early Childhood of Primary Placement) – EDU4RE2) Yes (Researching Evidence and Impact – EDU4REI) Yes (Researching Evidence and Impact – EDU4REI) Yes (Teacher as Researcher – ECE430) Yes (Literacy Teacher – Researchers in New Times – ECL410) Yes (Teacher as Researcher – EDECE4003) Yes (Action Research Project – PC406) Yes (Major Project – ES729.306)

No No No No

No No No No No No No No No No

No No No No No No No No No No No No No No No No No

(continued on next page) 15

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams

Table A2 (continued) University

Course/s

Research Unit (Yes/No)

Research Evaluation Unit (Yes/No)

Australian Capital Territory University of Canberra South Australia University of South Australia (All EC and Primary Bachelor of Education courses include honours)

– Early Childhood (Honours)

University of Adelaide (Only offers Bachelor of Teaching)

Middle (BTeach) Secondary (BTeach)

Primary (Honours)

Flinders University Northern Territory Charles Darwin University Batchelor Institute of Indigenous Tertiary Education (Higher education courses provided as part of a joint initiative with Charles Darwin Uni) Western Australia Curtin University Edith Cowan University

Early Childhood Primary Secondary Early Childhood Primary

Early Childhood Primary

Murdoch University, South Street Campus

Early Childhood and Primary Primary Secondary

University of Notre Dame Australia

Early Childhood Early Childhood and Primary Primary Secondary

Tasmania University of Tasmania

Early Childhood Primary

Yes (Honours A: Introduction to Research in Education – EDUC4224; Honours B: Research Project – EDUC4225) Yes (Honours A: Introduction to Research in Education – EDUC4224; Honours B: Research Project – EDUC4225) Yes (Professional Practice and Research – EDUC2002) Yes (Professional Practice and Research – EDUC2002; Education Research Skills – EDUC4XXX) – Yes Yes Yes Yes Yes

(Researching (Researching (Researching (Researching (Researching

Practice Practice Practice Practice Practice

– – – – –

EST302) EST302) EST302) EST302) EST302)

– Yes (School-based project – EDU4226 OPTIONAL FINAL SEMESTER UNIT) Yes (School-based project – EDU4226 OPTIONAL FINAL SEMESTER UNIT) Yes (Assessment and Action Research – BED200) Yes (Assessment and Action Research – BED200; Schooling and Society: Research Skills – BED300) Yes (Assessment and Action Research – BED200; Schooling and Society: Research Skills – BED300) + extra research units through the teaching area specialisations Yes (Introduction to Research and Scholarship: Mathematics – EDUC4321) (within specialisation units) (within specialisation units) (within specialisation units) Yes (Teachers as Reflective Practitioners and Action Researchers – ESH304) Yes (Teachers as Reflective Practitioners and Action Researchers – ESH304)

No No No No No

No No No No No No No No No No No

Call, K. (2018). Professional teaching standards: A comparative analysis of their history, implementation and efficacy. Australian Journal of Teacher Education, 43(3), 93–108. https://doi.org/10.14221/ajte.2018v43n3.6. *Campbell, A., & Jacques, K. (2004). Best practice researched: Teachers’ expectations of the impact of doing research in their classrooms and schools. Teacher Development, 7(1), 75–90. https://doi.org/10.1080/13664530400200187. Cochran-Smith, M., & Lytle, S. (1992). Inside/outside: Teacher research and knowledge. New York: Teachers College Press. Cochran-Smith, M., & Lytle, S. (1999). The teacher research movement: A decade later. Educational Researcher, 28(7), 15–25. https://doi.org/10.2307/1176137. *Colucci-Gray, L., Sharmistha, D., Gray, D., Robson, D., & Spratt, J. (2013). Evidencebased practice and teacher action-research: A reflection on the nature and direction of ‘change’. British Educational Research Journal, 39(1), 126–147. https://doi.org/10. 1080/01411926.2011.615389. Darling-Hammond, L. (Ed.). (2005). Professional development schools: Schools for developing a profession. New York: Teachers College Press. Department of Education (2011). Teachers’ Standards Guidance for school leaders, school staff and governing bodies. Retrieved fromUK: Department of Education. https://assets. publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/ file/665520/Teachers__Standards.pdf. Earl, L., & Timperley, H. (2015). Evaluative thinking for successful educational innovation. OECD Education Working Papers No. 122. Available fromhttps://doi.org/10.1787/ 5jrxtk1jtdwf-en. Education Endowment Foundation. (n.d.). Evidence teaching toolkit. Retrieved from: https://educationendowmentfoundation.org.uk/evidence-summaries/teachinglearning-toolkit. Feldhoff, T., Radisch, F., & Bischof, L. M. (2016). Designs and methods in school

References1 Ainscow, M., Dyson, A., Goldrick, S., & West, M. (2016). Using collaborative inquiry to foster equity within school systems: Opportunities and barriers. School Effectiveness and School Improvement, 27(1), 7–23. https://doi.org/10.1080/09243453.2014. 939591. Australia. House of Representatives Standing Committee on Education and Vocational Training (2007). Top of the class: Report on the inquiry into teacher educationCanberra: House of Representatives Pubishing Unit. Australian Institute of Teaching and School Leadership [AITSL] (2012). The Australian professional standards for teachers. Retrieved fromhttp://www.aitsl.edu.au/australianprofessional-standards-for-teachers/standards/list. BERA-RSA (2014). Research and the teaching profession: Building the capacity for a selfimproving education system. London, UK: BERA. *Bradley-Levine, J., Smith, J., & Carr, K. (2009). The role of action research in empowering teachers to change their practice. Journal of Ethnographic and Qualitative Research, 3, 152–161. Buckley, T., Thomas, A., Hargraves, M., & Trochim, W. (2015). Defining and teaching evaluative thinking: Insights from research in critical thinking. The American Journal of Evaluation, 36(3), 375–388. https://doi.org/10.1177/109821405581706. Bryk, A. (2010). Organising schools for school improvement. Kappan, 91(7), 23–30. Calderhead, J. (1989). Reflective teaching and teacher education. Teaching and Teacher Education, 5(1), 43–51. https://doi.org/10.1016/0742-051x(89)90018-8.

1

Classroom Classroom Classroom Classroom Classroom

No No

Note: Review studies are denoted with *. 16

Studies in Educational Evaluation 64 (2020) 100830

A. McFadden and K.E. Williams improvement research: A sytematic review. Journal of Educational Administration, 50(2), 209–240. https://doi.org/10.1108/JEA-07-2014-0083f. Freese, A. (2006). Reframing one’s teaching: Discovering our teacher selves through reflection and inquiry. Teaching and Teacher Education, 22, 100–119. https://doi.org/ 10.1016/j.tate.2005.07.003. Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Fancisco, CA: Jossey-Bass. Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x. *Goodnough, K. (2010). Teacher learning and collaborative action research: Generating a "knowledge-of-practice" in the context of science education. Journal of Science Teacher Education, 21(8), 917–935. https://doi.org/10.1007/s10972-010-9215-y. *Goodnough, K. (2011). Examining the long-term impact of collaborative action research on teacher identity and practice: The perceptions of K-12 teachers. Educational Action Research, 19(1), 73–86. https://doi.org/10.1080/09650792.2001.547694. Gore, J., Barron, R. J., Holmes, K., & Smith, M. (2016). Who says we are not attracting the best and brightest?: Teacher selection and the aspirations of Australian school students. Australian Educational Researcher, 43(5), 527–549. https://doi.org/10.1007/ s13384-016-0221-8. *Hagevik, R., Aydeniz, M., & Rowell, C. G. (2012). Using action research in middle level teacher education to evaluate and deepen reflective practice. Teaching and Teacher Education, 28(5), 675–684. https://doi.org/10.1016/j.tate.2012.02.006. Hajisoteriou, C., Karousiou, C., & Angelides, P. (2018). Successful compomnents of school improvement in culturally diverse schools. School Improvement & School Effectiveness, 29(1), 91–112. https://doi.org/10.1080/09243453.2017.1385490. Harris, J., Carrington, S., Ainscow, M., Coomber, B., Ehrich, L. C., Klenowski, V., et al. (2018). Promoting equity in schools: Collaboration, inquiry and ethical leadership. Milton Park, Abington, Oxon. and New York: Routledge. Harris, A., Chapman, C., Muijs, M., Russ, J., & Stoll, L. (2006). Improving schools in challenging contexts: Exploring the possible. School Effectiveness and School Improvement, 17(4), 409–424. https://doi.org/10.1080/09243450600743483. Harrison, J., Dunn, M., & Coombe, K. (2006). Making research relevant in preservice early childhood teacher education. Journal of Early Childhood Teacher Education, 27(3), 217–229. https://doi.org/10.1080/10901020600843434. Hatch, J. (2006). Teacher research: Questions for teacher educators. Washington DC: National Association for the Education of Young Children. Hatch, A., Greer, T., & Bailey, K. (2006). Student produced action research in early childhood teacher education. Journal of Early Childhood Teacher Education, 27(2), 205–212. https://doi.org/10.1080/1090102600675182. Hattie, J. A. C. (2003). Teachers make a difference: What is the research evidence? October Paper Presented at the Building Teacher Quality: What Does the Research Tell Us ACER Research Conference. Retrieved from http://research.acer.edu.au/research_ conference_2003/4. Hattie, J. (2015). What works best in education: The politics of collaborative expertise. London: Pearson. Institute of Education Sciences. (n.d.). What works clearinghouse. Retrieved from: https://ies.ed.gov/ncee/wwc/. *Jaipal, K., & Figg, C. (2011). Collaborative action research approaches promoting professional development for elementary school teachers. Educational Action Research, 19(1), 59–72. https://doi.org/10.1080/09650792.2011.547688. Making professional development schools work: Politics, practice, and policy. In M. Levine, & R. Trachtman (Eds.). The series on school reform. New York, NY: Teachers College Press. *Lu, M.-T. P., Ward, H. C., Overton, T., & Shin, Y. (2014). The synergetic approach to effective teachers’ research education: An innovative initiative for building educational research capacity in a Hispanic-Serving Institution. Journal of Hispanic Higher Education, 13(4), 269–284. https://doi.org/10.1177/1538192714536188. Mayer, D. (2014). Forty years of teacher education in Australia: 1974–2014. Journal of Education for Teaching International Research and Pedagogy, 40(5), 461–473. https:// doi.org/10.1080/02607476.2014.956536. *Medwell, J., & Wray, D. (2014). Pre-service teachers undertaking classroom research: Developing reflection and enquiry skills. Journal of Education for Teaching, 40(1), 65–77. https://doi.org/10.1080/02607476.2013.864018. Meier, D., & Henderson, B. (2007). Learning from young children in the classroom: The art and science of teacher research. New York, NY: Teacher College Press. Mills, G. E. (2007). Action research: A guide for the teacher research (3rd ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall. *Miranda, R. J., & Damico, J. B. (2013). Science teachers’ beliefs about the influence of their summer research experiences on their pedagogical practices. Journal of Science Teacher Education, 24(8), 1241–1261. https://doi.org/10.1007/s10972-012-9331-y. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009). Preferred reporting items for systematic reviews and meta- analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097. Mulholland, S. (2016). ADHD-specific knowledge and attitudes of teachers (ASKAT): Development and validation of a new research instrument. International Journal of Educational Research, 77, 109–116. https://doi.org/10.1016/j.ijer.2016.03.010. *Myers, N., & Dillard, B. R. (2013). An action research project’s impact on teachers’ leadership attitudes and perceptions. Journal of Collection Teaching & Learning, 10, 69–74. https://doi.org/10.19030/tlc.v10i1.7533. National Board for Professional Teaching Standards (2016). National board standards. Retrieved fromhttps://www.nbpts.org/standards-five-core-propositions/.

*Niemi, H., & Nevgi, A. (2014). Research studies and active learning promoting professional competences in Finnish teacher education. Teaching and Teacher Education, 43, 131–142. https://doi.org/10.1016/j.tate.2014.07.006. *Nimmo, J. W., & Park, S. (2009). Engaging early childhood teachers in the thinking and practice of inquiry: Collaborative research mentorship as a tool for shifting teacher identity. Journal of Early Childhood Teacher Education, 30(2), 93–104. https://doi.org/ 10.1080/10901020902885356. OECD (2013). Teachers for the 21st century: Using evaluation to improve teachingRetrieved from. OECD Publishinghttp://www.oecd.org/site/eduistp13/TS2013% 20Background%20Report.pdf. Patton, M. (2008). Utilization-focused evaluation. Thousand Oaks, CA: SAGE. Pawson, R. (2006). Evidence-based policy. A realist perspective. London: SAGE. Pollard, A. (2002). Reflective teaching: Effective and evidence-informed professional practice. London: Continuum. *Ross, J. R., & Bruce, C. D. (2012). Evaluating the impact of collaborative action research on teachers: A quantitative approach. Teacher Development, 16(4), 537–561. https:// doi.org/10.1080/13664530.2012.734746. Rowell, L. L., Yu Polish, E., Riel, M., & Bruewer, A. (2015). Action researchers' perspectives about the distinguishing characteristics of action research: A Delphi and learning circles mixed-methods study. Educational Action Research, 23(2), 243–270. https://doi.org/10.1080/09650792.2014.990987. Schon, D. (1983). The reflective practitioner. London, UK: Random House. Schwandt, T. A. (2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Stanford, CA: Stanford University Press. Social Ventures Australia (2018a). Evidence for learning. Retreived fromhttps://www. socialventures.com.au/ventures/evidence-for-learning/. Social Ventures Australia (2018b). Evidence for learning toolkit. Retrieved fromhttps:// evidenceforlearning.org.au/the-toolkit/implementation/. Social Ventures Australia (2015). Teachers, meet evidence. The seed for radical transformation? March 13, Retrieved fromhttps://www.socialventures.com.au/blog/ teachers-meet-evidence-the-seed-for-radical-transformation/. Somekh, B., & Zeichner, K. (2009). Action research for educational reform: Remodelling action research theories and practices in local contexts. Educational Action Research, 17(1), 5–21. https://doi.org/10.1080/09650790802667402. Sparks-Langer, G., & Colton, A. (1991). Synthesis of research on teachers’ reflective thinking. Educational Leadership, 48, 37–44. Stremmel, A. (2002). Teacher research: Nurturing professional and personal growth through inquiry. Young Children, 57(5), 62–69. Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, and applications (2nd ed.). John Hoboken, NJ: Wiley & Sons. Tatto, M. T. (2015). The role of research in the policy and practice of quality teacher education: An international review. Oxford Review of Education, 41(2), 171–201. Teacher Education Ministerial Advisory Group (TEMAG) (2014). Action now: Classroom ready teachers. Available online atCanberra, Australia: Australian Government. https://docs.education.gov.au/system/files/doc/other/action_now_classroom_ready_ teachers_accessible.pdf. *Tosun, C. (2014). Pre-service teachers’ opinions about the course on scientific research methods and the levels of knowledge and skills they gained in this course. Australian Journal of Teacher Education, 39(10), 96–112. https://doi.org/10.14221/ajte. 2014v39n10.7. *van der Linden, W., Bakx, A., Ros, A., Beijaard, D., & van den Bergh, L. (2015). The development of student teachers’ research knowledge, beliefs and attitude. Journal of Education for Teaching, 41(1), 4–18. https://doi.org/10.1080/02607476.2014. 992631. *van der Linden, W., Bakx, A., Ros, A., Beijaard, D., & Vermeulen, M. (2012). Student teachers’ development of a positive attitude towards research and research knowledge and skills. European Journal of Teacher Education, 35(4), 401–419. https://doi. org/10.1080/02619768.2011.643401. *van Ingen, S., & Ariew, S. (2015). Making the invisible visible: Preparing preservice teachers for first steps in linking research to practice. Teaching and Teacher Education, 51, 182–190. https://doi.org/10.1016/j.tate.2015.07.001. *Vrijnsen-de Corte, M. C. W. M., den Brok, P. J. P., Kamp, M. J. M., & Bergen, T. C. M. T. (2013). Measuring teachers’ and student teachers’ perceptions of practice-based research in PDS and non-PDS settings. Teaching and Teacher Education, 36, 178–188. https://doi.org/10.1016/j.tate.2013.07.006. Walkington, J. (2005). Becoming a teacher: Encouraging development of teacher identity through reflective practice. Asia-Pacific Journal of Teacher Education, 33(1), 53–64. https://doi.org/10.1080/1359866052000341124. Wandersman, A. (2014). Moving forward with the science and practice of evaluation capacity building (ECB): The why, how, what, and outcomes of ECB. The American Journal of Evaluation, 35, 87–89. https://doi.org/10.1177/1098214013503895. White, S., Nuttall, J., Down, B., Shore, S., Woods, A., Mills, M., et al. (2018). Strengthening a research-rich teaching profession for Australia. Australian Teacher Education Association (ATEA). Canberra: Australian Association for Research in Education (AARE), Melbourne: Australian Council of Deans of Education, (ACDE) Canberra. Willegems, V., Consuegra, E., Struyven, K., & Engels, N. (2017). Teachers and pre-service teachers as partners in collaborative teacher research: A systematic literature review. Teaching and Teacher Education, 64, 230–245. https://doi.org/10.1016/j.tate.2017. 02.014. Zeichner, K. M., & Liston, D. P. (2014). Reflective teaching; an introduction (2nd ed.). New York and London: Routledge.

17