Higher education teachers’ professional learning: Process and outcome

Studies in Educational Evaluation 46 (2015) 92–101

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/stueduc

Higher education teachers’ professional learning: Process and outcome Alenoush Saroyan a,*, Keith Trigwell b a b

McGill University, Educational and Counselling Psychology, 3700 McTavish Street, Montreal, QC, Canada, H3A 1Y2 University of Sydney, Sydney, NSW, Australia

A R T I C L E I N F O

A B S T R A C T

Article history: Received 24 December 2014 Received in revised form 10 March 2015 Accepted 26 March 2015 Available online 17 April 2015

In this paper, we first offer an analytic perspective on the papers in this volume, framing our discussion within the context of academic development as a discipline and highlighting common lines that cut across the nine contributions. We then offer insights about the current state of research on the evaluation of academic development and suggest directions that are likely to advance our theoretical and empirical knowledge in this area and anchor us firmly in evidence-based practice. We suggest that clarifying the mechanisms of change when professional learning occurs and measuring the changes in student learning outcomes that are associated with professional learning are important avenues for future research. ß 2015 Elsevier Ltd. All rights reserved.

Keywords: Professional learning Academic development Educational development Faculty development Teaching for learning Higher education

Introduction The necessity for improving quality teaching has never been as compelling or as challenging as it is today, in an ever-changing higher education climate. New trends continually define higher education, crossing institutional and national borders. They impact the way effective teaching is conceptualized, fostered and supported, evaluated, valued, and rewarded (Kezar & Eckel, 2004; OECD, 2013; Ramsden, 2003; Saroyan, 2010; Saroyan & Amundsen, 2001). The papers in this volume highlight why teaching quality is so important at this particular time. They point to inadequate preparation for academic work in graduate studies, the inability of faculty to transfer skills, the increasing complexity of academic environments, institutional expectations and accountability, the necessity to better prepare students with diverse needs, and the need to keep in step with shifts in knowledge and ongoing changes in the vocations. Professional learning, a fundamental cornerstone of dynamic, learning organizations (Garvin, 1993; Senge, 1990) and organizational learning (Cohen & Sproull, 1996; March, 1991) is an effective way to address the need for effective teaching. Dill (1999) suggests that university teaching and learning centres were created to provide ‘‘. . .structural support for organizational learning . . .’’ (p. 139). Indeed, the mandate of such centres coalesce around two

* Corresponding author. Tel.: +1 5143985329; fax: +1 5143986968. E-mail address: [email protected] (A. Saroyan). http://dx.doi.org/10.1016/j.stueduc.2015.03.008 0191-491X/ß 2015 Elsevier Ltd. All rights reserved.

types of activities: enhancing teaching and learning capacity and leadership in the academic community, and advocating for teaching and learning quality by drawing on evidence-based practice (Be´dard, Clement, & Taylor, 2010; Chalmers and O’Brien, 2004; Gosling, 2009; Land, 2004; Grabove et al., 2012; Saroyan & Amundsen, 2004; Saroyan & Frenay, 2010). The papers in this volume offer the reader a broad, albeit porous perspective on the literature on academic development, and an exposure to the microcosm of activities designed and implemented to foster professional learning about teaching. Our contribution to this volume is twofold. First, we offer an analytic perspective on the papers. We frame this discussion within the broader context of academic development: what the literature in general and the papers in this volume tell us about the state of our field as a discipline, and common lines that cut across the nine contributions. We then offer our insights about the current state of research on the evaluation of academic development and suggest directions that are likely to advance our theoretical and empirical knowledge in this area and more importantly, anchor us firmly in evidence-based practice. An analytic perspective Framing academic development as a discipline Well-established academic disciplines share a number of attributes. These include the use of common terminologies and technical terms, a dominant paradigm to orient the research,

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

shared theories and concepts that can organize the body of knowledge and provide direction for further advancement, common research methods, a body of specialized knowledge, and institutional presence in the form of academic programs taught in universities and professional associations affiliated with the discipline (Krishnan, 2009; Kuhn, 1962). Given these characteristics, what do the literature and the papers in this volume tell us about where we stand as a discipline? Our first observation is that a plethora of terms are used in the chapters to describe formative processes intended to foster improved pedagogies and teaching. The terms used in the chapters and those used in the broader literature (see for example, Amundsen & Wilson, 2012; Stes, Min-Leliveld, Gijbels, & Van Petegem, 2010; Taylor & Be´dard, 2010) include ‘‘faculty development’’, ‘‘educational development’’, ‘‘academic development’’, ‘‘instructional development’’, ‘‘professional development’’ and ‘‘professional competence’’. These are sometimes used synonymously and sometimes differently, rendering it difficult to make sense of findings and to generalize based on a cumulative body of knowledge. We also note a preference not to use the term ‘‘development’’ and ‘‘developer’’ so as not to undermine the role and agency of academics in the development process. van Schalkwyk, Herman, Leibowitz, and Farmer (2015) refer to professional development as a ‘‘catchall phrase’’ and following the recommendation of others (see for example, Trowler & Knight, 1999), suggest the use of ‘‘professional learning’’ in reference to activities that result in enhancing teaching and learning. We agree with this definition and from hereon, use the term professional learning to denote activities and processes that academics engage in to ameliorate their academic performance and the impact of their performance on student learning. While academic performance cannot be limited just to teaching and pedagogical matters, to remain aligned with the theme of this volume, we do not extend our discussion to performance related to research and service, which broadly constitute the other two dimensions of academic performance. Later on in this chapter, we draw similarities between the outcome of professional learning with Shulman’s (2005) notion of apprenticeships that result in thinking and acting responsibly in professional contexts. Our second observation is the variation in goals of professional learning projects elaborated in the papers in this volume. These goals cluster around changing conceptions and beliefs, learning about and applying new pedagogical skills, fostering reflection based on feedback from multiple sources and processes, developing a professional identity, fostering engagement in the scholarship of teaching and learning, and networking and community building. Again, this spread is more or less in line with what is reported in the literature, in particular in a framework proposed by Amundsen and Wilson (2012). Following a comprehensive and critical literature review, they identify clusters of ‘‘educational development’’ initiatives coalescing around skills, methods, reflection, the institution, disciplinary, and action research/inquiry. Amundsen and Wilson (2012) specify that skills, methods, and institutional focus clusters are outcome oriented while reflection, disciplinary, and action research or inquiry focus clusters are process oriented (p. 107). Two important points are to be made here. One is that the focus of an ‘‘educational development initiative’’ is seldom one-dimensional. For instance, Taylor and Znajda’s (2015) project aims to foster course design knowledge (skills), change in conceptions and reflective practice (reflection), institutional and community development (institutional), and interest in documenting professional learning (inquiry) (Taylor & Znajda, 2015). The second is that Amundsen and Wilson’s (2012) framework is not necessarily comprehensive and may not account for all processes and outcomes of educational development initiatives. Nevgi and Lo¨fstro¨m’s (2015) reported initiative is a case in point. In their study, they explored the potential of a formal University Teaching Development (UTD) program comprising

93

credited courses on basic and subject studies in university pedagogy (teaching and learning, course alignment, assessment, curriculum design, discipline specific approaches, research in higher education, and practical training) as a means of developing teacher identity. This construct is not included in Amundsen and Wilson’s (2012) framework. Indeed, the often multidimensional focus of professional learning programs, and the need to target new goals as the higher educational climate changes (for example, an emerging need to address fiscal constraints which may lead to larger enrollments and require greater efficacy in teaching large classes) highlight the complications inherent in designing research on academic development and the challenges involved in developing a common body of knowledge to advance the field. A third observation is the spread of the reported studies on the deductive–inductive continuum. Some (Hum, Amundsen, & Emmioglu, 2015; Nevgi & Lo¨fstro¨m, 2015; van Waes, van den Bossche, Moolenaar, Stes, & van Petergem, 2015) outline a theoretical or conceptual framework and the specific way in which it has informed the design and the interpretation of findings. Others report baseline research that is fundamental to theory building (e.g., Chalmers et al., 2015; van Schalkwyk et al., 2015). The value of both types of research cannot be overemphasized. However, it is also important to remind ourselves that there is a perennial shortcoming in our field with respect to theoretically driven research (see for example Steinert et al., 2006; Stes et al., 2010). Amundsen and Wilson (2012) even assert that little has changed with respect to theoretically driven research since the seminal review and recommendations offered by Levinson-Rose and Menges (1981) when they conducted the first critical review of the faculty development literature four decades ago. Clearly, to advance the field, we need to conceptualize and conduct studies that are theoretically driven and are conceptually and methodologically robust enough to contribute to theory development. The fourth and final observation is the variation in methodological approaches and the extent to which reliability measures and data treatment have been elaborated to convey trustworthiness and to readily support warrants for claim. For instance, reporting inter-rater reliability procedures and coefficients (e.g., Cohen’s Kappa, Krippendorff’s alpha), and specifying attempts to triangulate data would help establish a clearer relationship between the initiatives and their impact on professional learning. Samples size and data sources also require greater attention. In line with much of the literature on educational development, studies reported in this volume are conducted on a small scale and involve primarily self-reports. One exception is the study conducted by van Waes et al. (2015). Their use of social network theory to capture the extent to which participation in a professional learning initiative fosters networking demonstrates the potential of new methodologies and data sources, especially when the intent of professional learning extends to community building. While small-scale studies with robust methodologies have their own merit and value, especially if they use multiple data sources, they still require complementary large-scale correlational studies to render findings generalizable. Converging themes Five common lines cut across the papers in this volume. We highlight them below and draw attention to their prevalence. Institution The first common line is the role of the institution in supporting initiatives that foster improved teaching. In Amundsen and Wilson’s (2012) framework, the ‘‘institution’’ cluster signals a ‘‘coordinated plan for academic development at the institutional

94

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

level. . .[placing] emphasis on strategic planning and human resource development . . .[is] top-down and in response to an institutional or national agenda and sometimes involves multiple universities’’ (p. 101). This kind of support is evident in all the papers in this volume. We see for example institutional support of centres for teaching and learning (Hum et al., 2015; Taylor & Znajda, 2015), targeted funds to advance the scholarship of teaching (Hum et al., 2015), academic programs for faculty to gear teaching practice towards a particular educational approach embraced by the university administration and convening of task forces to do the ground work for implementing a particular policy or support a curriculum reform (Bickerstaff & Cormier, 2015; Hoekstra & Crocker, 2015; Nevgi & Lo¨fstro¨m, 2015), and a national project to identify the impact and effectiveness of teacher development programs and activities of teaching and learning development centres in Australian universities (Chalmers et al., 2015). Institutional support is heartening as it provides concrete evidence of a desire for organizational learning and helps us reconceptualize professional learning as a goal for the community at large rather than something intended for a limited number of individuals within the community (Laurillard, 1993). Small scale and voluntary participation A second common pattern in the papers that describe a professional learning program is the small scale, involving between 10 and 35 participants, and the voluntary nature of participation of faculty in these programs. One might argue that scale will depend on whether the initiative is classified as a onetime-event or one that is longer (Stes et al., 2010). One-time events are more likely to be scalable but they have less of a chance to make a profound impact on program participants. Conversely, longer initiatives are more likely to have an impact but at the same time, are organized on a very small scale to make it manageable with typically available resources. An exception may be where certification in university teaching and learning is a requirement before taking on an academic position. Scale has also implications in assessing impact. As one of the six core principles of teaching improvement, The Carnegie Foundation for the Advancement of Teaching (n.d.) recommends keeping the intervention at a scale where measuring impact is possible. Mechanism that foster professional learning A third common line pertains to mechanisms deployed for fostering professional learning for teaching. The literature alerts us about (a) the value of reflection in improving teaching, especially when it is based on feedback from multiple sources, including student course ratings, peers, and experts (McAlpine & Weston, 2000; Scho¨n, 1983), (b) the benefits of engaging in the scholarship of teaching and learning (Hutchings & Shulman, 1999; Shulman, 1987; Trigwell & Shale, 2004), (c) the role of communities of practice, networking, and peer interaction in supporting professional learning (Palmer, 1998; Simons & Ruijters, 2004; van Schalkwyk, Cilliers, Adendorff, Cattell, & Herman, 2012; Wenger, 1998), (d) the importance of situated learning in facilitating transfer (Brown, Collins, & Duguid, 1989; Lave & Wenger, 1991), and (e) the necessity and significance of developing a professional identity (McAlpine, Amundsen, & Turner, 2013). Aspects of all these mechanisms are present in the collections of papers in this volume and are highlighted below. Reflection We observe that reflective practice appears as both a process and an intended outcome in some chapters. For instance, Taylor and Znajda (2015) engage their participants in a course design workshop and peer consultation process and observe increased awareness of self and enhanced reflection on teaching practice. In

this study, reflection is fostered during the intervention but also highlighted as a targeted outcome or a ‘‘dependent variable’’. Hoekstra and Crocker (2015) describe the design, implementation and evaluation process of an ePortfolio approach and the extent to which the approach is adopted and valued and effects professional development. They conclude that feedback compiled from multiple sources including self, students, peers, external experts, and support staff, foster ‘‘a naturally occurring professional learning process’’, ‘‘increased awareness of . . .practice’’, and the potential of the ePortfolio approach ‘‘. . . to enhance reflective practice’’ (p. 5). In this study, reflection is not directly targeted as an outcome but is fostered through the process of discussing feedback. It is interesting to note that while not all participants in this project find the activity worthwhile, most indicate a positive view of using feedback from different sources to foster reflection. Engaging in the scholarship of teaching and learning One manifestation of the scholarship of teaching and learning (SoTL) reported in the literature (see for example, Brew, 2011; Kreber, 1999; Trigwell & Shale, 2004) is when academics ‘engage in content, process and premise reflection on research-based and experience-based knowledge in the areas of instruction, pedagogy and curriculum in ways that can be peer reviewed’’ (Kreber, 2002, p. 153). An example in this volume that underscores the intentionality of fostering SoTL is the TGDF project, reported by Hum et al. (2015). The grants, awarded to professors through this project are intended ‘‘for instructors to identify questions about teaching and learning of interest to them, conduct a systematic investigation, and to share their findings with colleagues.’’ The other manifestation is more organic in nature; an unplanned outcome of engaging in a professional learning activity that nurtures the desire to share gained experiences with other colleagues. We encounter this in the Taylor and Znajda (2015) paper. Communities of practice The notion of community development is implicit in some of the reported projects (e.g., Hoekstra & Crocker, 2015; van Schalkwyk et al., 2015), salient in others (e.g., Hum et al., 2015; Taylor & Znajda, 2015) but central to the study reported by van Waes et al. (2015). They explore if and how social interaction patterns of faculty shift before, during, and after an instructional development program and look for evidence of mediating or constraining factors. The most intriguing aspect of this contribution is the applied methodology and the promise it holds for future research on peer impact on professional learning. The limitation, as mentioned, is establishing a correlation between growth in networks with professional learning about teaching. Situated learning to facilitate transfer As we see in the projects described by Bickerstaff et al. (2015), Hoekstra et al. (2015), and Hum et al. (2015), project implementation is predicated on the availability of and access to experts, consultants, network of colleagues and other support mechanisms during the process. Bickerstaff et al. (2015) describe the real time support offered during an instructional reform in their multicampus college setting. They highlight the availability of ‘‘leaders who had developed the reform in their own classrooms’’ and could effectively communicate their personal experiences and the importance of appreciating the dynamic aspect of a phenomenon such as an instructional reform, which constantly brings to fore new questions that require prompt and just-in-time answers. Finally, they note that the authentic context of teaching in courses will enable faculty to ask more profound questions relevant to their pedagogical needs and these may require new or different kinds of scaffolding as they arise. The ‘‘task-force’’ designated to assist the implementation of the ePortfolio project described in the

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

Hoekstra et al. (2015) chapter is also an indication of the value attached to situated learning and the provision of requisite support to facilitate the resolution of emerging issues. Similarly, Hum et al. (2015) emphasize, as one of their project goals, ‘‘engag[ing] instructors in teaching as a socially situated practice’’. As part of their design, they provide on-going one-to-one feedback during the entire time that instructors develop a proposal of their choice to enhance their teaching. Professional identity The teacher identity of academics is a relatively recent topic of investigation and as such, it is not surprising to see it highlighted in only one chapter in this volume. Nevgi and Lo¨fstro¨m (2015) explore identity development within the context of a formal university teacher development (UTD) program, using ‘‘academics’ self-image and self-efficacy as teachers, motivation to teach and develop as a teacher, and task perception’’ (p. X) as measures of this identity. Their intriguing finding is that despite a common and rather lengthy UTD program, four different types of teacher identity are observed at the end of the program. More importantly, two of these identities correspond with that of an educational developer even though none of the participants was active in such a role prior to participating in the UTD. Interestingly, though not surprisingly, both reflection and peer interaction emerge as powerful mechanisms for fostering the development of a teacher identity. Conditions that foster and hinder uptake A fourth pattern is the attention paid to conditions that foster or hinder uptake. New learning necessitates change and the human condition cannot be ignored in this learning process. As Fullan (1993) noted insightfully decades ago, change cannot be mandated: ‘‘. . . you cannot make people change. You cannot force them to think differently or compel them to develop new skills’’ (p. 23). What we can do, however, is to draw on the psychological literature to understand what conditions best enable or are likely to hinder professional learning. Classical theories of achievement, motivation, and self-determination (see for example Atkinson, 1964; Bandura, 1989; Eccles et al., 1983; Ryan & Deci, 2000; Weiner, 1979) as well as Mezirow’s (1991) transformative adult learning theory are good starting points to explore the notion of change. Enabling conditions Drawing upon the theories listed above, Centra (1993) provides a useful operational model of change1 with respect to faculty learning. Briefly, the model highlights four conditions: (a) individuals are likely to acquire new knowledge if they acknowledge a gap and feel the need for acquiring new knowledge, (b) individuals must value and appreciate the relevance of the learning opportunity and feel that what is gained from participating in a professional learning activity is worth the invested resources (e.g., time, money), (c) the teacher must know how to change and this is best mediated by credible individuals, and (d) there must be an element of intrinsic and/or extrinsic motivation such as personal satisfaction or external rewards to facilitate change. We observe a manifestation of some but not all four conditions in the papers in this volume. The acknowledgement of a gap in one’s knowledge base, the sense of need for acquiring new knowledge to address this gap, and ultimately the drive from within to initiate learning are discussed in terms of agency, volition, and control. van Schalkwyk et al. (2015) highlight the importance of ‘‘personal’’ and ‘‘intentional’’ choices and provide an excerpt from one of their study participants that underscores the 1 The acronym ‘‘NVMH’’ was used for this model. The letters represent the first letter of each of the four conditions, highlighted in italics.

95

importance of agency: ‘‘I make learning about teaching a priority, but that is MY choice, not that of my faculty of department.’’ (p. 4). Hoekstra and Crocker (2015) intentionally incorporate the use of personal development plan to give control of the learning to their teacher participants and to allow them to ‘‘formulate goals for future growth based on their feedback and reflection’’ (p. 3, 4). The notion of a ‘‘personal plan’’ is also respected in the project initiated by Hum et al. (2015) as well as by Nevgi and Lo¨fstro¨m (2015). We note the condition of value and relevance throughout the papers. Boerboom, Stalmeijer, Dolmans, and Jaarsma (2015) refer to this as ‘‘tipping points towards change’’ and in their literature review paper, draw attention to prompts that have triggered change in clinical teachers: One was realising that there indeed was a problem concerning their teaching and something should be done with or without support from others. . . Additional tipping points were reached when clinical teachers recognised the benefits of change and felt confident to do something with the feedback (p. 8). Bickerstaff and Cormier (2015) elaborate on the importance of aligning professional learning opportunities offered to support the reform project in their college with teachers’ questions about the reform. They assert that ‘‘Workshops can fall flat if not aligned with faculty questions’’ (p. 20), and in their conclusion, talk about the ‘‘. . . limitations of . . . decontextualized conversation about teaching strategies’’ (p. 21). Clearly, the needs of individuals vary and random professional learning activities will not have the same relevance to everyone. We observe this varying need in the response of the sample in van van Schalkwyk et al.’s (2015) study. ‘‘The lecturers had differing perspective [italics added] with regard to the sort of support that might enable their professional learning.’’ (p. 9). The importance of knowing how to change and using credible and knowledgeable individuals to lead professional learning activities and workshops who have encountered experiences similar to those that the audiences are struggling with cannot be understated. Credible change agents and leaders exude confidence and can inspire and motivate even the most skeptical faculty. Conversely, unprepared and inexperienced leaders can quickly turn off the most interested faculty, but more importantly, they can put themselves in an uncomfortable zone. We hear about such discomfort when faculty leaders in Bickerstaff and Cormier’s (2015) project engage in professional development activities without having the necessary expertise or experience: ‘‘We didn’t want to tell them this is what you have to do because we didn’t really know what we wanted them [college teachers] to do anyway’’ (p. 17). Equally important is the availability of on-going support along the way, especially at implementation time. We note, for instance, the planned availability of on-going expert support throughout some initiatives (Hum et al., 2015; Taylor & Znajda, 2015) and the continuous involvement of a designated and knowledgeable taskforce who is open to learn about new issues and concerns as they arise and is able to address them just-in-time, in an ‘‘iterative design and evaluation process’’ (Hoekstra and Crocker, 2015). Finally, with respect to motivation, in addition to personal satisfaction, the extent to which an institution values and rewards teaching and learning can be a drive and extrinsic motivation for academics to prioritize professional learning amongst other academic duties. This will be evident from the resources the institution invests towards the advancement of teaching as well as from its mission statement, and its policies regarding tenure and promotion. van Schalkwyk et al.’s (2015, p. 10) highlight this aspect and warn that, inevitably, factors that weigh highly in career advancement are those that faculty dedicate greater time to and these, especially in research intensive institutions, are research related.

96

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

Hindering conditions While one might consider the absence of any of the above four enabling conditions inhibiting for professional learning, it is also useful to take into account other mitigating factors that Centra’s (1999) model does not account for. For this, we can refer to Windschitl’s (2002) notion of ‘‘dilemmas’’. Elaborated in the context of the adoption of constructivism in schools, Windschitl (2002) suggests that conceptual, pedagogical, cultural and political dilemmas may hinder uptake. These respectively deal with the difficulty of reconciling one’s personal beliefs with a promoted idea that is different, not knowing how to apply newly gained conceptual knowledge, not daring to contradict traditional, efficient, classroom routines, and not knowing how to introduce to students and other stakeholders new notions of the kinds of learning and activities that are to be valued. We observe manifestations of these dilemmas by the college teachers in Bickerstaff and Cormier’s (2015) institution; in the types of questions the teachers articulate about the curriculum reform in their institution and in their half-hearted buy-in into the reform process. From Windschitl’s (2002) perspective, the four question sets that Bickerstaff and Cormier (2015) have derived from their data are either pedagogical or cultural. For instance, questions about classroom practice (e.g., ‘‘Should the same text be used across both course components? How much time will I need to spend on each topic?’’ (p. 11)) and those about student learning (e.g., What will I do if I am not lecturing? What class activities will help my students understand fractions?’’ (p. 12)), would fit in the pedagogical dilemma category. Similarly, questions about the purpose and nature of the reform (e.g., ‘‘. . . implicit and explicit interrogations of the reform’s theory of action’’) and questions about reform implementation (e.g., How will we explain and market the new course to students?’’) would be classified as a cultural dilemma. These types of questions and the dilemmas they represent are common in projects that aim to bring about change in personal and organizational practice and Centra’s (1999) model as well as Windschitl’s (2002) dilemmas are valuable references when planning for professional learning and successful uptake. Assessing and documenting impact A final converging theme is the attempt to document change and measure impact. The literature recommends that the impact of professional learning in general and teaching improvement activities in particular be measured at multiple levels, no matter what the goal of the development program. For instance, Kirkpatrick’s (1994) training evaluation model specifies four levels, three of which are related to participants’ reaction, learning, and behavior, while the fourth is related to the results of the learning on the organization (i.e., cost, productivity, and return on investments). Guskey’s (2000) model, which is slightly more elaborate and more relevant to the academic context, includes the added level of organizational support and utilizes change and student learning instead of Kirkpatrick’s ‘‘results’’ dimension. Frameworks proposed by Gibbs and Coffey (2004), and Chism and Szabo (1998) also highlight the necessity to assess the impact of professional learning of teachers on the learning and development of students they teach. The paper by Chalmers and Gardiner (2015) offers the most recent addition to models intended to assess the impact of professional learning initiatives. Building on their own previous research (Chalmers & O’Brien, 2004) and the existing literature, their proposed model takes into account the focus of professional development programs and allows for the inclusion of contextually relevant indicators for measuring input, process, output and outcome. A framework such as this, while flexible, can also serve as a tool to operationalize the illusive indicators of quality teaching.

If we apply the Chalmers and Gardiner (2015) proposed framework to the studies reported in this volume, we observe certain patterns. Outcomes focus is on: (a) teacher knowledge, skills and practice, (b) teachers’ reflective practice and the scholarship of teaching, (c) policy, (d) (only implicitly about) resourcing, and (e) culture. Types of indicators are primarily input and process related, and output related only insofar as reporting the number of participants who have benefited from the program. Outcome indicators, especially as they relate to the ‘‘value added from learning experiences’’ (Chalmers et al., 2015) are not present. These shortcomings can be redressed by providing more detail on the measures used to ascertain the extent to which intended outcomes have been achieved in the described studies, by explicitly linking ‘‘. . . conceptual and theoretical grounding, core characteristics of the design, and learning’’ (Amundsen & Wilson, 2012, p. 94), and eventually, by extending the assessment of impact to student learning. Future research on the evaluation of academic professional learning In the second half of this paper, we focus on the future research directions provided by the authors of the papers in this volume, provide additional research models that might be used to organize these and new research directions and then argue for future research to focus more on three key areas. The first of the three areas is the means by which changes in student learning outcomes can be measured and the measurement of those changes. The second is the formative evaluation research that aims to clarify the mechanisms of change. The third is the gap in research at the institutional level identified by Simon and Pleschova (2013) in a review of the impact of professional learning programs. Many research ideas likely to be beneficial to the understanding of teachers’ professional learning in post-secondary education are proposed by the authors in the papers in this volume. They include ways in which changes in networking, reflection and academic identity might be used to measure the impact of professional learning (Hum et al., 2015; Nevgi & Lo¨fstro¨m, 2015; Taylor & Znajda, 2015; van Waes et al., 2015); connections between professional learning opportunities and changes to pedagogy and student learning (Bickerstaff & Cormier, 2015; Taylor & Znajda, 2015); and barriers preventing successful professional learning (Hoekstra & Crocker, 2015; van Schalkwyk et al., 2015). In the next section, three research models from the student learning literature are used to illustrate how, in their adaptation, new research ideas might be generated and used to inform future research directions. These include: (a) Hounsell and Anderson’s (2005) ways of thinking and practicing, (b) models that look at relations between aspects of the learning processes and learning outcomes (for example, constructive alignment (Biggs, 1996), and (c) workplace learning research (Kyndt & Baert, 2013). There are also a number of ways in which future research might be conducted to enhance reliability, validity and usability. These include utilizing complex and robust methodologies and analytic approaches to follow impact over time; applying measures of, or proxies for, changes in student learning; and more use of approaches that capture the impact for both individuals and whole groups. Here we deal with these topic areas and the methodology together, on the assumption that the way the research is conducted will be, in large part, related to the research question being asked. Adaptation of existing research models Bickerstaff and Cormier (2015) note that additional research is needed to make connections between professional learning

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

opportunities and changes to pedagogy—a study of relations between context and outcomes. Inquiry of this type might be informed by the extensive research that has been conducted into the student learning experience. These include relations between a wide range of variables such as motivation/self-efficacy and learning outcome. Most of the currently known research has been summarized by Richardson, Abraham, and Bond (2012) and is presented in five broad categories – personality traits, motivation factors, self regulatory learning strategies, students’ approach to learning and psychosocial contextual influences – and how they relate to learning outcomes. What is provided in this summary are models for investigating a range of professional learning factors and how they might be related to variation in the outcomes of professional learning. For example, Taylor and Znajda (2015) note that their case study suggests that future research on the alignment between stated teaching goals and actions is a rich area for educational development practice and research that can inform both practice and theory.van Waes et al. (2015) advocate the ‘‘Exploration of networking and collegiality as impact measures’’, and note that to do so, ‘‘. . . further research into the content and quality of relationships throughout instructional development programs is needed.’’ Ideas on what to include in such studies and ways to reveal additional research directions could be generated using existing learning alignment models. For example, application of Biggs’ (1996) constructive alignment model would suggest that questions on the relations between professional learning goals/expectations/teaching aims, the resources/methods used to achieve them, and the institutional values and systems (such as student evaluation of teaching items, promotion criteria, and reward systems) used in assessing or judging faculty staff could (should) be explored. In their identity study, Nevgi and Lo¨fstro¨m (2015) suggest that further research is needed to identify whether four teacher identity types found among academics who had engaged with a professional learning program might also be found among academics who had ‘‘limited or no pedagogical training’’. This type of research fits with, and might be further informed by the concept of ways of thinking and practicing (WTP). A WTP approach generates the question: In what ways do post-secondary teachers think about teaching and go about their teaching practice? In professional learning, academic staff learn ways of thinking and practicing that are characteristic of, and particular to, their own subject areas, and contain many facets that need further research. Hounsell and Anderson (2005) saw that for students: These ways of thinking and practising were not confined to knowledge and understanding, but could also take in subjectspecific skills and know-how, an evolving familiarity with the values and conventions governing scholarly communication within the relevant disciplinary and professional community, and even a nascent meta-understanding of how new knowledge within the field was generated. (pp. 1–2) The similarity with professional learning can be seen in Shulman’s (2005) description of a synthesis of three apprenticeships: A cognitive apprenticeship wherein one learns to think like a professional, a practical apprenticeship where one learns to perform like a professional, and a moral apprenticeship where one learns to think and act in a responsible and ethical manner that integrates across all three domains. (p. 3) Most of the studies reported in this volume include examples of WTP-based research and refer to the cognitive, practical and moral dimensions described by Shulman (2005). Boerboom et al. (2015)

97

ask for research to determine what is needed to sustain new attitudes and behaviors (in clinical supervision). Taylor and Znajda (2015) report an increase in reflection on and confidence about teaching among those who participated in a 5-day workshop on course design and teaching complemented with three cycles of classroom observation and feedback, and a monthly discussion group on teaching. Both sets of comments are asking for the accumulation of better knowledge of the changes in teachers’ ways of thinking and practicing. Access to this knowledge will enable these characteristics of the context and the teachers to be taken into account when assessing the impact of professional learning. There are many aspects of teachers’ ways of thinking and practicing, and relations between them that are yet to be thoroughly explored and that may provide insights into successful professional learning programs. A brief list would include personal aspects (reflection, critical thinking, self-esteem, identity, conceptions of teaching and learning, personality traits, approaches to teaching, stress and anxiety, organization, goal commitment, motivation, conceptions of assessment and teachers’ knowledge, skills, and behaviors), and institutional aspects (social support, collegiality, scholarship of teaching and learning, institutional integration). Professional learning happens in the workplace, and student learning models have been used in workplace learning research. For example, Vanthournout, Noyens, Gijbels, and Van den Bossche (2014) used variables from students’ experience of their learning environment and approaches to learning to develop a workplace climate questionnaire. While the student learning environment is not quite the same as the context of work, the study illustrates an example of how such an adaptation can trigger the consideration of important workplace factors. A similar adaptation taken into the teaching environment might involve teaching matters such as promotion and tenure requirements, politics, departmental cultures, and the constant need to secure funds. Boerboom et al. (2015) adopt a variant of this approach in their discussion of learning through feedback in the clinical workplace. Kyndt and Baert (2013) list many factors that might impact work-related learning, and more general workplace studies such as these might provide an equally good starting point for research into professional learning by teachers in the workplace. In suggesting approaches that are informed by the student learning research, we also appeal for those approaches to accommodate the rigorous methodologies that are an integral part of that published research. One aspect is inductive coding approaches which are valuable but up to a point. If we are to push the field forward, we ought to increasingly utilize existing conceptual and theoretical frames to inform the design of studies, to conduct deductive analysis, and to interpret results. Even with such rigour, to have more meaningful information on relations between teacher thinking and practice does not amount to meaningful impact research if the purpose of professional learning in instruction is improved student learning. This topic is taken up in the next section. Changes in student learning outcomes If the aim of the evaluation research is summative, the impact on students must be included, since professional learning support units in universities are established, in part, to engage in processes that lead to the improvement in student learning. Their purpose is not just to facilitate better networking, to enhance reflection, to foreground identity, to enrich teaching techniques and broaden conceptions. These are means to student learning ends, not ends in themselves. If achieving them does not lead to any change in student learning or student learning is not factored in when impact is measured, it could be argued that they have little value beyond

98

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

teacher satisfaction, maintaining currency in pedagogic practice, or academic career progression. The literature on teaching is unambiguous about principles of effectiveness and strongly favors learner oriented (APA, 1997), student centered approaches (Prosser & Trigwell, 1999; Trigwell & Shale, 2004). While the shift from teacher centeredness is noteworthy, what is more important is the inextricable connection between teaching and learning. ‘‘Good teaching and good learning are linked through the students’ experiences of what we do. It follows that we cannot teach better unless we are able to see what we are doing from their point of view’’ (Ramsden, 2003, p. 84). A recent initiative of the Organization for Economic Cooperation and Development (OECD), the Assessment of Higher Education Learning Outcomes (AHELO) feasibility study, which looks at graduate outcomes as evidence of quality teaching (Tremblay, Lalancette, & Rosevere, 2012), is an example of an international level effort to link quality teaching to student learning. There is no question that the measurement of change in student learning is a difficult research task, as shown by the work of Stes and van Petegem (2013) and the AHELO project, but this should not be seen as a deterrent. There have been attempts and some progress has been made. Two recently published research reports on this topic are worthy of mentioning for both the scope of their work and the methodological approaches used. Stes and van Petegem (2013) employed rigorous research approaches in a series of studies on professional learning. In one aimed at identifying the impact of professional learning on student learning, Stes (2008) was unable to detect any change despite including around one thousand students in the study. Trigwell, Caballero Rodriguez, and Han (2012) used analysis of variance methods to reveal evidence of a small but significant difference between students’ overall satisfaction with the quality of their course before and after the same teacher completed a professional learning program (averaged over 116 courses). No significant change was found in the averaged results over the same period for teachers who had not completed the professional learning program. Neither of these recent, substantial studies, nor any other known studies, report findings of any direct relationship between professional learning and student learning outcomes (McAlpine, Oviedo, & Emrick, 2008; Prebble et al., 2004) and while Chalmers and Gardiner’s (2015) comprehensive framework includes a cell on the student learning outcome, their framework does not go as far as advocating the measurement of student learning change or the value added to their learning. As they and others (Stes & van Petegem, 2013) note, this may be a change that can only be observed over time, by employing measures related to the learning outcomes and encompassing more rigorous methods than selfreporting and student satisfaction, such as the use of assessment outcomes and student interviews. Such studies need to be designed and implemented. Some of the multiple variable student learning models that incorporate student learning outcomes may be useful in designing and implementing this research. Even though most of this research has been conducted on secondary and tertiary student learning, it may have relevance for professional learning as shown for the case of the adaptation of the 3P model by Prosser and Trigwell (1999). The student learning variables in a student model, such as approach to learning, are replaced with teaching variables (approach to teaching) in a teaching model and then links between learning and teaching are explored using a combination of both models. As Hoekstra and Crocker (2015) declare in their ePortfolio study: Further studies are needed to determine the organizational conditions facilitating the implementation of a portfolio approach, including issues of power and human resources

processes. In addition, studies are needed to determine the design requirements for a portfolio approach that optimally facilitates self-reflection and self-directed learning, requires minimal learning on the part of the faculty, and facilitates performance assessment. Carefully designed studies investigating the impact of a well-designed portfolio tool accompanied by portfolio assessment processes should be able to shed light on the effects of such an approach on faculty development and ultimately on students’ learning experience. (p. x) The effects on student learning may only be detected in studies in which one element (such as ePortfolios) is monitored in relation to the many other elements that constitute the professional learning package. In this sense the study has parallels with the multi-variable student learning experience studies, such as those conducted by Vermunt (2005) and Trigwell, Ashwin, and Millan (2103), using structural equation modeling analyses. Change mechanisms The second major area of research that is needed is in ways of finding out what components of professional learning packages lead to what changes for what sort of people and why. Why do some teachers change and develop, and others do not? This is a form of formative evaluation research. If the professional learning programs are to be improved, knowledge of why something works or why it doesn’t is needed. Some reports exist of research of this form. Stes and van Petegem (2013) include a section on causes of impact, and van Waes et al. (2015) write that: Our results uncovered mechanisms of network change that may offer helpful leads to enhance the quality of university teaching. By focusing on the social networks that may support individual university teachers in improving their teaching, we take the first steps to simultaneously study the individual and social side of instructional development. These first steps may help unearth a new path to developing faculty members in their role as a teacher, and as such, contribute to university teaching excellence around the globe. (p. x) Bickerstaff and Cormier (2015) also report that in instances of community building across the institution the process used also reinforced the importance of the formative element of evaluation, in noting what it is that is enabling and facilitating the building of the communities. As with the search for change in student learning outcomes, a more systematic approach to research of this type is needed. One approach is through realistic evaluation (Pawson & Tilley, 1997). This is an approach that seeks to understand the relations between context, mechanism and outcome. Rigourously applied qualitative methods are needed in concert with the quantitative methods currently being widely employed as a way to ascertain the ways in which teachers’ intentions and actions are influenced by the program. As Trigwell (2013) has elaborated elsewhere, for such an approach the research question becomes: What is it about an intervention that might work for certain people in certain circumstances? rather than (say) whether the overall score for teaching practice is improved by this intervention. To answer this question what is needed is (i) a hypothesis of the relations between context, mechanism of change, and outcome; (ii) an investigation of these relations; and (iii) a revised hypothesis based on the data obtained. Such an approach is rarely adopted in teaching development program evaluation, or indeed in most teaching/learning contexts. The difficulty is hypothesizing the reasons why the program aims are expected to be achieved (i.e., what is the

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

causal ‘‘theory’’ underlying the approach taken to achieve those aims?). (p. 263) Dealing with this difficulty requires that more thought be given to why the outcomes being sought might result from the methods being used. So, for example, if the aim of the program is to change teachers’ conceptions of teaching, what is the conceptual change theory used in the program design to achieve that aim? If the aim is to develop a professional network to support self-development, what sort of network theory underpins the approach? Once a ‘‘causal’’ theory is articulated, its effectiveness can be tested and the evidence used in evaluation and to make modifications. The meso and macro level research gap Finally, we note that in a comprehensive analysis of research into the impact of the outcomes of instructional development published by Simon and Pleschova (2013), they identify a significant gap. Their analysis made use of a micro–meso–macro framework (Hannah & Lester, 2009; Roxa˚ & Ma˚rtensson, 2013) to illustrate where research on change had been conducted at three levels (the individual—student and teacher, the institution, and in areas beyond the institution). They note that there is reasonable coverage of research at the teacher-student level, but a gap in the research in the meso area mainly, and to a lesser extent at the macro level. Research with a focus at the meso level would include topics such as the relations between professional learning and institutional culture, institutional and department practices, network groups, teaching teams, and teaching-student groups (see for example, Hum et al., 2015). At the macro level, the focus would be on institutional responses to professional learning, the effect of government initiatives, and national and international comparisons. Conclusions In summary, the collection of papers in this volume exemplifies the diversity of terminology and disparate conceptual, theoretical, and methodological approaches to both scholarship and practice in the broader field of academic and educational development. It affirms that our field is still nascent and a long way from becoming a fully-fledged discipline. It points to the challenges in the journey ahead as we collectively move the field forward. The papers also address some common themes, albeit with different intensity and emphasis. These include an array of mechanisms to foster professional learning and conditions that foster or hinder uptake and transfer to authentic contexts. They also convey the important message that institutional support, both in terms of resources and underpinning values, are important in and conducive to professional learning about teaching. The small scale and the voluntary nature of professional learning projects bring to light potential shortcomings in rendering universities as true learning organizations. They force us to think of how effective professional learning opportunities can be scaled up and how their impact can be measured, not just in terms of gains for participants, but in terms of student learning. In terms of future research into the evaluation of professional learning, four directions and a call for more robust and analytical methodologies are proposed. First, inspiration for research approaches involving teaching might be drawn from existing research on student learning, such as the ways of thinking and practicing (WTP) research, and on research on workplace learning, such as on factors known to impact workplace learning. Second, more research is needed on ways to measure the impact of professional learning on student learning, and indeed on the impact itself. As the effects are likely to be small, longitudinal

99

studies may be necessary. Third, that there is a degree of impact from the professional learning programs that are being provided is no longer in doubt. What is needed if that degree of impact is to be enhanced is research that tells us how the impact came about and, for example, why some teachers gained a lot and others gained less. Without that knowledge, changes to enhance the programs become processes of trial and error, and many of the reported studies on professional learning fail to explain why such an intervention trial was even initiated, let alone why it might have succeeded or failed. Finally, as Simon and Pleschova (2013) reveal, research aimed at capturing the impact of professional learning at the meso and macro levels is in its infancy, and would benefit greatly from further studies. In all four areas, if the research is to be confidently acted upon, the challenge of utilizing complex and robust methodologies and analytic approaches must be met. References Amundsen, C., & Wilson, M. (2012). Are we asking the right questions?. A conceptual review of the educational development literature in higher education. Review of Educational Research, 82(1), 90–126. http://dx.doi.org/10.3102/ 0034654312438409 APA Work Group of the Board of Educational Affairs (1997 (November)). Learnercentered psychological principles: Guidelines for school reform and redesign. Washington, DC: American Psychological Association. Atkinson, J. (1964). An introduction to motivation. Princeton, NJ: Van Nostrand. Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44, 1175–1184. Be´dard, D., Clement, M., & Taylor, K. L. (2010). Validation of a conceptual framework: The meaning and scope of educational development. In A. Saroyan & M. Frenay (Eds.), Building teaching capacities in higher education. A comprehensive international model (pp. 168–187). Sterling, VA: Stylus. Bickerstaff, S., & Cormier, M. S. (2015). Examining faculty questions to facilitate instructional improvement in Higher Education. Journal of Studies in Educational Evaluation, 46, 74–80. Biggs, J. B. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 1–18. Boerboom, D. V. M., Stalmeijer, R. E., Dolmans, D., & Jaarsma, D. (2015). How feedback can foster professional growth of teachers in workplace settings: The current state of the literature within medical education. Journal of Studies in Educational Evaluation XX, xx. Brew, A. (2011). Higher education research and the scholarship of teaching and learning: The pursuit of excellence. International Journal for the Scholarship of Teaching and Learning, 5(2).. Available at: hhttp://digitalcommons.georgiasouthern.edu/ij-sotl/vol5/iss2/3i. Brown, J., Collins, A., & Duguid, S. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. Centra, J. A. (1993). Reflective faculty evaluation. San Francisco, CA: Jossey-Bass. Carnegie Foundation for the Advancement of Teaching. (n.d.). Six core principles of improvement. Retrieved December 12, 2014 from hhttp://www.carnegiefoundation.org/our-ideas/six-core-principles-improvement/i. Chalmers, D., & O’Brien, M. (2004). Education development units and the enhancement of university teaching. In K. Fraser (Ed.), Education development and leadership in higher education (pp. 50–71). London, UK: Routledge/Falmer. Chalmers, D., & Gardiner, D. (2015). An evaluation framework for identifying the effectiveness and impact of academic teacher development programs. Journal of Studies in Educational Evaluation, 46, 81–91. Chism, N., & Szabo, B. (1998). How faculty development programs evaluate their services. Journal of Staff, Program, and Organization Development, 15(2), 55–62. Cohen, M., & Sproull, L. (1996). Organizational learning. Thousand Oaks, CA: Sage. Dill, D. (1999). Academic accountability and university adaptation: The architecture of an academic learning organization. Higher Education, 38, 127–154. Eccles (Parsons), J., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., et al. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motivation (pp. 75–146). San Francisco, CA: Freeman. Fullan, M. (1993). Change forces: Probing the depths of educational reform. London, UK: Falmer. Garvin, D. (1993). Building a learning organization. Harvard Business Review, 71(4), 78–84. Gibbs, G., & Coffey, M. (2004). The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students. Active Learning in Higher Education, 5, 87–100. http://dx.doi.org/10.1177/ 1469787404040463 Gosling, D. (2009). Educational development in the UK: A complex and contradictory reality. International Journal of Academic Development, 14, 5–18. http://dx.doi.org/ 10.1080/13601440802659122 Grabove, V., Kustra, E., Lopes, V., Potter, M., Wiggers, R., & Woodhouse, R. (2012). Teaching and learning centres: Their evolving within Ontario colleges and universities. Toronto, ON: Higher Education Quality Council of Ontario. Guskey, T. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin/Sage.

100

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101

Hannah, S. T., & Lester, P. B. (2009). A multilevel approach to building and leading learning organizations. The Leadership Quarterly, 20, 34–48. Hoekstra, A., & Crocker, J. (2015). Design, implementation and evaluation of an ePortfolio approach to support faculty development in vocational education. Journal of Studies in Educational Evaluation, 46, 61–73. Hounsell, D., & Anderson, C. (2005). Ways of thinking and practising in biology and history: Disciplinary aspects of teaching and learning environments. Paper presented at the Higher Education Colloquium, Centre for Teaching, Learning and Assessment, Teaching and Learning within the Disciplines Retrieved December 2014 from hhttp://www.etl.tla.ed.ac.uk/publications.htmli. Hum, G., Amundsen, C., & Emmioglu, E. (2015). Evaluating a scholarship of teaching and learning grants program: Our framework, process, initial findings, and reflections. Journal of Studies in Educational Evaluation XX, xx. Hutchings, P., & Shulman, L. (1999). The scholarship of teaching: New elaborations, new developments. Change, 31, 10–15. Kezar, A., & Eckel, P. (2004). Meeting today’s governance challenges: A synthesis of the literature and examination of a future agenda of scholarship. The Journal of Higher Education, 75(4), 371–399. Kirkpatrick, D. (1994). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler. Kreber, C. (1999). A course-based approach to the development of teaching-scholarship: A case study. Teaching in Higher Education, 4, 309–325. Kreber, C. (2002). Controversy and consensus on the scholarship of teaching. Studies in Higher Education, 27, 151–167. Krishnan, A. (2009). What are academic disciplines? ESRC National Centre for Research Methods In Working paper series 03/09UK: University of Southampton, National Centre for Research Methods. Kuhn, T. (1962). The structure of scientific revolutions. Chicago, IL: University of Chicago. Kyndt, E., & Baert, H. (2013). Antecedents of employees’ involvement in workrelated learning: A systematic review. Review of Educational Research, 83(2), 273–313. Land, R. (2004). Educational development. Buckingham, UK: The Society for Research Into Higher Education and Open University Press. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Levinson-Rose, J., & Menges, R. (1981). Improving college teaching: A critical review of research. Review of Educational Research, 51, 403–434. Laurillard, D. (1993). Rethinking university teaching. London, UK: Routledge. March, J. (1991). Exploration and exploitation in organizational learning. Organization Science, 2(1), 71–87. McAlpine, L., & Weston, C. (2000). Reflection: Issues related to improving professors’ teaching and student learning. Instructional Science, 28, 363–385. http://dx.doi.org/ 10.1023/A:1026583208230 McAlpine, L., Amundsen, C., & Turner, G. (2013). Identity-trajectory: Reframing early career academic experience. British Educational Research Journal, 38(7) http:// dx.doi.org/10.1002/berj.3123. (OnLine first). McAlpine, L., Oviedo, G., & Emrick, A. (2008). Telling the second half of the story: Linking academic development to student experience of learning. Assessment & Evaluation in Higher Education, 33(6), 661–673. Mezirow, J. (1991). Transformative dimensions of adult learning. San Francisco, CA: Jossey-Bass. Nevgi, A., & Lo¨fstro¨m, E. (2015). Development of academics’ teacher identity: Enhancing reflection and task perception through university teacher development programme. Journal of Studies in Educational Evaluation XX, xx. OECD (2013). Trends shaping education 2013. Paris: OECD Publishing http://dx.doi.org/ 10.1787/trends_edu-2013-en. Palmer, P. (1998). The courage to teach. San Francisco, CA: Jossey-Bass. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London, UK: Sage. Prebble, T., Margraves, H., Leach, L., Naidoo, K., Suddaby, G., & Zepke, N. (2004). Impact of student support services and academic development programmes on student outcomes in undergraduate tertiary study: A best evidence synthesis. Wellington, NZ: New Zealand Ministry of Education. Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching. The experience in higher education. Buckingham, UK: Open University. Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). London, UK: RoutledgeFalmer. Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 353–387. Ryan, R., & Deci, E. (2000). Intrinsic and extrinsic motivation: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67. http://dx.doi.org/ 10.1006/ceps.1999.1020 Available online at hhttp://www.idealibrary.comi. Roxa˚, T., & Ma˚rtensson, K. (2013). How effects from teacher-training of academic teachers propagate into the meso level and beyond. In E. Simon & G. Pleschova (Eds.), Teacher development in higher education: Existing programs, program impact and future trends (pp. 213–233). London, UK: Routledge, Taylor & Francis Group. Saroyan, A. (2010). Applications of research on student learning to university teaching. In J. Mighty & J. Christensen (Eds.), Taking stock: Research on reaching and learning in higher education (pp. 95–110). Montreal, CA: McGill-Queen’s Press. Saroyan, A., & Amundsen, C. (2001). Evaluating university teaching: Time to take stock. Assessment and Evaluation in Higher Education, 26(4), 337–349. Saroyan, A., & Amundsen, C. (Eds.). (2004). Rethinking teaching in higher education. Sterling, VA: Stylus. Saroyan, A., & Frenay, M. (Eds.). (2010). Building teaching capacity in universities: From faculty development to educational development. Sterling, VA: Stylus.

Scho¨n, D. (1983). The reflective practitioner. New York, NY: Basic Books. Senge, P. (1990). The fifth dimension: The art and practice of the learning organization. New York, NY: Doubleday. Shulman, L. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–22. Shulman, L. (2005 (February)). The signature pedagogies of the professions of law, medicine, engineering and the clergy: Potential lessons for the education of teachers. In Paper presented to the Math Science Partnership (MSP) workshop, teacher education for effective teaching and learning. Irvine, CA: National Research Council’s Centre for Education. Simon, E., & Pleschova, G. (2013). Creating successful teacher development programmes. In E. Simon & G. Pleschova (Eds.), Teacher development in higher education: Existing programs, program impact and future trends (pp. 274–297). London, UK: Routledge, Taylor & Francis Group. Simons, P., & Ruijters, M. (2004). Learning professionals: Towards and integrated model. In H. P. Boshuizen, R. Bromme, & H. Gruber (Eds.), Professional learning: Gaps and transitions on the way from novice to expert (pp. 207–229). Dordrecht, The Netherlands: Kluwer. Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., & Prideaux, D. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education. Medical Teacher, 28, 497– 526. Stes, A. (2008). The impact of instructional development in higher education: Effects on teachers and students. Gent, Belgium: Academic Press. Stes, A., & van Petegem, P. (2013). Instructional development for university teachers: Causes of impact and practical implications. In E. Simon & G. Pleschova (Eds.), Teacher development in higher education: Existing programs, program impact and future trends (pp. 234–256). London, UK: Routledge, Taylor & Francis Group. Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of instructional development in higher education: The state-of-the-art of the research. Educational Research Review, 5, 25–49. http://dx.doi.org/10.1080/ 01421590600902976 Taylor, K. L., & Be´dard, D. (2010). Faculty development in Canadian universities. In A. Saroyan & M. Frenay (Eds.), Building teaching capacities in higher education: A comprehensive international model (pp. 23–42). Sterling, VA: Stylus. Taylor, K. L., & Znajda, S. (2015). Demonstrating the impact of educational development: The case of a course design collaborative. Journal of Studies in Educational Evaluation XX, xx. Tremblay, K., Lalancette, D., & Rosevere, D. (2012). Assessment of Higher Education Learning Outcomes (AHELO): Feasibility study report: Design and implementation (Vol. 1). Paris: FR. Trigwell, K., & Shale, S. (2004). Student learning and the scholarship of university teaching. Studies in Higher Education, 29(4), 523–536. http://dx.doi.org/10.1080/ 0307507042000236407 Trigwell, K., Caballero Rodriguez, K., & Han, F. (2012). Assessing the impact of a university teaching development programme. Assessment & Evaluation in Higher Education, 37(4), 499–511. Trigwell, K. (2013). Evaluating the impact of teaching development programmes: Methodologies that ask why there is an impact. In E. Simon & G. Pleschova (Eds.), Teacher development in higher education: Existing programs, program impact and future trends (pp. 257–273). London, UK: Routledge, Taylor & Francis Group. Trigwell, K., Ashwin, P., & Millan, E. S. (2013). Evoked prior learning experience and approach to learning as predictors of academic achievement. British Journal of Educational Psychology, 83, 363–378. Trowler, P., & Knight, P. (1999). Organizational socialization and induction in universities: Reconceptualizing theory and practice. Higher Education, 37, 177–195. van Schalkwyk, S., Herman, N., Leibowitz, B., & Farmer, J. (2015). Reflections on professional learning: Choices, context and culture. Journal of Studies in Educational Evaluation XX, xx. van Waes, S., van den Bossche, P., Moolenaar, N. M., Stes, A., & van Petergem, P. (2015). Uncovering changes in professional networks during instructional development in higher education: A longitudinal social network study. Journal of Studies in Educational Evaluation XX, xx. Weiner, B. (1979). A theory of motivation for some classroom experiences. Journal of Educational Psychology, 71, 3–25. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. New York, NY: Cambridge University. Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131–175. van Schalkwyk, S., Cilliers, F., Adendorff, H., Cattell, K., & Herman, N. (2012). Journeys of growth towards the professional learning of academics: Understanding the role of educational development. International Journal for Academic Development, 18(2), 139–151. http://dx.doi.org/10.1080/1360144X.2012.673490 Vanthournout, G., Noyens, D., Gijbels, D., & Van den Bossche, P. (2014). The relationship between workplace climate, motivation and learning approaches for knowledge workers. Vocations and Learning, 7, 191–214. Vermunt, J. D. (2005). Relations between student learning patterns and personal and contextual factors and academic performance. Higher Education, 49(3), 205–234. Alenoush Saroyan is Professor of Educational Psychology and the previous Chair of the Department of Educational and Counselling Psychology at McGill University in Montre´al, Canada. Her areas of specialization and research include the pedagogical development of academics, academic leadership, quality assurance, and the reform of postsecondary systems. Her publications include articles on university professors’ conceptions of teaching and edited books and book chapters on academic

A. Saroyan, K. Trigwell / Studies in Educational Evaluation 46 (2015) 92–101 development. She serves as a consultant to various international organizations including The World Bank and The OECD. She was appointed as the first honorary Chair in University Pedagogy at the Universite´ catholique de Louvain in Belgium, has been an officer of the Canadian Society for the Study of Higher Education, Program Chair for the American Educational Research Association Special Interest Group, Faculty Teaching, Development and Evaluation. She serves as a reviewer for several professional journals and as an adjudicator for the Canadian Social Sciences and Humanities Research Council. Keith Trigwell is Professor of Higher Education in the Institute for Teaching and Learning (ITL) at the University of Sydney. He was previously Director of The ITL

101

and of the Oxford Centre for Excellence in Preparing for Academic Practice, a Fellow of Kellogg College and Reader in Higher Education at the University of Oxford. His research interests include investigating qualitative differences in university teaching and students’ learning experiences, teaching-research relations and the scholarship of teaching, including development of the Approaches to Teaching Inventory. This work has yielded more than 100 journal articles, conference papers and books, including Understanding Learning and Teaching: The Experience in Higher Education, which summarises10 years of research with Professor Michael Prosser. He is a former co-president of the International Society for the Scholarship of Teaching and Learning, and in 2010 received a Lifetime Achievement Award (Leadership) from the Society.