Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education

G Model

JSEE-562; No. of Pages 13 Studies in Educational Evaluation xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/stueduc

Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education Annemarieke Hoekstra *, Jocelyn R. Crocker * Northern Alberta Institute of Technology, 11762-106 Street NW, Edmonton, AB, Canada T5G 2R1

A R T I C L E I N F O

A B S T R A C T

Article history: Received 15 May 2014 Received in revised form 18 February 2015 Accepted 26 March 2015 Available online xxx

This article provides an account of the design, implementation, and evaluation of an ePortfolio approach to faculty development and performance evaluation at a Canadian post-secondary vocational education institute. The approach was piloted in two phases in 13 departments. Survey and interview data were collected and analyzed to determine adoption, reception by faculty, and impact of the approach on faculty development. While adoption of the approach in the pilots was limited, participants who adopted the approach reported collecting more and different feedback, developing increased awareness of areas for improvement, and planning their professional learning more explicitly. Further studies are needed to determine what design elements of the portfolio optimally support professional development and performance evaluation. ß 2015 Elsevier Ltd. All rights reserved.

Keywords: Faculty development Professional development Professional learning Electronic portfolio Faculty evaluation Vocational education

Introduction As in many professions, the need for ongoing professional development exists for teaching professionals in post-secondary education. In Canadian post-secondary vocational education, this necessity stems from ongoing changes in the vocations being taught and an increasingly diverse student population (Bye, Pushkar, and Conway, 2007; Darwin, 2007; Harris et al., 2001; Statistics Canada, 2010). For instance, in an effort to further contribute to economic development, Canadian provincial governments now encourage post-secondary vocational education institutes to engage in applied research in support of business development (e.g., Polytechnics Canada, 2014). In addition, changes in educational approaches, such as a move to more authentic forms of student assessment, may require faculty to think differently about supporting student learning (e.g., Stiehl & Lewchuk, 2008). Finally, vocational educators may also be required to serve on committees and in leadership roles. These changes and additional duties require vocational educators to keep learning during their careers. To address this learning need, post-secondary institutes have traditionally invested in course or

* Corresponding authors. Tel.: +1 780 471 7862/+1 780 378 5197. E-mail addresses: [email protected], [email protected] (A. Hoekstra), [email protected] (J.R. Crocker).

conference attendance and the creation and delivery of in-house workshops. Professional development literature argues that behaviours learned during workshops and courses should be reinforced in the workplace (Blume, Ford, Baldwin, & Huang, 2010; Kirkpatrick, 2007). Post-secondary faculty development initiatives have aimed to incorporate workplace practices as a source of learning that uses, for instance, peer review or action research (Amundsen & Wilson, 2012; Stes, Min-Leliveld, Gijbels, & Van Petegem, 2010). However, in her review of professional development literature, Webster-Wright (2009) maintains that professional developers and educational leaders should move beyond an interventiontransfer-impact paradigm and focus on how professionals naturally learn at work. This paper describes the design, implementation, and evaluation of a faculty development initiative called the ePortfolio approach, which was designed to foster learning that occurs naturally in the workplace. In 2009, senior leaders at a Canadian post-secondary institute for vocational education mandated a taskforce to design a comprehensive approach to faculty development and performance evaluation. The paper describes how, based on a literature review and several rounds of faculty engagement, the taskforce designed the ePortfolio approach. This approach consisted of a comprehensive process for faculty to (1) collect feedback from multiple sources on multiple aspects of their work; (2) compile the feedback in their ePortfolio, reflect on it, and make changes to their

http://dx.doi.org/10.1016/j.stueduc.2015.03.007 0191-491X/ß 2015 Elsevier Ltd. All rights reserved.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 2

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

teaching practice if necessary; (3) formulate goals for future growth based on their feedback and reflection; and (4) share their ePortfolio with their supervisor to collaboratively assess their performance and determine goals for the next year. The paper also describes how the ePortfolio approach was implemented in two consecutive pilots and the survey and interview data collected in order to evaluate the pilots. The research questions addressed in this paper are as follows: (1) To what extent was the ePortfolio adopted in the participating departments in the first and second pilot? (2) To what extent did faculty in the first and second pilot value the ePortfolio approach as a whole? (3) What was the impact of the ePortfolio approach on the professional development of participating faculty members? Literature review We define faculty development in accordance with Taylor and Rege-Colet (2009) who describe it as development across ‘‘the full spectrum of academic work’’ (p. 143). Our view of faculty development also aligns with Steinert (2000), who broadly defines it as any type of activity that is aimed at renewing or assisting faculty in their roles. Our definition is wider in scope than simply instructional development (Stes et al., 2010) and educational development (Taylor & Rege-Colet, 2009), which both refer to the development of the faculty member in the areas of teaching and student learning. Our definition of faculty development includes professional learning in all aspects of the role of the faculty member including, for example, teaching, research, and administrative tasks. Research on professional learning generally suggests that such learning is deeply embedded in practice (Opfer & Pedder, 2011; Scho¨n, 1983) and informed by the way people within and outside of the organization conduct and understand their work (Bound, 2011; Engestro¨m, 2011; Lave & Wenger, 1991). Billett (2002) proposes that learning at work occurs through participating in work-related activities. These activities play a central role in our definition of professional learning. Most conceptualizations of learning imply a relatively lasting change in behaviour or capacity for behaviour (Shuell, 1986). In the present study, professional learning is defined as engaging in activities that lead to improved professional practice or the capacity to behave in improved ways. Such learning may be intentional or unintentional and conscious or beyond the learner’s awareness (Hoekstra, Beijaard, Brekelmans, & Korthagen, 2007; Eraut, 2004; Marsick & Watkins, 1990) and involve formal and/or informal learning activities (e.g., Hoekstra, Brekelmans, Beijaard, & Korthagen, 2009; Lohman & Woolf, 2001). Reflective practice and feedback Teacher educators, educational developers, and scholars who study workplace learning all stress the importance of reflective practice as part of ongoing professional development (Hoekstra & Korthagen, 2011; Amundsen & Wilson, 2012; Brookfield, 1995; Cross & Steadman, 1996; Van Woerkom & Croon, 2008; Zeichner & Liu, 2010). While reflection can occur individually based on input from individual experience, reflection can be greatly enhanced through incorporating feedback from others into the individual’s own reflective process (Day, 1999; McGovern, 2006). The taskforce thus determined that the ePortfolio approach should include student feedback as well as self-reflection on feedback (Hubball, Pratt, & Collins, 2006; McGovern, 2006; Seijts, Taylor, & Latham, 1998). Student feedback is a common component of faculty assessment in universities (e.g., Pallett, 2006) and two-year colleges (e.g., Cross & Steadman, 1996). In many post-secondary institutions,

formal student feedback is a mandatory part of college or university workplaces, and many post-secondary teachers also seek informal feedback from their students. However, as Halonen and Ellenberg (2006) highlight, students might provide a one-sided view of faculty performance or allow their feedback to be influenced by bias. Feedback from additional sources such as colleagues or faculty support staff might provide a more comprehensive picture of professional performance (Millis, 2006; Jimerson, 2014; Seijts et al., 1998; Wayman & Jimerson, 2014). Therefore, the taskforce established that feedback collection from multiple sources should be encouraged. Wayman and Jimerson (2014) also found that for feedback to be relevant to teachers and students, it needs to be timely (see also Halonen & Ellenberg, 2006) so faculty members can discuss the feedback with their students and adjust their teaching to accommodate students’ needs. Therefore, the third parameter for the ePortfolio approach was that student feedback should be sought 30–50% of the way through the semester with a quick turnaround of the feedback. Educational development literature describes learning with and from colleagues as a workplace learning strategy, using, for instance, faculty learning communities (Cox & Richlin, 2004) or mentoring (Johnson, 2006). Faculty might also initiate informal learning activities such as seeking feedback on course materials or discussing experiences during their break (Hoekstra et al., 2009; Eraut, 2004). Based on these findings, the taskforce determined that faculty members would be encouraged to share their portfolios with colleagues. Professional development plans and self-directed learning Literature on workplace learning, teacher learning, and faculty development all stress that conditions in the workplace impact on the quality of professional learning at work (Amundsen & Wilson, 2012; Clarke & Hollingsworth, 2002; Fuller & Unwin, 2011; Harris et al., 2001; Opfer & Pedder, 2011; Sambrook, 2005). It can thus be expected that enhancing faculty development requires faculty motivation to engage in learning activities; it also needs the workplace culture, structure, and systems to encourage and nurture engagement in these activities (Fuller & Unwin, 2011). One strategy to encourage more planning and regulation of professional development adopted in several professions, including K-12 teaching, is the use of a personal development plan (PDP), which aims to foster self-directed professional learning (Beausaert, Segers, Gouarge, & Gijselaers, 2013). According to Roberson and Merriam (2005), self-directed learning refers to ‘‘intentional and self-planned learning, where the individual is responsible for and in control of the learning’’ (p. 270). Beausaert et al. (2013) reviewed 54 studies and reported that attempts to implement PDPs have varying degrees of success. In their own study, Beausaert et al. compared the number of learning activities reported by PDP users with the number of activities of non-PDP users, concluding that, retrospectively, PDPs encourage participation in learning activities. Sharing their learning plans and assessment of their performance with the supervisor allows him or her to collaborate with the faculty member and become an accountability partner for the faculty member’s own development (Higgerson, 2006; Murray, 1997). Based on this literature, the taskforce determined that the ePortfolio approach should encourage faculty members to set their own learning goals and encourage department chairs to provide sufficient modes of support and guidance for achieving these goals. Performance evaluation and ePortfolios The taskforce intended to design a process that would not only foster faculty development but also support the assessment of

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

faculty performance in collaboration with the department chair. Peter Seldin’s book ‘‘Evaluating Faculty Performance’’ (Seldin, 2006) provides a strong case for the use of portfolios to assist in this process (see also Freeman, Harkness, & Urbackzewski, 2009). Portfolios can include a collection of an individual’s accomplishments, plans, reflections, observations, and work samples (Granberg, 2010). One advantage of ePortfolios as opposed to the traditional format is they can also incorporate digital files such as simulations and videos (Butler, 2006). Smith and Tillema (2003) identify four types of portfolios including the dossier portfolio (a record of achievements and work samples for promotional purposes), the training portfolio (a required collection of efforts accumulated during a program), the reflective portfolio (a personal array of work providing evidence of growth and accomplishments for promotion or admission), and the personal development portfolio (a reflective account of professional growth over time). Portfolios can be used as both formative assessment to support professional learning and summative assessment for promotion or certification (Smith & Tillema, 2003). When the focus is on formative assessment, portfolios can be used to motivate faculty to reflect on priorities, accomplishments, successes, and challenges and plan their learning accordingly (Devanas, 2006). Portfolios can thus easily integrate aspects of the personal development plan, including activities such as goal setting and collecting feedback specific to those learning goals. The 2013 study by Beauseart et al. suggests that including such activities in a portfolio approach would likely contribute to enhanced participation in learning activities. While portfolios are widely used in teacher education to support the development of student teachers (e.g., Granberg, 2010) and used increasingly with post-secondary students of various disciplines including nursing (e.g., Jasper & Fulton, 2005), medicine (e.g., Carraccio & Englander, 2004) and business (e.g., Flanigan, 2012), the use of portfolios in support of faculty development and evaluation has received much less attention (Butler, 2006; McColgan & Blackwood, 2009). With respect to universities, ePortfolios have been found to support and enhance professional development of faculty within the university promotion and tenure process (Blair, 2001; Freeman et al., 2009). However, little is known about the effect or use of portfolios in general and ePortfolios specifically in support of faculty development. Conceptually, ePortfolios have the potential to enhance and promote self-directed learning (Goliath, 2009) and self-regulated learning (Lamont, 2007). Based on this literature, the taskforce determined that the best way to satisfy the parameters was through the creation and sharing of professional ePortfolios. While not intended for purposes of tenure and promotion, the professional ePortfolio approach designed by the taskforce would cut across multiple types of portfolios as identified by Smith and Tillema (2003). Faculty would be invited to include a list of professional development sessions attended (as in a training portfolio), evidence of feedback collected, and documented reflections on their feedback (as in a reflective portfolio). The compilation of this evidence would serve to inform reflections on personal professional growth over time (as in a personal development portfolio). Methods The ePortfolio approach was implemented in two pilots. Following each pilot, both survey and interview data were collected to determine adoption, faculty perception, and faculty perception of the impact of the ePortfolio approach. Study context The site of study is a Western Canadian post-secondary vocational institute that offers apprenticeship training, two-year

3

diploma programs, degree programs, and professional certificates in fields including trades, building construction and design, engineering and applied sciences, health and safety, IT and electronics, business and administration, and hospitality and culinary arts. The institute has approximately 1100 faculty and 80,000 full-time and part-time students. Historically, the institute’s faculty members were required to create a performance management plan in which they outlined professional goals for the next year and annually discussed with their supervisors the extent to which they achieved their goals. There is no promotion and tenure process at the institute; in other words, faculty members are directly hired into salaried positions and remain faculty until retirement or job change. Professional development at the institute was most often conceptualized as attending conferences, courses, and workshops either on teaching or one’s trade/discipline. The institute had a process for collecting student feedback once per semester, but this feedback was not necessarily part of the performance management review cycle. ePortfolio design In 2009, a taskforce was formed to create a comprehensive approach to faculty development and performance evaluation. Previously, instructors who scored low on student feedback surveys had been sent to the faculty support office for coaching, leading to the impression that faculty development was remedial (see also Cross & Steadman, 1996) rather than a necessity for all faculty to stay current and up to date. In addition, there was a need to review and update the existing student feedback questionnaires and processes. The approach the taskforce was asked to design was not intended to replace existing human resources processes regarding personnel decisions; rather, it was intended to encourage faculty to collect comprehensive feedback and share it with their chair so as to collaboratively and continually work toward improving their practice. Based on literature on leading by engagement (Axelrod, 2010), the taskforce held a series of town hall meetings and focus groups with faculty and other staff to engage them in the design and implementation stage. During these sessions, members of the academic staff union indicated that the role of faculty members had significantly expanded beyond teaching and that faculty development and performance evaluation should cover this expanding array of faculty responsibilities; these views aligned closely with the findings of the literature review (e.g., O’Meara & Braskamp, 2005). The taskforce then used focus groups with institutional leadership, faculty members, and executive members of the academic staff union to collaboratively identify eight aspects of the role of a faculty member: teaching, formal leadership, formal administrative duties, professionalism, formal curriculum development, applied research, scholarship and professional development, and corporate citizenship. See page 4 of Online supplement 3 for a description of each of these aspects. During these focus groups, faculty members also indicated that feedback collected only from students might not be a reliable method to assess their performance. Taskforce members agreed with this point of view, based on the literature described earlier in this paper. Stakeholders and taskforce members collaboratively decided to identify the following sources of evidence of performance: self, students, peers, and ‘‘others,’’ the latter including externals and faculty support personnel. In further faculty engagement sessions, a matrix was created with the eight aspects as rows and the four sources as columns. Participants in the sessions collaboratively determined which sources would be able to provide feedback on which aspects of the faculty role. Faculty support staff then proceeded with developing feedback tools for those cells in the matrix that were identified as good

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

4

sources for feedback on certain roles. These tools were presented together as a feedback toolkit and made available through the institute’s intranet. The toolkit included informal student feedback tools, a standardized student survey, self-reflection resources, peer-review tools including classroom observation templates, and a standardized leadership survey. See Online supplement 1 for an overview of the feedback tools in the toolkit. Thus, based on the literature and faculty engagement sessions, the taskforce designed the ePortfolio approach as a faculty development and performance evaluation initiative that encouraged faculty to (1) collect feedback from multiple sources on multiple aspects of their role; (2) compile this feedback in their ePortfolio, reflect on it, and make changes to their teaching practice, if necessary; (3) formulate goals for future growth based on their feedback and reflection; and (4) share their ePortfolio with their supervisor to collaboratively assess their performance and determine goals for the next year. The ePortfolio software Mahara (mahara.org) is an open-source software program chosen as the web-based ePortfolio software tool because it is intuitive in use and works within any browser. Mahara permits password protection and allows the faculty member to customize different versions of the portfolio for different audiences, such as peer groups or supervisors. The faculty member can have full control over who sees which page(s) and can set limits for the duration intended audiences can access those pages. Mahara version 1.4.6 was slightly customized for use within the institute by creating a login page with the institute’s colours and logo. Pilot implementation The ePortfolio approach was piloted in two phases. In the first, the deans of all five faculties in the institute were requested to seek department chairs willing to try out the new approach in (one of their1) smaller programs. In the second, the deans were requested to seek department chairs of two to three larger programs. The first pilot was carried out from September–December 2011 with 32 participants (including 5 chairs and 27 faculty members) from five programs in trades, health sciences, business, and media and design. The second pilot was carried out from September– December 2012 with 75 participants from 12 programs in 9 departments2 including programs in trades, engineering, applied sciences, and health. One program participated in both pilots. The total number of participating departments was 13: four in the first pilot, eight in the second, and one that participated in both. In the first pilot, all participants, including the department chairs, attended the same orientation to the approach during two half-days in September of interactive introductory sessions organized by the taskforce. The approach was just as new to the department chairs as to the instructors. During the orientation, an open conversation between a faculty member and a chair regarding feedback of the faculty member was modeled. In the second pilot, departments received an orientation within their own program. In the orientation of both pilots, participants were introduced to the purpose of the ePortfolio approach: to support faculty development and performance evaluation. They were presented with an overview of the philosophy behind the approach, including the expanding role of faculty, eight aspects of the faculty role, the reasons why feedback should be collected from multiple sources, the value of reflective practice, and the role the department chair could play in supporting formative perfor1

Some department chairs oversee more than one program. One participating department included instructors who teach in three separate programs. 2

mance assessment and coaching. Specific attention was paid to the idea that performance evaluation should happen on an ongoing basis with the purpose of supporting professional growth. It was also highlighted that human resource processes around disciplinary measures were outside the scope of the ePortfolio approach. A follow-up session in a computer lab introduced participants to the online ePortfolio software Mahara. A mandatory part of both pilots included the automatic collection and reporting of student feedback through a newly developed online electronic student survey. In addition, supports were offered through a website, online feedback toolkit (see Online supplement 1), reminder emails, and additional workshops in which participants could practice the use of the ePortfolio tool. These additional workshops were offered to departments who requested extra help. Based on the findings from the first pilot, a number of changes were made to the implementation of the second pilot. The student and leadership surveys were refined, more diverse feedback tools were developed, and it was decided to provide a customized orientation and workshop to each participating department separately. Between the first and second pilot, the institute initiated a drastic academic change that involved major curriculum development for all credit programs, a new credit system, the introduction of common courses, and a profound revision of the academic administration software used by the institute. This change caused a great increase in workload of faculty. Instruments The evaluation focused on the impact of the ePortfolio approach and its implementation, including details regarding the orientation sessions held, the new student evaluation survey, the new survey used by chairs to collect feedback on their performance, student response rates, and the feedback tools. This paper focuses on those questions pertaining to adoption of the approach, how faculty experienced the ePortfolio as a whole, as well as the impact of the approach on their professional learning activities. Data regarding the orientation sessions and the composition and usefulness of individual feedback tools are not included in this paper. In collaboration with the taskforce, the researchers created a survey and interview guide aimed at collecting feedback on the orientation to the ePortfolio, the feedback tools developed, and the ePortfolio approach as a whole. An interview guide was developed to collect more detailed feedback on these elements and obtain a more in-depth understanding of the impact of the ePortfolio approach on faculty learning. Based on the responses from the first pilot, we also evaluated our survey questions. The survey used for the first pilot had many open-ended questions. The answers to those questions were not always easy to interpret, and not all participants answered. For instance, one question read as follows: How do you feel about the whole experience of collecting feedback, reflecting on it, and documenting it? One participant answered with ‘‘Informative.’’ Another wrote: ‘‘We are here because we are good at our trade. This generally means we are not all great writers.’’ The first answer is so general that it’s hard to interpret what exactly this participant found informative; the second response most likely points to a reaction to the encouragement to include written reflections. This question was amended to include more closed-ended questions, such as (1) I found the ePortfolio approach helpful for my professional development. (2) The ePortfolio approach is a useful way to set learning goals. While these amended questions yielded better data for evaluating the second pilot, these changes prevented us from comparing the pilot 1 survey data with the pilot 2 survey data. The second pilot survey also included more closed-ended questions pertaining to concerns the taskforce had heard about anecdotally, such as a lack of trust in the security of the Online data. The interview questions were the

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

same as for the first pilot and are listed in Online supplement 2. The survey questions are presented in Online supplement 3. Data collection The first pilot was carried out from September–December 2011 with 32 participants from five programs, including programs in trades, health sciences, business, and media and design. Before the start of data collection, research ethics approval was obtained from the institute’s research ethics board. This board verifies that proposed recruitment and data collection procedures are in accordance with In Canadian Tri-Council Framework for Research involving Human Subjects. Data collection took place in early 2012. To invite faculty to the survey and to select interview participants, the chairs of the participating programs were approached to provide a list of their participating faculty members. From this list, 1–2 faculty members per participating program were selected to be interviewed using a random number generator. In programs with more than six faculty members, two participants were selected to be interviewed. In the first pilot, we interviewed seven randomly selected participants: one from each of the three smaller programs (<6 faculty members) and two from the two larger programs (6 members). We also interviewed each of the five department chairs. We thus had 12 interview participants in pilot 1: seven faculty members and five department chairs. Of all 32 pilot 1 participants who were invited to anonymously respond to an online survey, 26 responded. The decision to collect anonymous feedback was made in an attempt to increase the chances of receiving honest feedback. Because a number of participating programs were small (3 faculty members) the survey did not ask participants which program they were part of since that information might identify them. The second pilot was carried out from September–December 2012 with 75 participants from 12 programs in 9 departments3 including programs in trades, engineering, applied sciences, and health. One program participated in both pilots. Data collection took place in early 2013. We used the same sampling methods as in pilot 1 to select interviewees. Three of the selected faculty members declined to participate. Research ethics guidelines stipulate that those approached to participate in the study may ‘‘opt out’’ of the data collection without having to provide a reason. We were therefore unable to establish their reasons for declining. Using a random number generator, alternative participants were selected from the remaining faculty members in those departments. Unfortunately, in one small program, the freedom to opt out meant that we could not interview anyone. As a result, 14 participants from 11 programs from the second pilot were interviewed. Time constraints prevented us from interviewing the department chairs. All 75 pilot 2 participants were invited to fill out the online pilot 2 survey. A reminder was sent two weeks after the survey was distributed, resulting in a total of 42 responses. Data analysis The responses to closed survey questions were summarized into frequency overviews. The responses to open survey questions were categorized and summarized. This paper only reports the results from the survey questions relating to the adoption of the ePortfolio approach and faculty members’ perception of the value and impact of the approach. Interviews were transcribed verbatim. Interview excerpts from the first pilot were organized according to the research questions of this paper using the themes (1) adoption of ePortfolio, (2) 3

One department offered three separate programs.

5

perceived value of ePortfolio approach, (3) impact of adoption of ePortfolio approach. Interviews from the first pilot were coded by two researchers, in most cases reaching a minimum of 85% agreement, and discrepancies were discussed. Interviews from the second pilot were analyzed by one researcher, using the same themes. For the analysis of interview data from both pilots, a matrix was created with each column representing a theme and each row, a participant. Interview excerpts were copied into the cells or, in the case of very large excerpts, summarized in the cells. This tool allowed the researchers to compare interview excerpts relating to one theme across the participants. Based on these comparisons, we created categories across participants as well as a count of how many participants would fall into this category. For instance, the theme adoption of ePortfolio approach included three categories: (1) no portfolio created, (2) portfolio page with some documents uploaded, and (3) more elaborate ePortfolio. See Online supplement 4 for a list of coding categories and the number of interviewees categories apply to. Because only two interviewees in the second pilot created more than one page in their portfolio, we limited the analysis of the second pilot interviews to the first two research questions on adoption and perception of value of the ePortfolio approach. In the next section, we answer the research questions by presenting survey and interview data. We describe the categories developed in our interview analysis, how many participants this category applied to, and examples of interview excerpts. Findings This section presents the findings in the order of our research questions: (1) To what extent was the ePortfolio adopted in the programs participating in the first and second pilot? (2) To what extent did faculty in the first and second pilot value the ePortfolio approach as a whole? (3) What was the impact of the ePortfolio approach on the professional development of participating faculty members? The extent of ePortfolio adoption in pilot 1 Using survey and interview data, the extent of the adoption of the ePortfolio approach was assessed in terms of whether or not the participants reported collecting feedback and/or compiling that feedback into their ePortfolio, reflecting on it, sharing it with their supervisor, and making changes if necessary. Most of the first pilot participants reported collecting some type of feedback. Table 1 provides the frequency overviews related to two survey questions: (1) Is this aspect4 part of your job? (2) Who did you collect feedback from regarding this aspect of your job? These results show that the majority of survey respondents (22 out of 26) were teaching during the pilot; cross-tabulation confirms that 20 of these 22 respondents collected student feedback on their teaching. Examination of the raw data file indicates that one survey respondent chose not to fill out any of the questions listed in Table 1. Similar cross-tabulations show that all respondents who collected feedback on certain aspects of their job also indicated that those aspects were in fact part of their job. Table 1 demonstrates that quite a few respondents collected feedback from peers on job aspects that applied to them. For instance, 13 out of 23 respondents to whom Scholarship/Professional Development applied sought peer feedback. Table 2 shows that the majority of participants (69%) collected more and/or different types of feedback in the pilot than they did before the pilot. Of the five comments to this question, three 4 These aspects were based on the eight aspects of the role of a faculty member as identified in the stakeholder consultation, which is described in the Method section.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

6

Table 1 Aspects part of respondents’ job and who they collected feedback from regarding that aspect. Part of job? Yes Teaching Professionalism Corporate citizenship Scholarship/professional development Applied research Formal curriculum development Formal administrative duties Formal leadership a

79% 86% 68% 82% 14% 64% 36% 36%

Feedback on this aspect collected from: No

(22) (24) (19) (23) (4) (18) (10) (10)

11% 4% 18% 7% 64% 21% 46% 46%

(3) (1) (5) (2) (18) (6) (13) (13)

Peersa

Studentsa

Supervisora

Othera

Nobody

Not Applicable

21% 50% 14% 46% 4% 29% 18% 29%

71% 21% 4% 0% 0% 7% 4% 7%

14% 39% 11% 18% 0% 25% 11% 14%

4% 7% 4% 4% 0% 0% 0% 0%

0% 4% 29% 29% 7% 18% 7% 4%

7% 4% 14% 0% 46% 11% 25% 32%

(6) (14) (4) (13) (1) (8) (5) (8)

(20) (6) (1) (0) (0) (2) (1) (2)

(4) (11) (3) (5) (0) (7) (3) (4)

(1) (2) (1) (1) (0) (0) (0) (0)

(0) (1) (8) (8) (2) (5) (2) (1)

(2) (1) (4) (0) (13) (3) (7) (9)

Respondents were able to select more than one response regarding data source for each aspect of their job.

Table 2 Responses to the question: As a result of the pilot, did you collect more/different feedback than you used to? More feedback

Yes No Total

Different feedback Yes

No

Total

35% (9) 8% (2)

27% (7) 31% (8)

16 10

11

15

26

indicated the difference was the collecting of peer feedback. The other comments stated ‘‘no difference.’’ Only 31% reported that they collected neither more nor different feedback than before. As for the documentation of feedback, growth, and development, most pilot participants created an ePortfolio page using Mahara during a 1-h workshop in the computer lab. Table 3 shows that 77% (35% + 42%) of respondents spent 5 h or less on documenting growth and development, with nine of them spending less than 1 h. Participants were also asked whether they included their feedback, reflections, and evidence of accomplishments in Mahara. Table 4 shows the majority included only some or none in their ePortfolios. Two respondents chose to include comments to this question; one indicated insufficient time, and the other asserted that building the ePortfolio seemed to be too much work. The interviews confirm the survey findings, in that the majority (9 out of 12) of interviewees collected feedback. Seven out of 12 interviewees created a portfolio, although only two created a more elaborate portfolio. From the interviews with faculty and chairs, we learned that each department adopted the ePortfolio approach to a different degree during the first pilot. Two chairs did not discuss and were not planning to discuss the portfolio with the faculty members who report to them. Three chairs had these discussions with some faculty members and were planning to sit down with the remaining faculty members. The five programs differed in the extent to which the ePortfolio approach was adopted. For example, one program had all its faculty members create and share their ePortfolios with one another. These ePortfolios included student and leadership feedback. Chair Kim5, for instance, reported that, regarding the ePortfolios, ‘‘We’ve had discussions as a group; we’ve had discussions as individuals’’ as s/he met with her/his staff to view the contents of their ePortfolios. On the other hand, in one program, all faculty had tried out the student survey and the leadership survey, no one in the program had been able to create an ePortfolio. They also did not have supervisor-staff conversations or a staff meeting about the ePortfolio. Addy, the chair of this program 5 To protect the identity of participants, all names used in this paper are genderneutral pseudonyms.

commented: ‘‘I wasn’t comfortable enough with the software, right. [. . .] I guess it just seemed like it was a lot of effort that I wasn’t prepared to put in, to be honest with you.’’ When asked whether s/he discussed the ePortfolios in a supervisory role with her/his staff, Addy replied: ‘‘No, because I don’t think their response to it was any better than mine. [. . .] I spoke to them. I did speak to them about it and ask them, and nobody had [made an ePortfolio].’’ In some programs, the interviews with multiple faculty from the same programs revealed that the extent to which the approach was adopted differed from one faculty member to the next. ePortfolio adoption in pilot 2 The second pilot included 75 participants, 42 of which responded to the survey. Table 5 shows that of the 37 participants who chose to respond to this question, half (49%) spent 2 h or less on the ePortfolio approach. This likely represents the time spent in the orientation and Mahara workshop. The other respondents spent more time, but only 16% spent more than 5 h on the ePortfolio approach. Table 6 shows participants’ answers to the question, Do you believe you sufficiently understand the purpose and principles underlying the ePortfolio approach? Only half of the participants (48%) claimed to understand the purpose of the ePortfolio approach. When asked whether respondents used Mahara as an ePortfolio tool, 22 (52%) responded ‘‘yes,’’ 20 (48%) responded ‘‘no.’’ Those who responded ‘‘yes’’ were directed to the question of whether they found Mahara easy to use. Of those who used Mahara, eight (36%) found it easy to use whereas the remaining 14 (64%) did not find it easy to use. Ten out of the 22 respondents (46%) who used Mahara said that they did not feel confident that their pages were private. None of the pilot 1 participants had voiced these concerns. In the five written comments to this question, one respondent commented that every server can be hacked. Two comments also indicated concerns about uploading personal information online. One respondent did not see the point of broadcasting professional achievements online; a fifth did not see a need for another online social media tool. Cross-tabulation of the question regarding understanding of the purpose and principles of the ePortfolio approach with the question of whether respondents used Mahara (see Table 6) indicates no apparent relation between perceived understanding of the approach and the use of Mahara. A cross-tabulation of the same question with the question of whether participants found the approach helpful for professional development (PD)(see Table 6) indicated that those who expressed uncertainty in understanding showed a higher percentage of respondents who were undecided regarding its helpfulness. Other than that, understanding of the purpose and principles does not seem related to whether or not

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

7

Table 3 Responses to the question: How much time did you spend on the following aspects of the ePortfolio approach?

Learning to use Mahara Collecting and reflecting on feedback Documenting your growth and development (including writing reflections, uploading in Mahara, etc.)

<1 h

1–5 h

5–10 h

10–20 h

>20 h

20% (5) 16% (4) 35% (9)

72% (18) 48% (12) 42% (11)

4% (1) 28% (7) 8% (2)

4% (1) 8% (2) 15% (4)

0% (0) 0% (0) 0% (0)

participants found the approach helpful. The participant who indicated not understanding the ePortfolio approach did not respond to the question regarding its helpfulness. The interviews provided more insight into the reasons for the poor adoption of the approach in pilot 2. Fourteen participants were interviewed; all reported placing a high value on receiving and learning from student feedback, yet only five of the 14 created an ePortfolio. Of the nine participants who did not build an ePortfolio, two missed the orientation, and one stated lack of interest due to imminent retirement. The other six indicated limited time and did not see the creation of an ePortfolio as a priority. Three interviewees put a few documents into Mahara, such as an ePortfolio page and some teaching materials. The remaining two interviewees built a more comprehensive ePortfolio, including some documentation of achievements and projects they were working on. None of the interviewees included student feedback in their ePortfolio. These findings suggest that adoption of the approach in pilot 2 was less than during pilot 1. Faculty perception of value of the ePortfolio approach in pilot 1 In response to the question of whether participants found the ePortfolio approach helpful for their professional development (see Table 7), 47% found the approach helpful or very helpful; 46% found the approach not so helpful; 8% found the approach not helpful at all. Nineteen out of the 26 respondents provided an answer to the open-ended question: What did you find helpful and/or not helpful about the ePortfolio approach? Seven indicated they found it helpful that the approach encouraged reflective practice. Responses in this category included ‘‘making reflection and action more deliberate and focused was helpful,’’ and ‘‘allows you to reflect on what you are doing.’’ An additional three respondents indicated they appreciated how the ePortfolio allowed them to collate relevant documents where the supervisor could see them; it was ‘‘helpful that everything could be in one place.’’ Four of the 19 comments indicated that participants found the approach unhelpful, redundant, or cumbersome though none provided a rationale. The remaining five responses commented on the use of Mahara and the need for more training or practice. The interview data provided more in-depth insight into the faculty experiences. From the 12 participants interviewed, three did not see any benefit or added value in the ePortfolio approach, either because they were not interested in learning due to imminent retirement or because the approach was similar to the learning processes they usually engaged in. Nine participants saw added value in the ePortfolio approach. Five of those nine thought the ePortfolio was a good opportunity to store feedback, reflections, and achievements in one place. Leslie liked the ePortfolio because ‘‘It is good to have a place to put all that kind of stuff down, your goals and even just Table 4 Responses to the question: Did you include your feedback, reflections, and evidence of accomplishments in Mahara? Response

Yes

Most of it

Some of it

No

Yes

8% (2)

19% (5)

46% (12)

27% (7)

your plans for the department.’’ Four participants thought that the ePortfolio approach was a good opportunity to reflect on performance for the purpose of identifying any gaps or areas for future growth. Three participants felt the ePortfolio approach was helpful in making supervisors aware of the work their faculty members are involved in. For example, Jamie expressed: And that’s where I do see the point where the supervisor might see that, Oh, wow. You’ve done this. Have you done this? Have you done this? You know they had forgotten, or they did not know because they’re busy managing their department. Three participants indicated it was nice to get good performance reaffirmed. Addy: ‘‘It reassured me that I was doing some things right.’’ After receiving feedback using the department survey for chairs, chair MacKenzie expressed that the ePortfolio process was a ‘‘good opportunity for the leaders to ask for assistance’’ from their own supervisors. A sub-theme of perceived value of ePortfolio approach was the view of Mahara, the online software used by the participants to create their ePortfolios. One participant did not access or use Mahara at all and therefore did not express any views on the software. Two participants felt that Mahara was difficult to learn. Neither of these participants used Mahara to create an ePortfolio because neither felt comfortable with the software. Addy explains: Back to Mahara, I guess I’m not a social media person. I don’t use Facebook. I don’t use MyPage or any of that, so I really wasn’t invested in it. If I was applying for a different job, I could see it, right. I wasn’t comfortable enough to use it for, like, a [performance review] presentation to my boss. Four participants did not see much added value in Mahara. Two participants did not like the process of uploading paper documentation to Mahara’s web site. Alex explains: ‘‘I think it would be just much easier just to have a folder and have a file in your hard drive just to deposit stuff.’’ Other participants felt that Mahara was too much work to learn. For example, Jamie expressed: ‘‘It did more than I needed it to do, and it was one more piece of software that I need to learn.’’ Despite these concerns, three of these participants created an ePortfolio using the software. Finally, five participants thought Mahara was easy to learn. For example, Tracy stated: ‘‘I think the tool was easy.’’ Faculty perception of value of ePortfolio approach in pilot 2 The participants in the second pilot saw less value in the ePortfolio approach, and they raised a number of concerns that were absent in the first pilot. Table 8 shows that approximately a quarter of the participants found the ePortfolio approach helpful for professional development, setting learning goals, assessing performance, and receiving coaching and support from their supervisor. The majority of respondents were undecided or did not find the approach useful or helpful at all. Ten participants took the opportunity to provide written comments to the questions in Table 8. They indicated that the ePortfolio was introduced at the wrong time (five comments) and that they were uncomfortable with posting information online (one comment). Three commented that the ePortfolio is not a good

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

8 Table 5 Time spent on ePortfolio approach in pilot 2.

How much time did you spend on the ePortfolio approach as a whole, including setting learning goals, collecting feedback, reflecting on feedback, and documenting these in Mahara?

<1 h

1–2 h

3–5 h

6–10 h

>10 h

27% (10)

22% (8)

35% (13)

5% (2)

11% (4)

Table 6 Cross-tabulation of participants’ perception of their understanding of the purpose and principles underlying the ePortfolio approach, and the question: ‘‘Did you use Mahara as an ePortfolio tool?’’. Response

I’m sure I understand the purpose and principles entirely I’m not sure I entirely understand the purpose and principles I don’t understand the purpose and principles of the ePortfolio approach Total a

Used Mahara Yes

No

24% (10)

24% (10)

29% (12)

21% (9)

Total

N/Aa

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

48% (20)

11% (2)

21% (4)

16% (3)

16% (3)

32% (6)

5% (1)

50% (21)

10% (2)

5% (1)

35% (7)

15% (3)

20% (4)

15% (3)

0% (0)

2% (1)

2% (1)

52% (22)

48% (20)

100% (42)

(0)

(0)

10% (4)

13% (5)

(0)

26% (10)

(0)

15% (6)

(0)

(0)

26% (10)

10% (4)

N/A = Not Applicable, SA = Strongly Agree, A = Agree, U = Undecided, D = Disagree, SD = Strongly Disagree.

use of time because only the format and not the content was discussed with the chair (one comment) or because they prefer to use a different tool for professional development (two comments). The remaining comment questioned whether the ePortfolio approach would actually add to faculty accountability or not: ‘‘We can choose what our supervisor sees, so we can just pick the best survey [results].’’ Of the 14 interview participants, only one interviewee, Dallas, expressed seeing a clear benefit to the approach. S/he believed in the ePortfolio’s abilities to create an overview of accomplishments and support reflection: I think, you know, a few years down the road it will be a nice timeline. Oh, I was working on this, [. . .] I’ve worked more into – I’ve been helping with some research here and this here, you know. So I think it will be an excellent tool for me to reflect on what I’ve been doing.

expressed mostly concerns with the approach. Francis explains how s/he normally goes about improving practice and how an ePortfolio would help in this process: I’m usually quite cognizant of getting feedback and reflecting. [. . .]. So I generally do it quite often, and I do it verbally quite a bit. [. . .] Sometimes I have the students kind of just write on a little sticky note what worked, what didn’t. [. . .] I kind of run some exercises by some colleagues before I try them in the class to see if it would work or not, [. . .] And then I create lesson plans where I have little [. . .] At the bottom I just have a little area for comments[. . .] And the purpose is eventually to be able to [. . .] kind of put that into some type of portfolio as part of the reflection. Hasn’t happened yet, yeah. While s/he values reflection, Francis did not create an ePortfolio online:

Dallas wished for more time to work on the ePortfolio and was eager to continue using the approach. The other four interview participants who had created at least one ePortfolio page

So I – like I said, the student feedback came in. I looked at it, I addressed it personally, and I addressed it in the classroom. [. . .]. I didn’t do anything with that feedback with the ePortfolio, unfortunately. I kind of had a bit of an – I guess a reservation

Table 7 Responses to the question about helpfulness of the ePortfolio approach.

To what extent did you find the ePortfolio approach helpful for your own professional development?

Very helpful

Helpful

Not so helpful

Not helpful at all

12% (3)

35% (9)

46% (12)

8% (2)

Table 8 Responses to the question: Please indicate to what extent you agree with the following statements.

I like the ePortfolio approach I found the ePortfolio approach helpful for my professional development The ePortfolio approach is a useful way to set learning goals The ePortfolio approach is a useful way to assess performance The ePortfolio is a useful way to receive coaching and support from my supervisor

Strongly Agree

Agree

Undecided

Disagree

Strongly disagree

Not applicable

10% (4) 10% (4)

18% (7) 13% (5)

33% (13) 26% (10)

8% (3) 15% (6)

26% (10) 26% (10)

5% (2) 10% (4)

3% (1) 3% (1) 3% (1)

21% (8) 18% (7) 20% (8)

28% (11) 23% (9) 30% (12)

13% (5) 25% (10) 15% (6)

28% (11) 25% (10) 25% (10)

8% (3) 8% (3) 8% (3)

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

about the ePortfolio about the fact that it’s online. I’m not really comfortable with putting information online. The interviewees who did not create an ePortfolio were not asked what they found helpful about the approach. One of the interviewees volunteered his expectation that it the approach would be useful to keep a record. A total of six of the interviewees reported concern over putting their information online. Robin, one of these six, also expressed discomfort with having to learn a new software tool. Quinn would rather the information be stored on a server hosted at the institute: ‘‘And then the other thing is who’s going to see my information in the long-term? How secure is it? Will my students have – be able to hack into it or access it or whatever?’’ Harper wondered whether the information might be accessed by upper management who might use it for salary decisions. While the taskforce had not wanted to be prescriptive in what participants should include in their ePortfolio, three interviewees expressed they were unsure what an ePortfolio might look like and expressed a desire for direction. Kelly indicated: ‘‘For sure, templates would help and, you know, without knowing the full details on ePortfolios, but templates and what’s expected throughout the whole and what you expect the finished product to look like.’’ Robin expressed questions about ways to provide evidence of interpersonal skills: And is it a true evaluation of my skills, I guess, is the other thing. I just kind of feel – I think I bring something unique to the classroom. I think I’m really personable with my students. I care a lot about them. I personally talk to them a lot. I end up being a counsellor most days. But, again, I don’t know how that – how those two mesh. Interviewer: So how can you show what your strengths are if they’re not in an actual project that you can post? Robin: Right. Exactly. Perceived impact of ePortfolio approach in pilot 1 After pilot 1, when asked whether they used the feedback they collected per aspect to improve their performance, more than 75% of respondents to whom this aspect applied answered affirmatively, with the exception of corporate citizenship and applied research (see Table 9). Although respondents who replied ‘‘no’’ did not provide an explanation, at least one participant, Jean, indicated the feedback was all positive and therefore not used to improve work:

Table 9 Responses to the question: Did you use the feedbackb to improve your work? Yesa Teaching Professionalism Corporate citizenship Scholarship/professional development Applied research Formal curriculum development Formal administrative duties Formal leadership a

81% 85% 56% 86%

No (17) (17) (5) (12)

19% 15% 44% 14%

Not applicable (4) (3) (4) (2)

(2) (2) (11) (7)

33% (1) 91% (10)

67% (2) 9% (1)

(13) (6)

75% (6)

25% (2)

(8)

89% (8)

11% (1)

(9)

Respondents could only select one option: yes, no, or not applicable. One respondent did not provide any answer while some did not provide an answer for some of these aspects. b This question was attached to the questions in Table 1; thus ‘the feedback’ refers to the type of feedback the participants collected as indicated in Table 1.

9

And the feedback I got was very positive. Nothing that surprised me because I talked to my students all the time, and I know where they’re at. So I didn’t personally find that feedback to be particularly of value to me. Of the 12 pilot 1 participants interviewed, two indicated they did not adopt the approach, experienced no impact, and did not intend to use the approach. A second group of two participants reported that, among other things, the approach helped make their chair aware of all the work they did. For instance, Jamie reported: And there’s a reason why I’m putting all this stuff in. Because I’m pretty sure our supervisor [the department chair] has a pretty good idea of what we’re doing, but sometimes he/she might not remember just all the work that all of us staff do. Six participants reported their participation in the ePortfolio approach has helped them become aware of areas of improvement to their practice. Chair MacKenzie reported, for instance, ‘‘It helped me pinpoint things that I need to work on for sure, yeah.’’ Addy, also a chair, described an improvement in practice as a result of feedback received: So I may talk to one associate chair and not the other. So one hand doesn’t know what the other hand is doing. So with that in mind, then I try to be more inclusive now [that I received that feedback] because I really wasn’t that conscious of it. Three participants reported that the ePortfolio approach made them think more explicitly about actively planning their professional development. For instance, Jean indicated: ‘‘I think the one thing it did for me was, it made me think, ‘‘Do I document enough?’’ Leslie commented: But, yeah, it does help me to outline my goals. Definitely it helped me to identify new ideas and kind of have a place to put them down and things that I want to do with the program to, you know – in terms of organization of the second year, clinical experience, different ways that we can [keep in touch with students]– because our students are so far away. Finally, Jordan indicated: Now I’ve got things that I’m thinking, I could do that to get better feedback, or I could do this. So it helped me think about what I’m going to do in the future for my professional development, definitely. It also identifies what I could probably use more training on. So it kind of helps me plan. Perceived impact of ePortfolio approach in pilot 2 As indicated previously, the adoption of the ePortfolio approach in the second pilot was limited. Because only one of the interviewees in pilot 2 had actually developed an ePortfolio, we did not analyze the interviews for impact of the ePortfolio on professional development. Our survey data (Table 10) show that similar to the first pilot, a majority of participants reported the feedback they collected impacted their practice. Discussion This paper reports on the design, implementation, and evaluation of two pilot studies of a faculty development and performance evaluation initiative called the ePortfolio approach at a Western Canadian institute for vocational education. In a response to calls for more detailed description of faculty development initiatives (Steinert et al., 2006; Stes et al., 2010) design principles were described as well as the stakeholder consultation process used. The purpose of the ePortfolio approach

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

10

Table 10 Responses to the question: Please indicate to what extent you agree with the following statements.

The feedback I received in the summary student feedback report influenced the way I teach The feedback collected through the chair survey influenced the way I supervise

Strongly agree

Agree

Undecided

Disagree

7% (2)

57% (16)

21% (6)

14% (4)

0% (0)

0% (0)

0% (0)

57% (4)

29% (2)

0% (0)

14% (1)

0% (0)

was to foster naturally occurring professional learning activities of participating faculty members. The ePortfolio approach was piloted in two rounds. The research questions addressed in this paper were: (1) To what extent was the ePortfolio adopted in the programs participating in the first and second pilot? (2) To what extent did faculty in the first and second pilot value the ePortfolio approach as a whole? (3) What was the impact of the ePortfolio approach on the professional development of participating faculty members? This section discusses our main findings, lessons learned, and implications for research. Added value of feedback In terms of the adoption of the ePortfolio approach, the majority of participants in both pilots indicated that they collected feedback, reflected on that feedback, and used it to improve their practice. They most likely would have collected some type of feedback if they had not been participating in the pilot; however, the results suggest that due to the pilot, they collected more and/or different types of feedback than they did previously. The participating department chairs reported finding value in gathering feedback by surveying the faculty in their department regarding their responsibilities as a chair, a practice that previously had not been common throughout the institute. All participants interviewed explained that they greatly valued learning from feedback, and many described ways in which they collected and discussed feedback with students in order to improve the students’ learning experience. This concurs with similar findings by Smith and Tillema, who concluded: Another aspect which attracted much attention in the face-toface interviews was the importance of feedback during the portfolio collection process. In three out of the four settings (dossier, training and reflective portfolios) external feedback is seen as a positive and essential part of the process. (p. 646) By encouraging faculty to collect more and different types of feedback, the ePortfolio approach did indeed foster a naturally occurring professional learning process of faculty. Creating an ePortfolio In both pilots, less than half of the participants who responded to the survey actually documented their feedback and reflections in the online ePortfolio tool Mahara, though the adoption of the ePortfolio approach was greater in the first pilot than in the second. Our participants’ weariness of the portfolio software and the fact that it exists online conflicts with Freeman et al. (2009) who report the majority of their study participants had positive perceptions of their ePortfolio technology. This difference in reception could be explained by the fact that the participants in the pilot Freeman et al. conducted all volunteered to use the ePortfolio software. Our participants, on the other hand, were included in the pilot because the department chairs agreed for the faculty in an entire program to participate in the pilot. This concurs with Smith and Tillema’s (2003) findings that the mandatory or voluntary nature of a

Strongly disagree

Not applicable

portfolio was a key contextual factor affecting the acceptance and uptake of portfolios by professionals. Goliath (2009) studied adoption of an ePortfolio by physicians. Similar to our findings, she described that individual factors such as comfort level with technology influenced participants’ adoption of the ePortfolio technology. Another reason for limited adoption that was mentioned by many participants in the second pilot was a lack of time. During the second pilot, an institute-wide curriculum redevelopment process was implemented. This has, at least in part, accounted for the fact that adoption was greater in the first pilot than in the second. Aside from a lack of time or interest, the participants provided other reasons for why they chose not to build an ePortfolio. A number of them found it cumbersome to learn the software, whereas others expressed concerns about the privacy and security of Mahara, a lack of trust in the organization where online information may be accessed and used for purposes other than formative assessment, and a lack of clarity about what kind of information could be included in an ePortfolio. These concerns could have been alleviated with additional preparation and support sessions. The need for advance preparation is echoed by Blair (2001), who identified separate workshops on using teaching portfolios for professional development and creating an electronic portfolio as being key to the success of an ePortfolio as a faculty development tool. Smith and Tillema (2003) also pointed to the importance of clarity of expectations: ‘‘The trust in the portfolio was strengthened when there were explicit criteria for the content of the portfolio, how to go about it and of how the portfolio was going to affect their professional status’’ (p. 644). This finding also confirms the need for clear performance expectations (e.g., Higgerson, 2006). Despite the challenges with the tool, participants reported that creating an ePortfolio was an opportunity for them to: (1) Identify issues and ask assistance from supervisor in dealing with these issues. (2) Collect more and different types of feedback than before. (3) Store learning goals, achievements and other information relevant to their development in one place. (4) Store feedback and accomplishments over time, which allows one to look back and reflect on performance and also allows for the formulation of goals for future improvement. (5) Show the supervisor all the different tasks one is involved in. The above benefits reported by our participants are consistent with Granberg’s (2010) findings, which established that student teachers expressed benefits of using ePortfolios as: portfolios for learning (opportunities 1 & 2 discussed above), portfolios as an archive (3 & 4), and portfolios for assessment (5). Impact of the ePortfolio approach One perceived impact of the approach involved an increased awareness by participants of areas of practice that could use improvement. Participants also thought more explicitly about planning their professional development. These findings suggest

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

that if adopted fully, the ePortfolio approach has the potential to enhance self-directed learning. This concurs with the findings of Goliath (2009) who found that all nine physicians in her study scored higher on the readiness scale for self-directed learning as a result of creating an ePortfolio. It also concurs with the impact Smith and Tillema (2003) found in that the personal development type of portfolio gave ‘‘participants space and opportunity for selfdirected learning, which was valued as contributing to professional growth’’ (p. 645). More research is needed to study whether these impacts can be sustained over time.

11

Fourthly, a four-month period might be too short to really try out the ePortfolio approach. Since the approach assumes a yearly cycle, and given that in the first cycle participants and department chairs are still learning the process, a two-year pilot might be more useful and could include an initial orientation and a number of follow up sessions. Regular formative feedback from the pilot participants on the approach itself, as well as the support sessions, could allow the faculty development support staff and the department chairs to address concerns as they arise. Limitations

Design and implementation, lessons learned Many faculty development initiatives serve a limited number of faculty members over a limited amount of time (e.g., Stes et al., 2010). Because of its potential to enhance reflective practice and self-directed learning (e.g., Goliath, 2009), an initiative like the ePortfolio approach could be considered as a way to systematically support the professional learning of all faculty members and over a prolonged period of time, possibly throughout their entire career at the institute. However, upon the evaluation of the two pilots of the ePortfolio approach, questions remain about whether the ePortfolio would be suitable for institute-wide implementation and, if so, how it should be implemented. First of all, based on the findings, the question arises whether the ePortfolio approach should be maintained in its current form. Participants indicated learning the ePortfolio software, Mahara, is cumbersome. In addition, they were wary of posting information online. Sharing portfolio documentation with their department chair does not require the use of ePortfolio software, nor does it require online storage. The literature suggests that composing an ePortfolio allows for enhanced reflective practice (e.g., Lamont, 2007), but the question is whether the extent to which this reflection is enhanced by the use of the software warrants the costs involved in using that software. These costs include online storage fees for third party storage provider, cost of technical support, cost of development and delivery of faculty workshops, as well as the time spent by faculty to learn to use the software. Our pilot data did not suggest that the software added to the quality of reflection but that it was the act of compiling documents that encouraged reflection on and formulating goals for professional development. A simplified, less costly version of an ePortfolio could potentially serve the same purpose. This simplified version could, for instance, include a template with questions for reflection and performance assessment to be filled out by faculty, along with a number of appendices including feedback, reflections, teaching materials, and documentation of achievements and other records of performance. Faculty could compile and store these documents on their own computers and show them to their chair during a performance assessment meeting. Secondly, the findings of both pilots point to lack of time. Wider implementation of the ePortfolio approach would need to be carefully timed and not coincide with other drastic institute-wide innovations. Thirdly, our findings also highlight how the supervisors and their understanding of the approach played a role in determining the need of their faculty for support and in setting expectations for their department members. Because of the important role they play, department chairs might need more guidance and preparation before the approach is introduced in their departments. In general, department chairs might play a larger role in fostering faculty development: for instance, by encouraging transfer of learning in courses and workshops into the workplace. (See, for instance, Lancaster, Di Milia, & Cameron (2012). Chairs would need to set clear expectations for both the content of the ePortfolio as well as how this content will be assessed.

The study of the impact of the ePortfolio approach was limited to self-reported outcomes described in interviews. The findings regarding the impact of the approach are limited because few participants actually adopted the approach in the way it was intended. The interview guide and survey used were designed to directly inform practice and required data collection, analysis, and reporting to be completed within three months. Studying the participants’ ePortfolios themselves could have increased our insight into the impact of the approach on professional learning, but this was considered to create an artificial environment where participants might feel the need to create an ePortfolio just for the researchers. In addition, it might have created undue stress in participants as they might be afraid the researchers would evaluate their performance rather than the impact of the ePortfolio on their professional learning. Other limitations of our methods include the fact that the pilot 2 survey included improved survey questions, which allowed the taskforce to gain more detailed insight into participants’ experiences, but unfortunately prevented the researchers from comparing survey data across pilots. Finally, our recruitment method for the survey in both pilots, and the interviews in the second pilot, may have contributed to sample bias. For the second pilot, we know that non-response was most likely due to lack of time. However, we do not have conclusive information about this, as research ethics procedures stipulate that survey and interview participation is voluntary, and no reason has to be provided for not responding. Implications for research While the use of portfolios for student assessment and student learning is taking hold in several disciplines, its use in supporting ongoing professional learning of post-secondary educators so far has been limited. Yet, the literature suggests it could be a valuable approach to supporting faculty learning throughout their careers. Further studies are needed to determine the organizational conditions facilitating the implementation of the ePortfolio approach, including issues of power and human resources processes. In addition, studies are needed to determine the design requirements for the ePortfolio approach that optimally facilitates self-reflection and self-directed learning, requires minimal learning on the part of the faculty, and facilitates performance assessment. Carefully designed studies investigating the impact of a well-designed ePortfolio tool accompanied by ePortfolio assessment processes should be able to shed light on the impacts of such an approach on faculty development and ultimately on students’ learning experience. Conclusion Our study suggests that an ePortfolio approach to faculty development and performance evaluation has the potential to support collection of various types of feedback, increase awareness of areas of improvement, and encourage more explicit thought about planning and monitoring of professional development.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 12

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

When adopted by a whole organization, the approach could integrate professional development plans, student feedback practices and other feedback on performance, as well as performance review processes. With careful implementation, a well-designed ePortfolio process potentially constitutes an effective learner-centred approach to faculty development and performance evaluation. Acknowledgements A heartfelt thanks to the pilot participants who provided their thoughts and feedback in our interviews and surveys. Many thanks to the taskforce members for their collaboration with the research team. Thanks to the students in the Captioning and Court Reporting program for their assistance with interview transcription. We are grateful for the thoughtful input and editing assistance from Susan Chaudoir, Jeffrey Kuntz, and Jennifer Semchuk in the final stages of writing. And finally, thanks to the blind reviewers for their helpful comments. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.stueduc.2015.03.007. References Amundsen, C., & Wilson, M. (2012). Are we asking the right questions? A conceptual review of the educational development literature in higher education. Review of Educational Research, 82(1), 90–126. http://dx.doi.org/10.3102/0034654312438409 Axelrod, R. H. (2010). Terms of engagement: New ways of leading and changing organizations (2nd ed.). San Francisco, CA: Berrett-Koehler Publishers. Beausaert, S., Segers, M., Fouarge, D., & Gijselaers, W. (2013). Effect of using a personal development plan on learning and development. Journal of Workplace Learning, 25(3), 145–158. Billett, S. (2002). Critiquing workplace learning discourses: Participation and continuity at work. Studies in the Education of Adults, 34, 56–68. Blair, K. (2001). Electronic portfolios in tenure and promotion decisions: Making a virtual case. In Retrieved from ERIC database (ED 466 138). Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A metaanalytic review. Journal of Management, 36, 1065–1105. http://dx.doi.org/10.1177/ 0149206309352880 Bound, H. (2011). Vocational education and training teacher professional development: Tensions and context. Studies in Continuing Education, 33(2), 107–119. http://dx.doi.org/10.1080/0158037X.2011.554176 Brookfield, S. D. (1995). Becoming a critically reflective teacher. San Francisco, CA: JosseyBass. Butler, P. (2006). A review of the literature on portfolios and electronic portfolios. Palmerston North: Massey University College of Education Retrieved from hhttp://www.eportfoliopracticequt.edu.au/docs/Butler%20-%20Review%20of%20lit%20on%20ePortfolio%20research%20-%20NZOct%202006.pdfi. Bye, B., Pushkar, D., & Conway, M. (2007). Motivation, interest, and positive affect in traditional and nontraditional undergraduate students. Adult Education Quarterly, 57(2), 141–158. Carraccio, C., & Englander, R. (2004). Evaluating competence using a portfolio: A literature review and web-based application to the ACGME competencies. Teaching and Learning in Medicine, 16(4), 381–387. Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and Teacher Education, 18, 947–967. http://dx.doi.org/10.1016/ S0742-051X(02)0005307 Cox, M. D., & Richlin, L. (Eds.). (2004). Building faculty learning communities: New directions for teaching and learning. San Francisco, CA: Jossey-Bass. Cross, K. P., & Steadman, M. H. (1996). Classroom research: Implementing the scholarship of teaching. San Francisco, CA: Jossey-Bass. Darwin, S. (2007). The changing contexts of vocational education: Implications for institutional vocational learning. International Journal of Training and Research, 5(1), 55–71. Day, C. (1999). Developing teachers: The challenges of lifelong learning. London: Falmer Press. Devanas, M. A. (2006). Teaching portfolios. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 111–130). San Francisco, CA: Jossey-Bass. Engestro¨m, Y. (2011). Activity theory and learning at work. In M. Malloch, L. Cairns, K. Evans, & B. N. O’Connor (Eds.), The SAGE handbook of workplace learning (pp. 86– 104). Los Angeles, CA: SAGE. Eraut, M. (2004). Informal learning in the workplace. Studies in Continuing Education, 26(2), 247–273. http://dx.doi.org/10.1080/158037042000225245

Flanigan, E. J. (2012). ePortfolios and technology: Customized for careers. International Journal of Information and Communication Technology Education, 8(4), 29–37. Freeman, L., Harkness, M., & Urbaczewski, A. (2009). International Journal of Global Management Studies, 1(2), 31–46. Fuller, A., & Unwin, L. (2011). Workplace learning and the organization. In M. Malloch, L. Cairns, K. Evans, & B. N. O’Connor (Eds.), The SAGE handbook of workplace learning (pp. 46–59). Los Angeles, CA: SAGE. Granberg, C. (2010). E-portfolios in teacher education 2002–2009: The social construction of discourse, design and dissemination. European Journal of Teacher Education, 33(3), 309–322. Goliath, C. L. (2009). Diffusion of an e-Portfolio to assist in the self-directed learning of physicians: An exploratory study. University of Akron (Dissertation). Halonen, J. S., & Ellenberg, G. B. (2006). Teaching evaluation follies: Misperception and misbehavior in student evaluation of teachers. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 150– 165). San Francisco, CA: Jossey-Bass. Harris, R., Simons, M., Hill, D., Smith, E., Pearce, R., et al. (2001). The changing role of staff development for teachers and trainers in vocational education and training. Leabrook, Australia: The National Centre for Vocational Education Research (NCVER) Ltd. Retrieved from hhttp://www.ncver.edu.au/publications/595.htmli. Higgerson, M. L. (2006). Building a climate for faculty evaluation that improves teaching. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 35–49). San Francisco, CA: Jossey-Bass. Hoekstra, A., Beijaard, D., Brekelmans, J. M. G., & Korthagen, F. A. J. (2007). Experienced teachers’ informal learning from classroom teaching. Teachers and Teaching: Theory and Practice, 13(2), 189–206. http://dx.doi.org/10.1080/13540600601152546 Hoekstra, A., Brekelmans, M., Beijaard, D., & Korthagen, F. (2009). Experienced teachers’ informal learning: Learning activities and changes in behaviour and cognition. Teaching and Teacher Education, 25(5), 663–673. http://dx.doi.org/10.1016/ j.tate.2008.12.007 Hoekstra, A., & Korthagen, F. (2011). Teacher learning in a context of educational change: Informal learning versus systematically supported learning. Journal of Teacher Education, 62(1), 76–92. http://dx.doi.org/10.1177/0022487110382917 Hubball, H. T., Pratt, D. D., & Collins, J. B. (2006). Enhancing reflective teaching practices: Implications for faculty development. The Canadian Journal of Higher Education, 35(3), 57–81. Jasper, M., & Fulton, J. (2005). Marking criteria for assessing practice-based portfolios at masters’ level. Nurse Education Today, 25(5), 377–389. Jimerson, J. B. (2014). Thinking about data: Exploring the development of mental models for ‘‘data use’’ among teachers and school leaders. Studies in Educational Evaluation, 42, 5–14. http://dx.doi.org/10.1016/j.stueduc.2013.10.01 Johnson, B. W. (2006). On being a mentor: A guide for higher education faculty. Mahwah, NJ: Lawrence Erlbaum Associates. Kirkpatrick, D. (2007). Four levels of evaluation: Measurement and evaluation. Pewaukee, WI: American Society for Training and Development. Lamont, M. (2007). What are the features of e-portfolio implementation that can enhance learning and promote self-regulation? Paper presented at the European Institute for E-Learning (EIfEL) conference. Lancaster, S., Di Milia, L., & Cameron, R. (2012). Supervisor behaviours that facilitate training transfer. Journal of Workplace Learning, 25(1), 6–22. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lohman, M. C., & Woolf, N. H. (2001). Self-initiated learning activities of experienced public school teachers: Methods, sources, and relevant organizational influences. Teachers and Teaching: Theory and Practice, 7(1), 59–74. Marsick, V. J., & Watkins, K. E. (1990). Informal and incidental learning in the workplace. London/New York: Routledge. McColgan, K., & Blackwood, B. (2009). A systematic review protocol on the use of teaching portfolios for educators in further and higher education. Journal of Advanced Nursing, 65(12), 2500–2507. McGovern, T. G. (2006). Self-evaluation: Composing an academic life narrative. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 96–110). San Francisco, CA: Jossey-Bass. Millis, B. J. (2006). Peer observations as a catalyst for faculty development. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 82–95). San Francisco, CA: Jossey-Bass. Murray, J. P. (1997). Successful faculty development and evaluation: The complete teaching portfolio. In ERIC Document Reproduction Service No. ED405759. Washington, DC: Graduate School of Education and Human Development, The George Washington University. O’Meara, K. A., & Braskamp, L. (2005). Aligning faculty reward systems and development to promote faculty and student growth. National Association of Student Personnel Administrators Journal, 42(2), 223–240. Opfer, V. D., & Pedder, D. (2011). Conceptualizing teacher professional learning. Review of Educational Research, 81(3), 376–407. http://dx.doi.org/10.3102/ 0034654311413609 Pallett, W. (2006). Uses and abuses of student ratings. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 50–65). San Francisco, CA: Jossey-Bass. Polytechnics Canada (2014). Research and innovation. Retrieved from hhttp://www.polytechnicscanada.ca/polytechnic_advantage/applied_research_commercialization/ ProgramsCourses/FacultiesSchoolsCentres/TeachingLearning/Departments/ Academici. Roberson, D. N., & Merriam, S. B. (2005). The self-directed learning process of older, rural adults. Adult Education Quarterly, 55(4), 269–287. http://dx.doi.org/10.1177/ 0741713605277372

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007

G Model

JSEE-562; No. of Pages 13 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx Sambrook, S. (2005). Factors influencing the context and process of work-related learning: Synthesizing findings from two research projects. Human Resource Development International, 8(1), 101–119. http://dx.doi.org/10.1080/1367886052000352591 Scho¨n, D. (1983). The reflective practitioner: How professionals think in action. New York, NY: Basic Books. Seijts, G., Taylor, L., & Latham, G. (1998). Enhancing teaching performance through goal setting, implementation and seeking feedback. International Journal for Academic Development, 3(2), 156–167. Seldin, P. (2006). Essential operating principles and key guidelines. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research, and service (pp. 20–34). San Francisco, CA: Jossey-Bass. Shuell, T. J. (1986). Cognitive conceptions of learning. Review of Educational Research, 56, 411–436. Smith, K., & Tillema, H. (2003). Clarifying different types of portfolio use. Assessment and Evaluation in Higher Education, 28(6), 625–648. Statistics Canada (2010). Projections of the diversity of the Canadian population, 2006 to 2031. Ottawa, ON: Author. Steinert, Y. (2000). Faculty development in the new millennium: Key challenges and future directions. Medical Teacher, 22(1), 44–50. Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., et al. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. Medical Teacher, 28(8), 497– 526. Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of instructional development in higher education: The state-of-the-art of the research. Educational Research Review, 5(1), 25–49. http://dx.doi.org/10.1016/ j.edurev.2009.07.001 Stiehl, R., & Lewchuk, L. (2008). The outcomes primer: Reconstructing the college curriculum. Corvallis, OR: The Learning Organization. Taylor, L., & Rege-Colet, N. (2009). Making the shift from faculty development to educational development: A conceptual framework grounded in practice. In A.

13

Saroyan & M. Frenay (Eds.), Building teaching capacities in higher education: A comprehensive international model. Sterling, VA: Stylus Publishing. Van Woerkom, M., & Croon, M. (2008). Operationalising critically reflective work behavior. Personnel Review, 37, 317–331. Wayman, J. C., & Jimerson, J. B. (2014). Teacher needs for data-related professional learning. Studies in Educational Evaluation, 42, 25–34. http://dx.doi.org/10.1016/ j.stueduc.2013.11.001 Webster-Wright, A. (2009). Reframing professional development through understanding authentic professional learning. Review of Educational Research, 79, 702–739. http://dx.doi.org/10.3102/0034654308330970 Zeichner, K., & Liu, Y. (2010). A critical analysis of reflection as a goal for teacher education. In N. Lyons (Ed.), Handbook of reflective inquiry. New York, NY: Springer.

Annemarieke Hoekstra Ph.D., is a Teaching and Learning Specialist at the Northern Alberta Institute of Technology (NAIT). Her research interest is in teacher professional learning. She currently holds a grant from the Canadian Social Sciences and Research Council to study the professional learning of vocational educators. She obtained her Ph.D. in Education from Utrecht University in 2007. Her dissertation focused on informal teacher learning in the workplace. From 2007 until 2009, she worked as an instructor in the JR Shaw School of Business at NAIT.

Jocelyn R. Crocker M.Ed., has been an instructor with the department of Biological Sciences at the Northern Alberta Institute of Technology (NAIT) since 2004. She has received multiple teaching awards including the NAIT Instructional Excellence Award. She obtained a Master of Education from the Department of Secondary Education at the University of Alberta in 2010; her thesis focused on the experiences of foreign-trained professional women in adult upgrading. Her current research interests are in metacognition and teacher professional learning.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007