- Email: [email protected]
Going Provincial: Preceptoring and Mentoring Revisited in the Health Sciences
BRIEF REPORT
Going Provincial: Preceptoring and Mentoring Revisited in the Health Sciences Part II – The Evaluation Carol Mitchell, MSN, was the Project Manager of the Preceptor & Mentor Initiative for Health Sciences in BC, Vancouver.
Brenda SawatzkyGirling, MHA, is a healthcare policy and management consultant in Vancouver.
by Carol Mitchell and Brenda Sawatzky-Girling
Abstract Part I of this two part series, which was published in the Spring 2003 issue of Healthcare Management FORUM, described the Preceptor & Mentor Initiative for Health Sciences in BC, a provincial strategy to facilitate preceptoring and mentoring. Activities included creation of an educational web site and a UBC Continuing Studies preceptor course, and provincial workshops on preceptoring and mentoring. The purpose of this second article is to outline the philosophy, framework, results and interpretation of the Preceptor & Mentor Initiative’s evaluation component.
T
he Preceptor & Mentor Initiative for Health Sciences in BC Project Team developed an evaluation plan during the initial program planning phase in 2001. To bring objectivity to this critical function, the evaluation framework was designed with input from, but independent of, the Project Team. We needed representative evidence from health and education professionals in BC to assess whether the following objectives of the Initiative had been met: • to increase recruitment and retention of health sciences professionals; • to build capacity to educate these same professionals; • to contribute to a culture of lifelong learning; and • to address the needs of many diverse groups.
In addition, the evaluation was designed to elicit recommendations on future preceptoring and mentoring structures and activities in the province. The evaluation approach was based on the quality movement [structureprocess-outcome] and aligned with a logic model. When translated into recommendations, these findings become a decision-making tool for policy makers to determine the Initiative’s value. We expect that such a program’s future depends not only on its performance but also on whether the program goals remain desirable given competing priorities.1
Evaluation Framework/Process Our evaluation framework, a Program Logic Model of Preceptor & Mentor Initiative, was designed by a doctoral student at UBC’s Centre for Health Services and Policy Research (see figure 1). Although the long-term outcomes are complex and difficult to measure, the Logic Model was designed to provide clear and explicit criteria for success as indicated by the short-term outcomes. The literature indicates the short-term outcomes identified
28
Healthcare Management Forum Gestion des soins de santé
Figure 1: PROGRAM LOGIC MODEL OF PRECEPTOR & MENTOR INITIATIVE Component
Process
Output
Needs assessment
Baseline assessment survey
Outline of current status of programs in the province
Consultation with stakeholders
Outline of needs of stakeholders in developing programs
Baseline assessment survey; collecting existing resources
Outline of theoretical concepts Outline of best practices
Compilation and development of guidelines and resources
Dissemination of information
Networking of users
Outcome Short-term
Contribution to increased recruitment and retention of nurses and health sciences professionals Broader implementation of preceptor and mentor programs
Guidelines for managing, planning, implementing and evaluating programs that are sensitive to diverse needs (incl. reward and recognition processes and estimated costs)
Literature review
Consultation with stakeholders
Clearinghouse of existing tools and resources
Web site
Web site and users
Workshops to showcase the initiative
Workshops and participants
Provincial and national conferences
Conferences and participants
Web site Interactive discussion forum
Outcome Long-term
Contribution to increased capacity for student placement
More standard implementation of best practices
Web site and users Contribution to culture of continuing learning
Sustaining initiative
Developing business plan and applying for funding
Funding application
Secure funds
Figure 2: Knowledge Management in Action (broader implementation of preceptor and mentor programs, and more standard implementation of best practices) correlate with the Initiative goals.2
Provincial Infrastructure & Coordination
1 Surveys – Needs Assessment, Follow-up New ge wled Kn o
Compilation & Development of Guidelines, Resources
2 Preceptor & Mentor Initiative Web site 3 Resource Team Workshops rm rt Te Sho omes Outc
rm g Te Lon omes Outc
4 From Practitioner to Preceptor Course
Diss em Kno inating wled ge
Conferences Journals
Broader implementation of preceptor & mentor programs More standard implementation of best practices
Research
Support preceptors and mentors in order to: Increase recruitment of health sciences professionals Increase capacity to educate health sciences professionals Increase retention of health sciences professionals Foster a culture of continuous learning Address the needs of diverse groups
Each component in the Logic Model contributed to a Framework for Evaluation Implementation that identified research questions, performance indicators, success level or target dates, and how data would be collected. With this information, we could make process-output linkages. We anticipated early on that data collection would be a challenge because this type of database does not exist provincially, within regions or organizations and only minimally at the department level. The sources of data we used are described below and are illustrated in figure 2.
Healthcare Management Forum Gestion des soins de santé
29
❶ Surveys – We sent an Internet-based baseline survey to BC’s health and education stakeholders in September 2001. A follow-up survey was conducted in February 2003 to compare the two points in time.
Figure 3: Pre & Post: State of Preceptor and Mentor Program Development in BC
❷ Stakeholder Consultations – Our Resource Team of preceptor/mentor experts and end-users grew to include 221 people from around BC. We held 15 workshops (12 of these outside the Vancouver area) to help participants identify what resources they already had and to gather feedback as to whether our activities and plans were meeting their needs. We used this information to continually monitor and refine future workshops and other program activities and outputs. ❸ Web site – The evidence-based Preceptor & Mentor Initiative web site, launched in January 2002, was tracked in terms of utilization and feedback. “We need a ‘one minute preceptor course’ - we have little time for courses!” (Workshop Participant, Prince George BC, February 2002). ❹ Preceptor Course – We developed a preceptor course in a workshop/on-line learning format with UBC Continuing Studies. The From Practitioner to Preceptor course, launched in January 2003, was developed from our web site and the Children’s & Women’s Health Centre of BC preceptor education program. The C&W Program had undergone an extensive evaluation in 2002. Participants provided feedback on the course itself and the Initiative web site.
Results Overall, the activities of the Preceptor & Mentor Initiative proved effective in achieving the desired outcomes. The following describes how we know various Initiative activities directly achieved broader implementation of preceptor and mentor programs in BC and more standard implementation of best practices:
30
❶ Surveys Twenty-three out of 66 (35%) follow-up surveys were completed and returned. Respondents were representative of both health and education sectors and diverse in terms of discipline. In comparing the two points in time between baseline and follow-up surveys (see figure 3), trends indicate that respondents were further along in terms of planning, implementing and evaluating preceptor and mentor programs, notably preceptor programs. ❷ Stakeholder Consultations Workshop evaluation forms and informal comments were overwhelmingly positive.The follow-up survey asked whether respondents had attended or arranged workshops in their areas (15/23 respondents replied yes). Of these 15, 100% found the workshops helpful in developing programs, establishing crucial networks and identifying resources in their areas. ❸ Web site Web site utilization rose steadily since its launch. Web site feedback was systematically collected and the web site revised based on this input:
Healthcare Management Forum Gestion des soins de santé
• Over 1,400 visits were made to the web site with the most intense use in western Canada. A significant number of users visited the web site several times. • On-line feedback forms were minimal, but all were positive. • Eighteen out of 23 follow-up survey respondents had accessed the web site; 17/18 of this group found the web site to be useful to them in implementing preceptor/mentor programs and best practices. • Users who had web site demonstrations in workshops or who used the web site in the preceptor course provided positive comments on how comprehensive and useful the web site was, and that it met their needs. ❹ Preceptor Course This introductory course directly achieves the outcomes of broader implementation of preceptor programs and more standard implementation of best practices. There is demand for the course from different regions of BC. Feedback from the students in the initial course offering regarding quality and usefulness was consistently positive.
Interpretation – Lessons Learned
Recommendations
Acknowledgements
The Preceptor & Mentor Initiative unfolded
The Preceptor & Mentor Initiative’s major recommendations include:
The authors would like to thank Alice Chen, (PhD Candidate at UBC’s Department of Health Care & Epidemiology), for sharing her expertise and developing the Program Logic Model of Preceptor & Mentor Initiative and supporting frameworks. Our sincere thanks also go to the BC Academic Health Council (in particular Dr. John Gilbert and Kathy Copeman-Stewart) for their continuous support.
amidst the many system challenges with which healthcare managers are undoubt-
Continue to:
edly very familiar. Key challenges we
• value and pursue the goals of the Initiative;
experienced in rolling out this program included: lack of data on the current state of preceptoring and mentoring; different systems that are huge, complex and not well interconnected; tremendous changes
• update the Preceptor & Mentor web site as a provincial resource, emphasize on-line networking groups;
infrastructure that made it difficult to target leaders and communicate (for
Establish:
example, in 2002 the BC health system
• a provincial competency-based mentoring system for healthcare professionals and managers;
reorganized from over 50 to six health authorities); and inherent inequities in the various systems in terms of employment contracts, targeted funding and discipline-specific roles and boundaries. In 2001, the BC Ministry of Health Planning recognized the need for a
1. Grembowski D. The Practice of Health Program Evaluation. Thousand Oaks, CA: Sage Publications; 2001. 2. References of studies linking aspects of preceptoring and
• support preceptor/mentor education for all health disciplines throughout the province.
in terms of people, organizations and
References and Notes
mentoring with outcomes are available on the Preceptor & Mentor Initiative web page at: http://www.couth.bc.ca/managingprograms/evaluating/how to.htm 3. HSPnet (Health Services Placement Network of BC) is a province-wide web-enabled database for coordinating and improving student placements for all professions in BC. Visit the project’s web site at: http://www.hspbc.net
• a provincial database on preceptoring and mentoring (HSPnet3 may achieve this). "A workplace needs to have ‘mentoring’ as a norm." (Workshop Participant, Vancouver BC, January 2002).
coordinated approach to preceptoring and mentoring that bridged the health and education sectors. The resulting model was a small infrastructure that compiled and developed supportive and knowledge-based
resources
while
coordinating and working with diverse communities to facilitate their development. Our experience with the Preceptor & Mentor Initiative and the results it yielded indicate that such a model can make sustainable changes at the local level.
Opcom Healthcare 1/4 page (repeat)
If we were to do this project again, with identical system challenges, we would: • develop the From Practitioner to Preceptor course earlier; • target and assist the less organized communities of BC earlier; and • streamline the web site content management process.
Healthcare Management Forum Gestion des soins de santé
31
Going Provincial: Preceptoring and Mentoring Revisited in the Health Sciences Part II – The Evaluation Carol Mitchell, MSN, was the Project Manager of the Preceptor & Mentor Initiative for Health Sciences in BC, Vancouver.
Brenda SawatzkyGirling, MHA, is a healthcare policy and management consultant in Vancouver.
by Carol Mitchell and Brenda Sawatzky-Girling
Abstract Part I of this two part series, which was published in the Spring 2003 issue of Healthcare Management FORUM, described the Preceptor & Mentor Initiative for Health Sciences in BC, a provincial strategy to facilitate preceptoring and mentoring. Activities included creation of an educational web site and a UBC Continuing Studies preceptor course, and provincial workshops on preceptoring and mentoring. The purpose of this second article is to outline the philosophy, framework, results and interpretation of the Preceptor & Mentor Initiative’s evaluation component.
T
he Preceptor & Mentor Initiative for Health Sciences in BC Project Team developed an evaluation plan during the initial program planning phase in 2001. To bring objectivity to this critical function, the evaluation framework was designed with input from, but independent of, the Project Team. We needed representative evidence from health and education professionals in BC to assess whether the following objectives of the Initiative had been met: • to increase recruitment and retention of health sciences professionals; • to build capacity to educate these same professionals; • to contribute to a culture of lifelong learning; and • to address the needs of many diverse groups.
In addition, the evaluation was designed to elicit recommendations on future preceptoring and mentoring structures and activities in the province. The evaluation approach was based on the quality movement [structureprocess-outcome] and aligned with a logic model. When translated into recommendations, these findings become a decision-making tool for policy makers to determine the Initiative’s value. We expect that such a program’s future depends not only on its performance but also on whether the program goals remain desirable given competing priorities.1
Evaluation Framework/Process Our evaluation framework, a Program Logic Model of Preceptor & Mentor Initiative, was designed by a doctoral student at UBC’s Centre for Health Services and Policy Research (see figure 1). Although the long-term outcomes are complex and difficult to measure, the Logic Model was designed to provide clear and explicit criteria for success as indicated by the short-term outcomes. The literature indicates the short-term outcomes identified
28
Healthcare Management Forum Gestion des soins de santé
Figure 1: PROGRAM LOGIC MODEL OF PRECEPTOR & MENTOR INITIATIVE Component
Process
Output
Needs assessment
Baseline assessment survey
Outline of current status of programs in the province
Consultation with stakeholders
Outline of needs of stakeholders in developing programs
Baseline assessment survey; collecting existing resources
Outline of theoretical concepts Outline of best practices
Compilation and development of guidelines and resources
Dissemination of information
Networking of users
Outcome Short-term
Contribution to increased recruitment and retention of nurses and health sciences professionals Broader implementation of preceptor and mentor programs
Guidelines for managing, planning, implementing and evaluating programs that are sensitive to diverse needs (incl. reward and recognition processes and estimated costs)
Literature review
Consultation with stakeholders
Clearinghouse of existing tools and resources
Web site
Web site and users
Workshops to showcase the initiative
Workshops and participants
Provincial and national conferences
Conferences and participants
Web site Interactive discussion forum
Outcome Long-term
Contribution to increased capacity for student placement
More standard implementation of best practices
Web site and users Contribution to culture of continuing learning
Sustaining initiative
Developing business plan and applying for funding
Funding application
Secure funds
Figure 2: Knowledge Management in Action (broader implementation of preceptor and mentor programs, and more standard implementation of best practices) correlate with the Initiative goals.2
Provincial Infrastructure & Coordination
1 Surveys – Needs Assessment, Follow-up New ge wled Kn o
Compilation & Development of Guidelines, Resources
2 Preceptor & Mentor Initiative Web site 3 Resource Team Workshops rm rt Te Sho omes Outc
rm g Te Lon omes Outc
4 From Practitioner to Preceptor Course
Diss em Kno inating wled ge
Conferences Journals
Broader implementation of preceptor & mentor programs More standard implementation of best practices
Research
Support preceptors and mentors in order to: Increase recruitment of health sciences professionals Increase capacity to educate health sciences professionals Increase retention of health sciences professionals Foster a culture of continuous learning Address the needs of diverse groups
Each component in the Logic Model contributed to a Framework for Evaluation Implementation that identified research questions, performance indicators, success level or target dates, and how data would be collected. With this information, we could make process-output linkages. We anticipated early on that data collection would be a challenge because this type of database does not exist provincially, within regions or organizations and only minimally at the department level. The sources of data we used are described below and are illustrated in figure 2.
Healthcare Management Forum Gestion des soins de santé
29
❶ Surveys – We sent an Internet-based baseline survey to BC’s health and education stakeholders in September 2001. A follow-up survey was conducted in February 2003 to compare the two points in time.
Figure 3: Pre & Post: State of Preceptor and Mentor Program Development in BC
❷ Stakeholder Consultations – Our Resource Team of preceptor/mentor experts and end-users grew to include 221 people from around BC. We held 15 workshops (12 of these outside the Vancouver area) to help participants identify what resources they already had and to gather feedback as to whether our activities and plans were meeting their needs. We used this information to continually monitor and refine future workshops and other program activities and outputs. ❸ Web site – The evidence-based Preceptor & Mentor Initiative web site, launched in January 2002, was tracked in terms of utilization and feedback. “We need a ‘one minute preceptor course’ - we have little time for courses!” (Workshop Participant, Prince George BC, February 2002). ❹ Preceptor Course – We developed a preceptor course in a workshop/on-line learning format with UBC Continuing Studies. The From Practitioner to Preceptor course, launched in January 2003, was developed from our web site and the Children’s & Women’s Health Centre of BC preceptor education program. The C&W Program had undergone an extensive evaluation in 2002. Participants provided feedback on the course itself and the Initiative web site.
Results Overall, the activities of the Preceptor & Mentor Initiative proved effective in achieving the desired outcomes. The following describes how we know various Initiative activities directly achieved broader implementation of preceptor and mentor programs in BC and more standard implementation of best practices:
30
❶ Surveys Twenty-three out of 66 (35%) follow-up surveys were completed and returned. Respondents were representative of both health and education sectors and diverse in terms of discipline. In comparing the two points in time between baseline and follow-up surveys (see figure 3), trends indicate that respondents were further along in terms of planning, implementing and evaluating preceptor and mentor programs, notably preceptor programs. ❷ Stakeholder Consultations Workshop evaluation forms and informal comments were overwhelmingly positive.The follow-up survey asked whether respondents had attended or arranged workshops in their areas (15/23 respondents replied yes). Of these 15, 100% found the workshops helpful in developing programs, establishing crucial networks and identifying resources in their areas. ❸ Web site Web site utilization rose steadily since its launch. Web site feedback was systematically collected and the web site revised based on this input:
Healthcare Management Forum Gestion des soins de santé
• Over 1,400 visits were made to the web site with the most intense use in western Canada. A significant number of users visited the web site several times. • On-line feedback forms were minimal, but all were positive. • Eighteen out of 23 follow-up survey respondents had accessed the web site; 17/18 of this group found the web site to be useful to them in implementing preceptor/mentor programs and best practices. • Users who had web site demonstrations in workshops or who used the web site in the preceptor course provided positive comments on how comprehensive and useful the web site was, and that it met their needs. ❹ Preceptor Course This introductory course directly achieves the outcomes of broader implementation of preceptor programs and more standard implementation of best practices. There is demand for the course from different regions of BC. Feedback from the students in the initial course offering regarding quality and usefulness was consistently positive.
Interpretation – Lessons Learned
Recommendations
Acknowledgements
The Preceptor & Mentor Initiative unfolded
The Preceptor & Mentor Initiative’s major recommendations include:
The authors would like to thank Alice Chen, (PhD Candidate at UBC’s Department of Health Care & Epidemiology), for sharing her expertise and developing the Program Logic Model of Preceptor & Mentor Initiative and supporting frameworks. Our sincere thanks also go to the BC Academic Health Council (in particular Dr. John Gilbert and Kathy Copeman-Stewart) for their continuous support.
amidst the many system challenges with which healthcare managers are undoubt-
Continue to:
edly very familiar. Key challenges we
• value and pursue the goals of the Initiative;
experienced in rolling out this program included: lack of data on the current state of preceptoring and mentoring; different systems that are huge, complex and not well interconnected; tremendous changes
• update the Preceptor & Mentor web site as a provincial resource, emphasize on-line networking groups;
infrastructure that made it difficult to target leaders and communicate (for
Establish:
example, in 2002 the BC health system
• a provincial competency-based mentoring system for healthcare professionals and managers;
reorganized from over 50 to six health authorities); and inherent inequities in the various systems in terms of employment contracts, targeted funding and discipline-specific roles and boundaries. In 2001, the BC Ministry of Health Planning recognized the need for a
1. Grembowski D. The Practice of Health Program Evaluation. Thousand Oaks, CA: Sage Publications; 2001. 2. References of studies linking aspects of preceptoring and
• support preceptor/mentor education for all health disciplines throughout the province.
in terms of people, organizations and
References and Notes
mentoring with outcomes are available on the Preceptor & Mentor Initiative web page at: http://www.couth.bc.ca/managingprograms/evaluating/how to.htm 3. HSPnet (Health Services Placement Network of BC) is a province-wide web-enabled database for coordinating and improving student placements for all professions in BC. Visit the project’s web site at: http://www.hspbc.net
• a provincial database on preceptoring and mentoring (HSPnet3 may achieve this). "A workplace needs to have ‘mentoring’ as a norm." (Workshop Participant, Vancouver BC, January 2002).
coordinated approach to preceptoring and mentoring that bridged the health and education sectors. The resulting model was a small infrastructure that compiled and developed supportive and knowledge-based
resources
while
coordinating and working with diverse communities to facilitate their development. Our experience with the Preceptor & Mentor Initiative and the results it yielded indicate that such a model can make sustainable changes at the local level.
Opcom Healthcare 1/4 page (repeat)
If we were to do this project again, with identical system challenges, we would: • develop the From Practitioner to Preceptor course earlier; • target and assist the less organized communities of BC earlier; and • streamline the web site content management process.
Healthcare Management Forum Gestion des soins de santé
31