- Email: [email protected]
Aligning Requirements of Training and Assessment in Radiation Treatment Planning in the Era of Competency-Based Medical Education
Journal Pre-proof Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education Nikitha Moideen, MD, Catherine de Metz, MD FRCPC, Maria Kalyvas, MD FRCPC, Eleftherios Soleas, PhD(c), Rylan Egan, PhD, Nancy Dalgarno, PhD PII:
S0360-3016(19)33873-8
DOI:
https://doi.org/10.1016/j.ijrobp.2019.10.005
Reference:
ROB 25975
To appear in:
International Journal of Radiation Oncology • Biology • Physics
Received Date: 17 May 2019 Revised Date:
30 August 2019
Accepted Date: 7 October 2019
Please cite this article as: Moideen N, de Metz C, Kalyvas M, Soleas E, Egan R, Dalgarno N, Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education, International Journal of Radiation Oncology • Biology • Physics (2019), doi: https:// doi.org/10.1016/j.ijrobp.2019.10.005. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Elsevier Inc. All rights reserved.
Title Page Title: Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education Running Title: Treatment planning education in the era of CBME
Authors: Nikitha Moideen, MD * - Department of Oncology Queen’s University, Kingston Health Sciences Centre Cancer Centre of Southeastern Ontario at Kingston Health Sciences Centre 25 King Street West Kingston, ON K7L 5P9 Tel: (613) 539-4250 Email: [email protected] Catherine de Metz, MD FRCPC - Department of Oncology Queen’s University, Kingston Health Sciences Centre Maria Kalyvas, MD FRCPC - Department of Oncology Queen’s University, Kingston Health Sciences Centre Eleftherios Soleas, PhD(c) - Queen’s University Faculty of Education, Educational Consultant, Office of Professional Development and Educational Scholarship Rylan Egan, PhD - Queen's University Faculty of Health Sciences, Office of Health Sciences Education Nancy Dalgarno, PhD - Queen's University Faculty of Health Sciences, Director of Educational Scholarship in the Office of Professional Development and Educational Scholarship
*Corresponding author There are no conflicts of interest to disclose. This project was funded by the Queen’s University Department of Oncology Education Research Award.
1 Title: Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education
Running Title: Treatment planning education in the era of CBME
2 Abstract Purpose Radiation treatment planning (RTP) is a unique skill that requires interdisciplinary collaboration among radiation oncologists (ROs), dosimetrists, and medical physicists (MP) to train and assess residents. With the adoption of competency-based medical education (CBME) in Canada, it is essential residency program curricula focuses on developing competencies in RTP to facilitate entrustment. Our study investigates how radiation oncology team members’ perspectives on RTP education align with requirements of the CBME approach, and its implications for improving residency training.
Methods This qualitative research study took place in the department of oncology at one mid-size academic institution. Through convenience sampling, focus groups were conducted with ROs (n=11), dosimetrists (n=7), MPs (n=7), and residents (n=7). Thematic design was adopted to analyze the transcripts through open coding resulting in three overarching themes.
Results The results identified existing strengths and weaknesses of the residency program, and future opportunities to redesign the curriculum and assessment process within a CBME model. Three overarching themes emerged from the analysis: (1) the strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for
3 change. Stakeholders were optimistic CBME will help enrich resident learning with the increased frequency and quality of competency-based assessments. Participants suggested building a library of cases and developing computer based learning resources so as to provide a safe environment to develop skills in contouring, dosimetry, and plan evaluation, in accordance with CBME training.
Conclusion This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies including case libraries, computer-based learning and quality assessments were highlighted in designing an innovative RTP planning curriculum.
4 Introduction Currently, postgraduate medical education is grounded on a time-based curriculum where training occurs through exposure to specific concepts within a fixed time frame.1 In an effort to improve residency training across all Canadian specialty programs, the Royal College of Physicians and Surgeons of Canada (RCPSC) developed the competency by design framework (CBD) based on a competency-based medical education (CBME) model.2,3 In this framework, residents are evaluated on their ability to perform predetermined skill-sets through timely and effective feedback and assessments, primarily through direct observation.2,4,5 Learners progress through training by demonstrating achievement of these skills and knowledge rather than through time spent in training.
Arguments against CBME implementation include the concept of reductionism, which detracts from the art of medicine. CBME detractors argue learning and assessment processes within CBME is diminished to a check-list approach.6,7 Holmboe et al. suggests this is an unfounded fear and competency-based medicine encompasses a more holistic approach to medical education.7 They argue that breaking down concepts into component parts drive meaningful learning by facilitating more useful feedback; something that is lacking in a time-based educational model. Evidence suggests under traditional feedback frameworks, assessors often promote trainees who would benefit from additional training and give higher scores on evaluations than is deserved.8 Reasons for failure to report poor performance include inadequate documentation of prior performance, lack of direct observation of trainees, as well as fear of retribution through
5 poor faculty evaluations.8,9 This negates the developmental learning approach supported by CBME.
A recent publication by Best e al. on the readiness to practice of RO residents demonstrated that radiation planning workflow which is inclusive of contour review, plan evaluation and quality assurance practices such as on-board imaging review and troubleshooting of treatment unit issues such as patient set up are deficiencies not only for RO residents, but also for newly transitioned independent practicioners.10 This clearly highlights gaps in the current treatment planning education in Canada. The authors identified graded responsibility, simulation, formal/informal teaching and mentorship as potential solutions.10
Our study investigates inter-professional radiation oncology (RO) team members’ perspectives on strategies to improve upon the current radiation treatment planning (RTP) curriculum using a CBME approach. The study institution was the first radiation oncology residency program in Canada to transition to a CBME curriculum in 2016. In July 2019, all Canadian radiation oncology programs have transitioned to a CBME curriculum and follow a shared national curriculum. We recognize there is an ongoing movement to improve and standardize residency education in a multitude of domains with utilization of a CBME approach. The purpose of this study is to inform and provide foundational knowledge for competency-based RTP curriculum development in RO.
6 Methods Using an exploratory qualitative design and through convenience sampling, focus groups were conducted with radiation oncologists (ROs) (n=11, 12 invited), dosimetrists (n=7, 7 invited), medical physicists (n=7, 7 invited), and residents (n=7, 9 invited). The medical physicists focus group consisted of 6 medical physicists and 1 medical physics residents. The dosimetrist focus group consisted of 5 medical dosimetrists and 2 radiation therapy managers. To preserve confidentiality, address fidelity, mitigate bias, and collect candid perspectives on current and desired resident education efforts, the interviewer was not affiliated with the RO department. All participants provided verbal and written consent. Each one-hour focus group was audio-recorded and transcribed verbatim. Data were deidentified prior to coding. Participants offered their perspectives and when there were different perspectives on shared concerns all were considered equitably.
Data Analysis Thematic design was adopted to analyze transcripts through open coding in Atlas.ti 8.3.1114
Each transcript was independently coded by two researchers. Subsequently, they
compared coding to ensure shared meaning, and the codes were revised to ensure intercoder reliability (measure of the extent to which different coders agreed; Kappa = 0.823; very good).15,16 Through consensus, themes and sub-themes emerged.17
Points of view brought up by different stakeholders were differentiated. When a specific group (i.e. medical physicists, dosimetrist, attending physician, or residents) had views that were not mutually shared with the rest of the group this distinction was highlighted using qualitative language.
7
Results Three overarching themes emerged from the analysis: (1) strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for change (Table 1).
Theme 1: Strengths of Experiential RTP in the CBME environment The first theme includes two sub-themes: 1) competency-based RTP education has potential to enrich student learning, and 2) increased engagement acts as a catalyst for more useful and frequent feedback.
Enriched student learning Stakeholders felt CBME provided required checks and balances to monitor resident development in RTP. All stakeholders agreed this method of training would identify struggling residents in early phases of training, allowing faculty to proactively readjust learning trajectories based on emerging difficulties.
Useful and frequent feedback Attending physicians acknowledged that feedback will be more effectively documented in CBME, and challenges will be more objectively identified, allowing residents to utilize this knowledge to modify learning. Residents acknowledged that although they receive quality feedback on contouring and clinical interactions, feedback on the remainder of the RTP evaluation process is lacking. All stakeholders emphasized the need for more
8 opportunities for feedback based on direct observations of the resident plan evaluation process.
Theme 2: Challenges of Experiential RTP in CBME The second theme is composed of three subthemes: 1) workload demands 2) clear expectations for competency at each CBME stage, and 3) the need for systemic culture change.
Workload demands All parties stressed time was a barrier to providing and receiving useful feedback. Attending physicians and dosimetrists spoke about competing priorities such as falling behind on tasks due to heavy workloads and pressure to prioritize patient care over resident teaching. Attending physicians reported completing RTP tasks independently after work hours as it was not achieved during the work day. Barriers to feedback included finding time to observe a direct clinical encounter and providing feedback during a busy clinic or procedure.
Clear CBME expectations All participants identified need for clear expectations for competency at each training stage. They believed meaningful change would occur if there were clearly defined competencies and if they were accessible to everyone involved in resident teaching. There were also concerns residents would complete training in a cursory fashion by achieving checkmarks while not understanding the nuances of day to day practice in RO.
9 Need for systemic cultural change All stakeholders were hopeful CBME would improve the current residency training experience. They believed CBME would help residents graduate without deficiencies which could negatively impact patient care. To ensure competency, participants agreed institutional changes would be required to foster a culture more accepting of giving and receiving feedback. This should include prioritizing timely feedback, increasing direct observation assessments, and providing constructive feedback.
Theme 3: Opportunities for Change This theme identified three sub-themes: 1) developing of a library of cases, 2) structured formative RTP assessments, and 3) innovative teaching and learning strategies.
Case library development All ROs, residents and dosimetrists suggested the need to develop a centralized library of cases. Ideally, this would include rare cases, difficult cases, and examples of classic cases of various disease sub-sites which residents could use to develop competency skills for RTP.
RTP assessments All stakeholders agreed that developing structured formative RTP assessments was crucial to the success of the CBME model of RTP education. Residents expressed desire for more frequent formative assessments in a non-exam setting. Residents reasoned a
10 low-stakes environment would allow them to make mistakes and learn in a non-punitive setting.
Innovative teaching and learning strategies Innovative teaching strategies were identified to improve current teaching practices. Strategies included teaching residents side-by-side comparisons of similar radiation plans and justifying reasons for selecting one plan over another. Residents also asked for computer-based clinical vignettes to help them contour and plan treatments based on expert demonstrations.
Discussion Qualitative analysis revealed RO stakeholders are committed to CBME and hopeful that it will improve training of the next generation of ROs. All participants acknowledged CBME will enrich resident learning through increased participation, increased engagement, and improved feedback
Our findings emphasize the need for innovative teaching strategies including the development of a case library and computer-based clinical vignettes to aid in teaching RTP. Adelman et al. reported on gaps in training RO residents treatment plan evaluation.18 To address this issue, they created a high-fidelity computer based simulation with diverse disease sub-sites which provides residents immediate feedback on their RTP skills and displays suggestions for acceptable plans. Additionally, residents requested computer-based clinical vignettes to observe an experienced physician model the nuances
11 of contouring and RTP. Computer-based learning utilizing multimedia resources is a viable option in medical education by optimizing learning delivery and increasing content accessibility.19 Prior literature has showed multimedia resources such as workshops offered through the E-contouring platform has improved contouring accuracy.20 This highlights the need for increased engagement and the need for investing in the development of computer-based learning resources.
RO residents emphasized the need for more RTP formative assessments in a safe, nonpunitive environment. Strengths of formative feedback include the ability to drive selfreflection and future learning.21 This is in contrast to summative assessments which emphasize accountability and are primarily high-stakes.21
ROs emphasized workload demand as a challenge to teaching residents. ROs and other multidisciplinary team members are often under immense pressure to complete tasks without delay and teaching residents adds another level of complexity to ensuring quality patient care. This calls for increased professional development to support and provide resources for resident educators to include designated time for teaching, curriculum development and feedback delivery.5
This exploratory qualitative study is one of the first to address the question of multidisciplinary team member perceptions towards CBME and how best to develop a CBME RTP curriculum. There was a high response rate with nearly all members of the
12 multidisciplinary education team participating in the research. The limitations of this study include that it is a small study and may not be broadly generalizable to medical education in programs outside of RO. However, in a RO context, we believe it is generalizable to training programs in Canada, the United States and internationally as treatment planning is a core competency of the discipline and this present study highlights an innovative and informative approach to RTP education. The small sample number in this present study is representative of the small size of RO residents nationwide in Canada. The RO training programs in Canada in recent years have admitted 21 residents per year nationwide.22 In 2018, there were only 108 residents in years 1to 5 of post-graduate training in RO in Canada.22 The RO training program at the study institution is representative of an average sized program in Canada at the time of this study. We have demonstrated a high level of participation in the focus groups by key informants involved in the residency training program with 100 percent of invited physicists and dosimetrist participating and only two residents and one attending not participating in the focus groups. We would argue that other institutions transitioning to CBME or following a collaborative approach to RTP education would likely face these same educational challenges and will benefit from the identified opportunities highlighted by this present study for improving teaching and learning in their trainee programs.
Conclusion This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies including case libraries and computer-based learning, and quality assessments were
13 highlighted in designing a robust RTP planning curriculum. Our next steps include a redesign of the residency curriculum based on findings of this study and the development of an online library of cases. We also plan to confirm these findings at other Canadian institutions as they too transition to a CBME paradigm in the coming years. We expect that the findings of our multi-institutional study would facilitate the evolution of these themes to be representative of the practices of radiation oncology in the aggregate.
References: 1.
Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77(5):361-367.
2.
Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631-637.
3.
Frank J, Snell L, Sherbino J. The draft CanMEDS 2015 physician competency framework–series IV. Ottawa: The Royal College of Physicians and Surgeons of Canada. 2014.
4.
Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943-954.
5.
Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90(4):411-413.
6.
Huddle TS, Heudebert GR. Taking apart the art: the risk of anatomizing clinical competence. Acad Med. 2007;82(6):536-541.
14 7.
Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. A call to action: The controversy of and rationale for competency-based medical education. Med Teach. 2017;39(6):574-581.
8.
McQueen SA, Petrisor B, Bhandari M, Fahim C, McKinnon V, Sonnadara RR. Examining the barriers to meaningful assessment and feedback in medical training. Am J Surg. 2016;211(2):464-475.
9.
Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80(10 Suppl):S84-87.
10.
Best LR, Sengupta A, Murphy RJL, et al. Transition to practice in radiation oncology: Mind the gap. Radiother Oncol. 2019;138:126-131.
11.
Atlas ti : Qualitative Data Analysis [computer program]. 2015.
12.
Friese S. Qualitative data analysis with ATLAS. ti. SAGE Publications Limited; 2019.
13.
Friese S, Soratto J, Pires D. Carrying out a computer-aided thematic content analysis with ATLAS. ti. 2018.
14.
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3(2):77-101.
15.
Basit TN. Conducting research in educational contexts. Bloomsbury Publishing; 2010.
16.
Patton MQ. Qualitative research & evaluation methods: Integrating Theory and practice. 4th ed: Sage Publications Inc; 2015.
15 17.
Giacomini MK, Cook DJ. Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? EvidenceBased Medicine Working Group. Jama. 2000;284(3):357-362.
18.
Adleman J, Winter J, Purdie T, McNiven A, Croke J. Creation and Implementation of High-Fidelity Computer-Based Simulation to Improve Resident Competency in Radiation Therapy Treatment Plan Evaluation. International Journal of Radiation Oncology• Biology• Physics. 2018;102(3):e383-e384.
19.
Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81(3):207-212.
20.
Mailhot Vega RB, Ishaq OF, Ahmed I, Rene L, Amendola BE, Hu KS. Novel Pilot Curriculum for International Education of Lymphoma Management Using E-Contouring. Journal of global oncology. 2018;4:1-9.
21.
Epstein RM. Assessment in medical education. New England journal of medicine. 2007;356(4):387-396.
22.
Loewen SK, Doll CM, Halperin R, et al. National Trends and Dynamic Responses in the Canadian Radiation Oncology Workforce From 1990 to 2018. International Journal of Radiation Oncology • Biology • Physics.
Table 1: Themes and Subthemes of Content Analysis Themes Strengths of Treatment
Sub-Themes •
Planning in CBME
Selected Quotations
Competency based
“I think making sure that residents meet
assessments enrich student
benchmarks at each level will ensure that
learning
you don’t get to a PGY5 level… [with] huge gaps in your knowledge.” -Medical Physicist
“I absolutely think getting more timely •
Challenges of Treatment
•
Increased engagement in the
and more direct feedback based on this
feedback process will act as a
particular stage of [residency] is actually
catalyst for more useful and
going to help them.” -Attending
frequent feedback.
Physician
Workload demands
“I need to be able to be patient and allow
Panning in CBME
them [residents] to push the buttons, to do the work, to make mistakes, to correct the mistakes, to take the feedback, [and] to grow—to learn and then not have it clinically impact my day to the point where I have completely fallen behind on all my timelines.” -Dosimetrist
“If it is documented what those •
Clear expectations for
competencies are. If the details are there,
competency at each training
then it would be very beneficial.”
stage
-Dosimetrist
“I don’t think that [CBME] is going to help if we continue with the current path •
Opportunities for
•
Change
Need for systemic cultural
unless we make some changes.”
change
-Resident Physician
Development of a library of
“If magically I could have a bank of all
cases
scenarios of various cancers including rare ones that they can practice in terms of treatment planning…contouring and the dosimetry and plan evaluation… I think we could magically have that kind of thing it would be amazing.”
-
Attending Physician
“We would like more low stress assessments of treatment planning •
Structured formative
evaluation.”
treatment planning
-Resident Physician
assessments
“It is quite common in the European training programs where they use audio •
Innovative teaching and visual resources. So, they have a recorded learning strategies to support treatment plan for contouring and the staff the development of quality themselves is narrating every step and treatment plans why he is making this judgment and why
he is including this volume. I think this is helpful and is really easy and we could refer to these in our free time at home when you can learn more about contouring and treatment planning.” -Resident Physician
S0360-3016(19)33873-8
DOI:
https://doi.org/10.1016/j.ijrobp.2019.10.005
Reference:
ROB 25975
To appear in:
International Journal of Radiation Oncology • Biology • Physics
Received Date: 17 May 2019 Revised Date:
30 August 2019
Accepted Date: 7 October 2019
Please cite this article as: Moideen N, de Metz C, Kalyvas M, Soleas E, Egan R, Dalgarno N, Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education, International Journal of Radiation Oncology • Biology • Physics (2019), doi: https:// doi.org/10.1016/j.ijrobp.2019.10.005. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Elsevier Inc. All rights reserved.
Title Page Title: Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education Running Title: Treatment planning education in the era of CBME
Authors: Nikitha Moideen, MD * - Department of Oncology Queen’s University, Kingston Health Sciences Centre Cancer Centre of Southeastern Ontario at Kingston Health Sciences Centre 25 King Street West Kingston, ON K7L 5P9 Tel: (613) 539-4250 Email: [email protected] Catherine de Metz, MD FRCPC - Department of Oncology Queen’s University, Kingston Health Sciences Centre Maria Kalyvas, MD FRCPC - Department of Oncology Queen’s University, Kingston Health Sciences Centre Eleftherios Soleas, PhD(c) - Queen’s University Faculty of Education, Educational Consultant, Office of Professional Development and Educational Scholarship Rylan Egan, PhD - Queen's University Faculty of Health Sciences, Office of Health Sciences Education Nancy Dalgarno, PhD - Queen's University Faculty of Health Sciences, Director of Educational Scholarship in the Office of Professional Development and Educational Scholarship
*Corresponding author There are no conflicts of interest to disclose. This project was funded by the Queen’s University Department of Oncology Education Research Award.
1 Title: Aligning requirements of training and assessment in radiation treatment planning in the era of competency-based medical education
Running Title: Treatment planning education in the era of CBME
2 Abstract Purpose Radiation treatment planning (RTP) is a unique skill that requires interdisciplinary collaboration among radiation oncologists (ROs), dosimetrists, and medical physicists (MP) to train and assess residents. With the adoption of competency-based medical education (CBME) in Canada, it is essential residency program curricula focuses on developing competencies in RTP to facilitate entrustment. Our study investigates how radiation oncology team members’ perspectives on RTP education align with requirements of the CBME approach, and its implications for improving residency training.
Methods This qualitative research study took place in the department of oncology at one mid-size academic institution. Through convenience sampling, focus groups were conducted with ROs (n=11), dosimetrists (n=7), MPs (n=7), and residents (n=7). Thematic design was adopted to analyze the transcripts through open coding resulting in three overarching themes.
Results The results identified existing strengths and weaknesses of the residency program, and future opportunities to redesign the curriculum and assessment process within a CBME model. Three overarching themes emerged from the analysis: (1) the strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for
3 change. Stakeholders were optimistic CBME will help enrich resident learning with the increased frequency and quality of competency-based assessments. Participants suggested building a library of cases and developing computer based learning resources so as to provide a safe environment to develop skills in contouring, dosimetry, and plan evaluation, in accordance with CBME training.
Conclusion This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies including case libraries, computer-based learning and quality assessments were highlighted in designing an innovative RTP planning curriculum.
4 Introduction Currently, postgraduate medical education is grounded on a time-based curriculum where training occurs through exposure to specific concepts within a fixed time frame.1 In an effort to improve residency training across all Canadian specialty programs, the Royal College of Physicians and Surgeons of Canada (RCPSC) developed the competency by design framework (CBD) based on a competency-based medical education (CBME) model.2,3 In this framework, residents are evaluated on their ability to perform predetermined skill-sets through timely and effective feedback and assessments, primarily through direct observation.2,4,5 Learners progress through training by demonstrating achievement of these skills and knowledge rather than through time spent in training.
Arguments against CBME implementation include the concept of reductionism, which detracts from the art of medicine. CBME detractors argue learning and assessment processes within CBME is diminished to a check-list approach.6,7 Holmboe et al. suggests this is an unfounded fear and competency-based medicine encompasses a more holistic approach to medical education.7 They argue that breaking down concepts into component parts drive meaningful learning by facilitating more useful feedback; something that is lacking in a time-based educational model. Evidence suggests under traditional feedback frameworks, assessors often promote trainees who would benefit from additional training and give higher scores on evaluations than is deserved.8 Reasons for failure to report poor performance include inadequate documentation of prior performance, lack of direct observation of trainees, as well as fear of retribution through
5 poor faculty evaluations.8,9 This negates the developmental learning approach supported by CBME.
A recent publication by Best e al. on the readiness to practice of RO residents demonstrated that radiation planning workflow which is inclusive of contour review, plan evaluation and quality assurance practices such as on-board imaging review and troubleshooting of treatment unit issues such as patient set up are deficiencies not only for RO residents, but also for newly transitioned independent practicioners.10 This clearly highlights gaps in the current treatment planning education in Canada. The authors identified graded responsibility, simulation, formal/informal teaching and mentorship as potential solutions.10
Our study investigates inter-professional radiation oncology (RO) team members’ perspectives on strategies to improve upon the current radiation treatment planning (RTP) curriculum using a CBME approach. The study institution was the first radiation oncology residency program in Canada to transition to a CBME curriculum in 2016. In July 2019, all Canadian radiation oncology programs have transitioned to a CBME curriculum and follow a shared national curriculum. We recognize there is an ongoing movement to improve and standardize residency education in a multitude of domains with utilization of a CBME approach. The purpose of this study is to inform and provide foundational knowledge for competency-based RTP curriculum development in RO.
6 Methods Using an exploratory qualitative design and through convenience sampling, focus groups were conducted with radiation oncologists (ROs) (n=11, 12 invited), dosimetrists (n=7, 7 invited), medical physicists (n=7, 7 invited), and residents (n=7, 9 invited). The medical physicists focus group consisted of 6 medical physicists and 1 medical physics residents. The dosimetrist focus group consisted of 5 medical dosimetrists and 2 radiation therapy managers. To preserve confidentiality, address fidelity, mitigate bias, and collect candid perspectives on current and desired resident education efforts, the interviewer was not affiliated with the RO department. All participants provided verbal and written consent. Each one-hour focus group was audio-recorded and transcribed verbatim. Data were deidentified prior to coding. Participants offered their perspectives and when there were different perspectives on shared concerns all were considered equitably.
Data Analysis Thematic design was adopted to analyze transcripts through open coding in Atlas.ti 8.3.1114
Each transcript was independently coded by two researchers. Subsequently, they
compared coding to ensure shared meaning, and the codes were revised to ensure intercoder reliability (measure of the extent to which different coders agreed; Kappa = 0.823; very good).15,16 Through consensus, themes and sub-themes emerged.17
Points of view brought up by different stakeholders were differentiated. When a specific group (i.e. medical physicists, dosimetrist, attending physician, or residents) had views that were not mutually shared with the rest of the group this distinction was highlighted using qualitative language.
7
Results Three overarching themes emerged from the analysis: (1) strengths of RTP in the CBME environment; (2) challenges of RTP in CBME; and (3) opportunities for change (Table 1).
Theme 1: Strengths of Experiential RTP in the CBME environment The first theme includes two sub-themes: 1) competency-based RTP education has potential to enrich student learning, and 2) increased engagement acts as a catalyst for more useful and frequent feedback.
Enriched student learning Stakeholders felt CBME provided required checks and balances to monitor resident development in RTP. All stakeholders agreed this method of training would identify struggling residents in early phases of training, allowing faculty to proactively readjust learning trajectories based on emerging difficulties.
Useful and frequent feedback Attending physicians acknowledged that feedback will be more effectively documented in CBME, and challenges will be more objectively identified, allowing residents to utilize this knowledge to modify learning. Residents acknowledged that although they receive quality feedback on contouring and clinical interactions, feedback on the remainder of the RTP evaluation process is lacking. All stakeholders emphasized the need for more
8 opportunities for feedback based on direct observations of the resident plan evaluation process.
Theme 2: Challenges of Experiential RTP in CBME The second theme is composed of three subthemes: 1) workload demands 2) clear expectations for competency at each CBME stage, and 3) the need for systemic culture change.
Workload demands All parties stressed time was a barrier to providing and receiving useful feedback. Attending physicians and dosimetrists spoke about competing priorities such as falling behind on tasks due to heavy workloads and pressure to prioritize patient care over resident teaching. Attending physicians reported completing RTP tasks independently after work hours as it was not achieved during the work day. Barriers to feedback included finding time to observe a direct clinical encounter and providing feedback during a busy clinic or procedure.
Clear CBME expectations All participants identified need for clear expectations for competency at each training stage. They believed meaningful change would occur if there were clearly defined competencies and if they were accessible to everyone involved in resident teaching. There were also concerns residents would complete training in a cursory fashion by achieving checkmarks while not understanding the nuances of day to day practice in RO.
9 Need for systemic cultural change All stakeholders were hopeful CBME would improve the current residency training experience. They believed CBME would help residents graduate without deficiencies which could negatively impact patient care. To ensure competency, participants agreed institutional changes would be required to foster a culture more accepting of giving and receiving feedback. This should include prioritizing timely feedback, increasing direct observation assessments, and providing constructive feedback.
Theme 3: Opportunities for Change This theme identified three sub-themes: 1) developing of a library of cases, 2) structured formative RTP assessments, and 3) innovative teaching and learning strategies.
Case library development All ROs, residents and dosimetrists suggested the need to develop a centralized library of cases. Ideally, this would include rare cases, difficult cases, and examples of classic cases of various disease sub-sites which residents could use to develop competency skills for RTP.
RTP assessments All stakeholders agreed that developing structured formative RTP assessments was crucial to the success of the CBME model of RTP education. Residents expressed desire for more frequent formative assessments in a non-exam setting. Residents reasoned a
10 low-stakes environment would allow them to make mistakes and learn in a non-punitive setting.
Innovative teaching and learning strategies Innovative teaching strategies were identified to improve current teaching practices. Strategies included teaching residents side-by-side comparisons of similar radiation plans and justifying reasons for selecting one plan over another. Residents also asked for computer-based clinical vignettes to help them contour and plan treatments based on expert demonstrations.
Discussion Qualitative analysis revealed RO stakeholders are committed to CBME and hopeful that it will improve training of the next generation of ROs. All participants acknowledged CBME will enrich resident learning through increased participation, increased engagement, and improved feedback
Our findings emphasize the need for innovative teaching strategies including the development of a case library and computer-based clinical vignettes to aid in teaching RTP. Adelman et al. reported on gaps in training RO residents treatment plan evaluation.18 To address this issue, they created a high-fidelity computer based simulation with diverse disease sub-sites which provides residents immediate feedback on their RTP skills and displays suggestions for acceptable plans. Additionally, residents requested computer-based clinical vignettes to observe an experienced physician model the nuances
11 of contouring and RTP. Computer-based learning utilizing multimedia resources is a viable option in medical education by optimizing learning delivery and increasing content accessibility.19 Prior literature has showed multimedia resources such as workshops offered through the E-contouring platform has improved contouring accuracy.20 This highlights the need for increased engagement and the need for investing in the development of computer-based learning resources.
RO residents emphasized the need for more RTP formative assessments in a safe, nonpunitive environment. Strengths of formative feedback include the ability to drive selfreflection and future learning.21 This is in contrast to summative assessments which emphasize accountability and are primarily high-stakes.21
ROs emphasized workload demand as a challenge to teaching residents. ROs and other multidisciplinary team members are often under immense pressure to complete tasks without delay and teaching residents adds another level of complexity to ensuring quality patient care. This calls for increased professional development to support and provide resources for resident educators to include designated time for teaching, curriculum development and feedback delivery.5
This exploratory qualitative study is one of the first to address the question of multidisciplinary team member perceptions towards CBME and how best to develop a CBME RTP curriculum. There was a high response rate with nearly all members of the
12 multidisciplinary education team participating in the research. The limitations of this study include that it is a small study and may not be broadly generalizable to medical education in programs outside of RO. However, in a RO context, we believe it is generalizable to training programs in Canada, the United States and internationally as treatment planning is a core competency of the discipline and this present study highlights an innovative and informative approach to RTP education. The small sample number in this present study is representative of the small size of RO residents nationwide in Canada. The RO training programs in Canada in recent years have admitted 21 residents per year nationwide.22 In 2018, there were only 108 residents in years 1to 5 of post-graduate training in RO in Canada.22 The RO training program at the study institution is representative of an average sized program in Canada at the time of this study. We have demonstrated a high level of participation in the focus groups by key informants involved in the residency training program with 100 percent of invited physicists and dosimetrist participating and only two residents and one attending not participating in the focus groups. We would argue that other institutions transitioning to CBME or following a collaborative approach to RTP education would likely face these same educational challenges and will benefit from the identified opportunities highlighted by this present study for improving teaching and learning in their trainee programs.
Conclusion This study identified future opportunities to redesign the RTP curriculum and assessment process within a CBME model. The need for innovative teaching and learning strategies including case libraries and computer-based learning, and quality assessments were
13 highlighted in designing a robust RTP planning curriculum. Our next steps include a redesign of the residency curriculum based on findings of this study and the development of an online library of cases. We also plan to confirm these findings at other Canadian institutions as they too transition to a CBME paradigm in the coming years. We expect that the findings of our multi-institutional study would facilitate the evolution of these themes to be representative of the practices of radiation oncology in the aggregate.
References: 1.
Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77(5):361-367.
2.
Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631-637.
3.
Frank J, Snell L, Sherbino J. The draft CanMEDS 2015 physician competency framework–series IV. Ottawa: The Royal College of Physicians and Surgeons of Canada. 2014.
4.
Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943-954.
5.
Holmboe ES. Realizing the promise of competency-based medical education. Acad Med. 2015;90(4):411-413.
6.
Huddle TS, Heudebert GR. Taking apart the art: the risk of anatomizing clinical competence. Acad Med. 2007;82(6):536-541.
14 7.
Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. A call to action: The controversy of and rationale for competency-based medical education. Med Teach. 2017;39(6):574-581.
8.
McQueen SA, Petrisor B, Bhandari M, Fahim C, McKinnon V, Sonnadara RR. Examining the barriers to meaningful assessment and feedback in medical training. Am J Surg. 2016;211(2):464-475.
9.
Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80(10 Suppl):S84-87.
10.
Best LR, Sengupta A, Murphy RJL, et al. Transition to practice in radiation oncology: Mind the gap. Radiother Oncol. 2019;138:126-131.
11.
Atlas ti : Qualitative Data Analysis [computer program]. 2015.
12.
Friese S. Qualitative data analysis with ATLAS. ti. SAGE Publications Limited; 2019.
13.
Friese S, Soratto J, Pires D. Carrying out a computer-aided thematic content analysis with ATLAS. ti. 2018.
14.
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3(2):77-101.
15.
Basit TN. Conducting research in educational contexts. Bloomsbury Publishing; 2010.
16.
Patton MQ. Qualitative research & evaluation methods: Integrating Theory and practice. 4th ed: Sage Publications Inc; 2015.
15 17.
Giacomini MK, Cook DJ. Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? EvidenceBased Medicine Working Group. Jama. 2000;284(3):357-362.
18.
Adleman J, Winter J, Purdie T, McNiven A, Croke J. Creation and Implementation of High-Fidelity Computer-Based Simulation to Improve Resident Competency in Radiation Therapy Treatment Plan Evaluation. International Journal of Radiation Oncology• Biology• Physics. 2018;102(3):e383-e384.
19.
Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81(3):207-212.
20.
Mailhot Vega RB, Ishaq OF, Ahmed I, Rene L, Amendola BE, Hu KS. Novel Pilot Curriculum for International Education of Lymphoma Management Using E-Contouring. Journal of global oncology. 2018;4:1-9.
21.
Epstein RM. Assessment in medical education. New England journal of medicine. 2007;356(4):387-396.
22.
Loewen SK, Doll CM, Halperin R, et al. National Trends and Dynamic Responses in the Canadian Radiation Oncology Workforce From 1990 to 2018. International Journal of Radiation Oncology • Biology • Physics.
Table 1: Themes and Subthemes of Content Analysis Themes Strengths of Treatment
Sub-Themes •
Planning in CBME
Selected Quotations
Competency based
“I think making sure that residents meet
assessments enrich student
benchmarks at each level will ensure that
learning
you don’t get to a PGY5 level… [with] huge gaps in your knowledge.” -Medical Physicist
“I absolutely think getting more timely •
Challenges of Treatment
•
Increased engagement in the
and more direct feedback based on this
feedback process will act as a
particular stage of [residency] is actually
catalyst for more useful and
going to help them.” -Attending
frequent feedback.
Physician
Workload demands
“I need to be able to be patient and allow
Panning in CBME
them [residents] to push the buttons, to do the work, to make mistakes, to correct the mistakes, to take the feedback, [and] to grow—to learn and then not have it clinically impact my day to the point where I have completely fallen behind on all my timelines.” -Dosimetrist
“If it is documented what those •
Clear expectations for
competencies are. If the details are there,
competency at each training
then it would be very beneficial.”
stage
-Dosimetrist
“I don’t think that [CBME] is going to help if we continue with the current path •
Opportunities for
•
Change
Need for systemic cultural
unless we make some changes.”
change
-Resident Physician
Development of a library of
“If magically I could have a bank of all
cases
scenarios of various cancers including rare ones that they can practice in terms of treatment planning…contouring and the dosimetry and plan evaluation… I think we could magically have that kind of thing it would be amazing.”
-
Attending Physician
“We would like more low stress assessments of treatment planning •
Structured formative
evaluation.”
treatment planning
-Resident Physician
assessments
“It is quite common in the European training programs where they use audio •
Innovative teaching and visual resources. So, they have a recorded learning strategies to support treatment plan for contouring and the staff the development of quality themselves is narrating every step and treatment plans why he is making this judgment and why
he is including this volume. I think this is helpful and is really easy and we could refer to these in our free time at home when you can learn more about contouring and treatment planning.” -Resident Physician