- Email: [email protected]
Evaluating Curricular Modules in the Care of Children With Medical Complexity: A Mixed-Methods Randomized Controlled Trial
Journal Pre-proof
Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial Kathleen Huth MD MMSc , Tobey Audcent MD , Sara Long-Gagne MD MSc , Anne Marie Sbrocchi MD , Natalie Weiser MA , Doug Miller , Danielle Arje , Derek Stephens MSc , Nathalie Major MD , Kheirie Issa MD , Eyal Cohen MD MSc , Julia Orkin MD MSc PII: DOI: Reference:
S1876-2859(19)30385-7 https://doi.org/10.1016/j.acap.2019.09.002 ACAP 1409
To appear in:
Academic Pediatrics
Received date: Accepted date:
3 May 2019 7 September 2019
Please cite this article as: Kathleen Huth MD MMSc , Tobey Audcent MD , Sara Long-Gagne MD MSc , Anne Marie Sbrocchi MD , Natalie Weiser MA , Doug Miller , Danielle Arje , Derek Stephens MSc , Nathalie Major MD , Kheirie Issa MD , Eyal Cohen MD MSc , Julia Orkin MD MSc , Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial, Academic Pediatrics (2019), doi: https://doi.org/10.1016/j.acap.2019.09.002
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier Inc. on behalf of Academic Pediatric Association
Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial
Kathleen Huth MD MMSc1, Tobey Audcent MD2, Sara Long-Gagne MD MSc3, Anne Marie Sbrocchi MD3, Natalie Weiser MA4, Doug Miller5, Danielle Arje4, Derek Stephens MSc4, Nathalie Major MD2, Kheirie Issa MD2, Eyal Cohen MD MSc4,5, Julia Orkin MD MSc4,5.
Boston Children’s Hospital, Boston, MA; 2Children’s Hospital of Eastern Ontario, University of Ottawa, Ottawa, ON; 3Montreal Children’s Hospital, McGill University, Montreal, QC; 4 Paediatric Medicine, The Hospital for Sick Children, Toronto, ON; 5 Department of Paediatrics, University of Toronto, Toronto, ON. 1
Corresponding author Kathleen Huth Boston Children’s Hospital 300 Longwood Ave, Boston MA, 02115 Tel: 617-355-6162 Email: [email protected]
Keywords Complex care, residency education, postgraduate medical education, pediatrics, self-efficacy.
Running title Mixed-methods evaluation of a complex care curriculum
Funding sources This work was supported by the Norman Saunders Complex Care Initiative and the Paediatric Consultant’s Education Development and Innovation Grants, The Hospital for Sick Children, Toronto Ontario, Canada.
Conflicts of interest The authors have no conflicts of interest to disclose.
What’s New We performed a randomized control trial of a standardized complex care curriculum for pediatric residents using quantitative and qualitative methods. Our findings provide insight into effective educational strategies for increasing resident self-efficacy in the care of children with medical complexity.
ABSTRACT
Objective: Children with medical complexity (CMC) are a growing population, yet training in complex care varies across pediatric residency programs. The purpose of this study was 1) to evaluate the effectiveness of a curriculum for pediatric residents in improving performance in a simulated clinical scenario; and 2) to explore residents’ perceived self-efficacy in caring for CMC.
Methods: A randomized controlled trial was conducted supplemented by qualitative inquiry. Pediatric residents from two residency programs were randomly assigned to participate in interactive modules on: (a) clinical assessment, care planning and technological dependency or (b) non-complex care topics. The primary outcome was mean score on an Observed Structured Clinical Examination (OSCE) of tracheostomy care. Semi-structured interviews were conducted post-intervention and analyzed using qualitative content analysis.
Results: Ninety-four eligible residents were randomized. Residents who attended all modules and the OSCE and consented to participate [intervention (n=20) and control (n=24)] were included in the final analysis. At baseline, few (9%) reported being comfortable caring for CMC. There was no significant difference in mean OSCE score between intervention and control groups (39.0±1.1 vs. 38.0±1.0, p=0.48). Qualitative analysis revealed three emerging themes related to resident self-efficacy: building a system of care, navigating uncertainty and professional identity formation.
2
Conclusion: A standardized complex care curriculum delivered in a classroom setting did not lead to improved performance in an OSCE station despite increased resident-reported self-efficacy in approaching care for CMC. These findings highlight the need for multidimensional educational interventions and assessments in complex care.
INTRODUCTION Children with medical complexity (CMC) are characterized by chronic medical conditions and functional limitations with dependence on medical technology.1 Due to their unique and specialized care needs, CMC are at increased risk of medical error and care fragmentation, and therefore require care that is comprehensive, coordinated, and proactive.2 The prevalence of CMC is increasing due to improved medical technologies and advances in care.3 General pediatricians have taken on a larger role in the management of their chronic multisystem diseases and medical technology, both in hospital and in community practice.2
Currently, pediatric residency programs across North America offer variable clinical exposure and educational opportunities in complex care. As complex care is an evolving field of practice, structured training programs are lacking despite recognition of increasing need.4–6 Pediatric residents may have different levels of expertise and comfort in the care of CMC depending on their motivation to seek additional training in complex care at their academic center. Numerous publications have highlighted the need for formal training during pediatric residency to support the delivery of safe, evidence-based care for CMC.4,5,7 Equipping future pediatricians with clinical skills in complex care is essential for the delivery of comprehensive care for CMC in their community, which may help optimize outcomes including improved patient/family experience, quality and safety performance measures and reduced healthcare 3
utilization.8–11
Responding to the educational gaps in complex care, members of the study team (KH, SLG, AMS) designed a training experience for pediatric residents to develop competence in caring for CMC.12,13 Curriculum development and evaluation was guided by self-efficacy as a theoretical framework, which indicates that an expectation of personal efficacy in a particular area determines what actions will be taken, what goals are made, how much effort is expended, and what outcomes are expected.14 We hypothesized that an educational experience focused on standard clinical approaches in complex care would impact residents’ clinical encounters with patients/families as well as enhance confidence in their ability to provide high-quality care for CMC as future pediatricians.
The overarching study aim was to identify whether a standardized complex care curriculum improves pediatric residents’ clinical performance and self-efficacy. Specific study objectives were: 1. To determine the comparative effectiveness of participation in a standardized curriculum on performance on an objective structured clinical exam (OSCE) station of tracheostomy care, relative to residents receiving an educational session unrelated to complex care, evaluated by total mean score on an objective structured clinical exam (OSCE) of tracheostomy care. 2. To explore changes in residents' self-efficacy in caring for CMC following participation in the curriculum.
4
METHODS Study Participants The study was conducted at The Hospital for Sick Children and the Children’s Hospital of Eastern Ontario in Ontario, Canada. The pediatric residency programs consisted of 41 (Ottawa) and 72 (Toronto) learners across four postgraduate years and both had weekly academic half-day teaching. All pediatrics residents enrolled at the study sites were eligible for participation. Exclusion criteria were enrolment in subspecialty residency training.
Study Design We conducted a randomized controlled trial (RCT) of a standardized curriculum. This was a mixedmethods study with sequential explanatory design, wherein quantitative data collection was followed by qualitative study to better understand the findings from the first phase. Residents were randomized (allocation ratio 1:1, random block size stratified by site) to attend either a standardized complex care curriculum or regular academic sessions unrelated to complex care. One month after collection of outcome measurements, each group received the alternative training. A survey was administered at baseline including level of training, previous complex care experience, and comfort in caring for CMC.
Curriculum Development We used a systematic approach to curriculum development to meet training needs in complex care for the graduating pediatrician, with learning objectives aligned with the Canadian CanMEDS Physician Competency Framework.15,16 The intervention is a set of four curricular modules focused on key areas of competence in complex care: understanding the needs of CMC; clinical assessment; advocacy, leadership
5
and care planning; and care of the child with technology dependence; two of which are publicly available on MedEdPORTAL with associated formative assessments.12,13 The modules focused on skill development in approaching history-taking and physical examination for a child with medical complexity using a clinical case, developing proactive action plans with families, and routine care and management of common tracheostomy, feeding tube, and central line issues. The modules are organized into 60-minute sessions consisting of interactive case-based activities with accompanying facilitator guides. The curriculum was piloted with pediatrics residents in another Canadian program not participating in the study and revised based on participant and facilitator feedback.
Outcomes Our primary outcome was the score obtained on a scenario on history-taking for a child with tracheostomy in the Spring 2017 national pediatric in-training OSCE, two months after participation in the curriculum. The OSCE examiners received standard faculty development currently offered in each center for OSCE evaluation. They were provided with a standardized checklist for scoring data gathering, data interpretation and management of a child with blood-tinged secretions from his tracheostomy, in addition to a global assessment rating, for a total score of 50. OSCE examiners were blinded to group assignment.
A secondary outcome of the study was resident self-efficacy in the care of CMC, explored in semistructured interviews. A purposive sample of residents representing different levels of training and clinical exposure to CMC were approached by a research assistant (RA) one to three months after curriculum delivery, for voluntary participation in a semi-structured interview. Interviews were conducted over the telephone by a RA with experience in qualitative interviewing, and explored residents’ perceived capability in caring for CMC. The interview guide was pilot-tested and adapted iteratively. 6
Written informed consent was obtained from all participants. The study was approved by the institutional research ethics boards at both study sites. No financial incentives were provided for any part of the study. ClinicalTrials.gov Identifier: NCT03349541.
Data Analysis Quantitative Analysis Descriptive statistics were used to summarize demographics, survey responses and OSCE scores. Mean OSCE scores from the intervention and control groups were compared using a two-way ANOVA, with an interaction between group (intervention and control) and PGY group (junior and senior). Based on a prior study reporting mean and standard deviation for OSCE scores across all years of training,17 in order to detect a difference of one standard deviation in OSCE scores between intervention and control groups with 80% power at the 5% level of significance, a minimum sample size of 18 participants in each group was required. Aggregate OSCE scores from prior years at one of the study sites showed that the total mean score increased by 5-10% with each PGY, so this was thought to represent a clinically important difference.17,18
Qualitative Analysis Interviews were analyzed using conventional content analysis.19 Interviews were audio-recorded, transcribed, and de-identified. Transcripts were uploaded to qualitative data analysis software (NVivo 10). An immersive reading of each transcript was performed independently by three researchers (KH, 7
NW, JO) to identify themes emerging from the data. The researchers collaborated to discuss initial interpretations and develop a preliminary coding scheme. Transcripts were re-analyzed using this coding scheme to refine concepts. Data collection and analysis occurred iteratively, with sampling until the point of thematic saturation. A checklist was used to ensure we adhered to appropriate standards for qualitative research.20
RESULTS Quantitative Ninety-four residents across both residency programs were eligible for participation and randomized to the intervention or control groups (Figure 1). Forty-four residents participated in all curricular modules, attended the OSCE and consented to release their scores. Demographics of the study sample are described in Table 1. Residents were in PGY1 (40%), PGY2 (30%) and PGY3 (30%). Few residents (9%) reported being quite or extremely comfortable caring for CMC at baseline. Two percent had completed a complex care clinical rotation prior to the training session. Residents described themselves as being frequently/very frequently involved in inpatient care of CMC (66%); fewer residents described this level of involvement in outpatient care (9%).
Senior residents (PGY2-3) had significantly higher mean OSCE scores than junior residents (PGY1) (p=0.02). There was no significant difference in mean OSCE score between intervention and control groups (39.0±1.1 vs. 38.0±1.0, p=0.48). There was no significant interaction effect for PGY group (p=0.62).
8
Qualitative We conducted 13 interviews. Analysis revealed three emergent themes in relation to the care of CMC: 1) building a system of care, 2) navigating uncertainty and 3) professional identity formation (Table 2). Residents provided feedback on the educational experience, describing overall satisfaction with the modules, appreciation for dedicated complex care training, and increased comfort in approaching care for CMC. Residents also identified perceived learning gaps in complex care and next steps for curriculum development.
Building a system of care Residents recognized the importance of a collaborative approach to care for CMC within a larger system of care. Many described challenges with clinical decision-making in cases where they had limited understanding of the roles within the care team: I think the hardest part with complex care kids is the number of people involved in their care and figuring out who the appropriate person to contact is... you want to change something but you might not understand the context in which the original decisions were made for that child. So I think that’s the most difficult part from a medical standpoint for them. (#21)
Residents identified the need for various resources to inform care decisions for children with high healthcare utilization, including the interdisciplinary team, the shared care plan and the caregiver. Leveraging resources and recognizing the team as a support system was recognized as critical in approaching the care of CMC: Relying on staff, again relying on parents, like I said, is a huge one and then, like, relying on, like, the expertise of the complex care team. I think the more heads in the game… The better. It helps, like, alleviate some of that feeling of overwhelming. Kind of like sharing it. (#35).
9
Residents noted that the approach to complex care that they learned involved collaboration within a broader system of care, and this increased their comfort in caring for CMC.
Navigating uncertainty Residents described experiences navigating uncertainty as being central to their care of CMC. They identified knowledge gaps and limitations in the care of this specialized population compared to their relative knowledge caring for patients with simple or single clinical issues. Residents described the nature of complexity as involving multiple interdependent issues: …my concern is sometimes if I take the traditional way or the way I know how to correct a problem, is that, does that have contraindications to their other issues?... is that gonna then send, you know, something else way out of whack?... You know how to tackle all their issues individually, but then once you put them in the same kid… that makes it more difficult. (#52)
Residents described having less confidence in their clinical skills due to a lack of predictability in the care of CMC. …we can’t anticipate the way that they’re going to respond to an intervention, like I feel that we can with a lot of other children. And so that’s, that’s just hard because you’re kind of in this unknown realm where you might try something but the reaction you might get from the child is going to be not, completely not what you expected. (#14)
A number of residents described feeling more comfortable in the face of uncertainty, often through partnership with the child’s caregiver. They recognized the importance of acknowledging limitations and exploring this uncertainty with caregivers.
10
Professional identity formation Developing a professional identity as a future pediatrician caring for CMC was a predominant theme. Residents described a transition from seeing acute medical issues “with a narrow-minded focus” to “thinking more holistically”, recognizing the pediatrician’s role in working with caregivers toward a “bigger picture” as it was modelled by curriculum facilitators.
Some residents described feeling anxious when collaborating with caregivers in their child’s care, particularly if the conversation was related to a more chronic issue. One resident said they had felt “really uncomfortable or inadequate… you’re supposed to be this expert”. With further training and clinical exposure, residents re-envisioned their relationship with the caregiver as more of a partnership, rather than as a threat to their role: these parents both know their kids and usually know like why or how it’s related to their problem so it’s kind of like you have a partner working with you so I… find it really helpful. (#11)
Residents described developing their clinical skills through repeated experiences with CMC. Providing continuity of care and serving as a manager or coordinator of care, beyond episodic management of acute issues, was identified as being central to the pediatrician’s role. Residents were motivated to be able to oversee the care of CMC as a future pediatrician, with the goal being “I’m kind of like the go-to person for that family”.
Educational experience Residents provided feedback on aspects of the complex care curriculum that contributed to their comfort in caring for CMC, as well as ongoing training needs (Table 3). For almost all residents, this intervention 11
was their first formal training experience in complex care despite frequently providing clinical care for this population: it’s hard because we’re expected to care for these patients but we don’t actually get any, formal teaching in, in caring, or I guess, formal clinical experience in caring for them, except for now that they’re introduced this half-day, so that’s been really, really helpful. (#14)
Residents appreciated having an approach to clinical assessment of CMC as well as understanding more about life outside of the hospital: the [curriculum] provided us with kind of an approach on how to take an adequate history or what resources are out there, when you’re managing a complex care patient… what the home environment might look like… And it also provided context for, why families might be tired or annoyed… what their day-to-day might look like… definitely helped build my comfort in managing these patients... (#7)
Educational gaps identified included lack of exposure to home and community care of CMC, gaining deeper insight into interdisciplinary roles and need for regular clinical teaching in different care settings throughout training. Residents advocated for classroom-based teaching about care of CMC in the home and community to be accompanied by meaningful clinical experiences: I just wish I had more experience with seeing them out in the community… that’s a burden of the family of getting them ready in the morning, getting to the appointment, dealing with whatever happens in the appointment… That was something that was highlighted [in the curriculum] and so I think that needs to be reinforced by, like, us going and seeing. Like, if I ask, you know, Johnny to come in for an appointment, what is involved in getting him there and getting him home? (#18)
The complex care curriculum allowed residents to recognize educational gaps, and motivated them to seek immersive training experiences to increase comfort in the care of CMC.
12
DISCUSSION This multi-site mixed-methods RCT evaluated the impact of participation in a standardized curriculum on measures of pediatric residents’ knowledge, skills and attitudes in the care of CMC, providing insight into effective strategies and outstanding needs in complex care education. Participation in a standardized curriculum in a classroom setting did not lead to improved performance on an OSCE station on historytaking for a child with tracheostomy. However, following participation in the curriculum, residents described increased comfort in caring for CMC and recognizing their role in a care team. Our study responds to an unmet educational need in pediatrics, intending to prepare future pediatricians to provide high-quality care for a vulnerable pediatric population.
We found that pediatric residents described challenges, discomfort and educational gaps in caring for CMC at baseline, similar to those described by trainees and practicing pediatricians in the literature.4,7,21 Prior work in complex care education has focused on hospital-based care and expertise in the management of medical technology. Shah et al. evaluated web-based training in complex care and reported knowledge acquisition and improved verbal handoff of a simulated patient with complex care needs.22 Yet there is a lack of dedicated training on the lives of CMC outside of a hospital setting, and previous study describes residents’ lack of comfort in identifying appropriate supports and care plans for these patients and families.4 Pediatric residency training typically emphasizes time on inpatient wards and in acute care, and the time that is dedicated to ambulatory care is often focused on addressing a specific chief complaint. Many residency programs do not have a dedicated complex care clinical rotation, and existing rotations are often elective and primarily inpatient. Additional educational strategies are needed to foster competence in managing common and challenging issues in complex care, including simulation, direct observation of clinical encounters and feedback throughout residency training.
13
We found no significant change in mean OSCE score for residents who participated in the complex care curriculum. Our negative RCT highlights limitations in one-off training sessions without accompanying longitudinal clinical experiences and limitations in existing assessments of competence in complex care. Our finding may provide insight into the “dose” of training necessary to see an effect on clinical skills in complex care. The lack of an effect of the training session on the primary outcome may also be explained by discordance between training on a comprehensive clinical approach emphasizing team communication, and an assessment focused on knowledge-based expertise for a specific medical technology. We used an OSCE score as our primary outcome measure to evaluate knowledge transfer to a simulated clinical setting. While the OSCE does not directly translate to workplace performance, it does fall relatively high on Miller’s framework for assessing levels of competence,23 testing behavior which can show how individuals may apply knowledge in the future. Yet performance in a simulated setting may not predict future clinical practice. The validity of the OSCE as an assessment tool varies on the measures used, the context in which the examination is administered and the characteristics of the learners.24 There is no validity data in our particular context and population of interest, in particular with the ability of the OSCE to measure communication, collaboration and advocacy skills, all of which are important domains of physician performance.16 Evaluation of competence in complex care must systematically address these roles across multiple and distinctive clinical encounters and practice settings. Multidimensional assessments including direct observation and multisource feedback would be more appropriate methods of establishing competence in this diverse field. Next steps in complex care training and assessment should include defining entrustable professional activities for complex tasks such as care coordination and team management, which would provide clear markers of progressive skill development and translation of complex care competencies into clinical practice.25
Qualitative analysis enabled us to gain a deeper understanding of the impact of dedicated training on a holistic approach to the care of CMC, and to generate hypotheses about how new insights about complex 14
care may influence clinical practice. We identified emerging themes related to residents’ self-efficacy in the care of CMC. The provided training experience acknowledges the inherent uncertainty in clinical decision-making for CMC and models problem-solving for common challenges, fostering trainees’ ability to address these challenges. Bandura’s social cognitive theory described factors that determine motivation and behavior, and the cognitive processes that allow people to achieve their goals.14 Beliefs in one’s efficacy is at the center of what actions one is motivated to take.26 People with high self-efficacy view challenges as possible to overcome with perseverance and improved skills.27 In medical education, selfefficacy has been recognized as an enabler of complex decision-making and development of “practical wisdom” through experience.28 In our study, following the training residents described increased comfort in the approach to clinical assessment of CMC, management of chronic and interdependent issues, and motivation to collaborate with families and an interdisciplinary team. A pediatric resident’s sense of selfefficacy, or perceived capability, in caring for CMC, is important to foster for pediatricians to navigate challenges, partner with families and ultimately provide high-quality care for CMC.
We used a randomized controlled trial design with blinded outcome assessments to minimize risk of bias, supplemented by rigorous qualitative analysis, but nevertheless this study had a number of important limitations. Similar to challenges with other educational RCTs, we had a small sample size, high dropout rate, variability in how the intervention is administered and its timing, and multiple contextual factors impacting the outcome.29 Though 94 residents were eligible, only about half participated in all aspects of the educational intervention and the OSCE evaluation. A similar proportion of resident attrition has been reported in the literature,22 though our study had a smaller sample size, which may have contributed to the lack of our ability to detect the effect of training on clinical skills. There were no large differences in resident characteristics including PGY and gender between those that were included in the analysis and those that were not. Factors that affected participation in all curricular modules included vacation, illness, post-call, or off-site rotation, all of which should be considered when designing a curriculum evaluation. 15
We are not able to determine what changes in knowledge and skill in complex care may be expected over the course of residency training, and we were not powered to identify differences in knowledge gain by PGY, though previous studies have shown larger gains for more junior trainees.22 The OSCE focused on medical technology management, which did not allow us to adequately identify improved clinical skills related to other curriculum objectives. The OSCE may not be the optimal method to assess competency in complex care, and future research should consider using direct observation in clinical care of CMC as an assessment tool.
The study was performed in two large pediatric residency programs at academic institutions with established complex care programs in North America, which may limit the generalizability of these results to other settings and countries. Although both sites were tertiary-care referral centers with many patients defined as CMC, there may have been differences in the number or nature of CMC seen by each learner across sites. Our study did not evaluate long-term impact of training or direct impact on care of CMC. We anticipate that our findings will inform future study of longitudinal educational interventions in complex care incorporating direct observation and evaluation of clinical outcomes.
CONCLUSIONS Residents’ simulated performance in an OSCE station on tracheostomy care did not improve with dedicated complex care training, however residents reported increased self-efficacy in approaching care of CMC and developing their role within a care team. The complex care curriculum was recognized as a valuable educational experience that motivated trainees to seek opportunities to care for CMC. Our findings support the development of integrative and longitudinal clinical learning opportunities and multidimensional assessments of competence in complex care.
16
ACKNOWLEDGEMENTS This study was supported by the Norman Saunders Complex Care Initiative and the Paediatric Consultant’s Education Development and Innovation Grants at The Hospital for Sick Children. The authors would like to thank the Complex Care Special Interest Group of the Canadian Paediatric Society, the Canadian Pediatric Program Directors, and particularly Dr. Hilary Writer (University of Ottawa) and Dr. Adelle Atkinson (University of Toronto) for their support of this work.
References 1.
Cohen E, Kuo DZ, Agrawal R, et al. Children with medical complexity: An emerging population for clinical and research initiatives. Pediatrics. 2011;127(3):529-538. doi:10.1542/peds.20100910.
2.
Dewan T, Cohen E. Children with medical complexity in Canada. Paediatr Child Health. 2013;18(10):518-522.
3.
Tennant PW, Pearce MS, Bythell M, Rankin J. 20-year survival of children born with congenital anomalies: a population-based study. Lancet. 2010;375(9715):649-656. doi:10.1016/S01406736(09)61922-X.
4.
Nazarian BL, Glader L, Choueiri R, Shipman DL, Sadof M. Identifying what pediatric residents are taught about children and youth with special health care needs and the medical home. Pediatrics. 2010;126 Suppl(December):S183-9. doi:http://dx.doi.org/10.1542/peds.2010-1466O.
5.
Sbrocchi AM, Millar C, Henin C, Allard M, Patel H. Complex care core curriculum for pediatric post-graduate trainees: Results of a North American needs assessment. J Pediatr Rehabil Med. 2015;8(2):141-146. doi:10.3233/PRM-150327.
6.
Hewitt-Taylor J. Caring for children with complex needs: Staff education and training. J Child Heal Care. 2005;9(1):72-86. doi:10.1177/1367493505050506.
7.
Bogetz JF, Bogetz AL, Rassbach CE, Gabhart JM, Blankenburg RL. Caring for children with medical complexity: Challenges and educational opportunities identified by pediatric residents. Acad Pediatr. 2015;15(6):621-625. doi:10.1016/j.acap.2015.08.004.
8.
Turchi R, Berhane Z, Bethell C, Pomponio A, Antonelli R, Minkovitz C. Care coordination for CSHCN: Associations with family-provider relations and family/child outcomes. Pediatrics. 2009;124 Suppl:S428-34. doi:10.1542/peds.2009-1255O. 17
9.
Cohen E, Lacombe-Duncan A, Spalding K, et al. Integrated complex care coordination for children with medical complexity : A mixed-methods evaluation of tertiary care-community collaboration. BMC Health Serv Res. 2012;12:366.
10.
Liptak GS, Burns CM, Davidson PW, McAnarney ER. Effects of providing comprehensive ambulatory services to children with chronic conditions. Arch Pediatr Adolesc Med. 1998;152:1003-1008. doi:10.1001/archpedi.152.10.1003.
11.
Antonelli R, Stille C, Antonelli D. Care coordination for children and youth with special health care needs: A descriptive, multisite study of activities, personnel costs, and outcomes. Pediatrics. 2008;122(1):E209-E216.
12.
Huth K, Long-Gagne S, Mader J, Sbrocchi A. Understanding the needs of children with medical complexity. MedEdPORTAL. 2018;14(10709).
13.
Huth K, Long-Gagne S, Mader J, Sbrocchi A. Approach to clinical assessment of children with medical complexity. MedEdPORTAL. 2018;14(10765).
14.
Bandura A. Self-efficacy: Toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191-215. doi:10.1037/0033-295X.84.2.191.
15.
Kern D, Thomas PA, Hughes MT. Curriculum Development for Medical Education : A Six-Step Approach. 2nd ed. Baltimore, MD: Johns Hopkins University Press; 2009.
16.
Frank J, Snell L, Sherbino J, eds. The Draft CanMEDS 2015 Physician Competency framework— Series IV. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2015.
17.
HIlliard R, Tallett S, Tabak D. Use of an Objective Structured Clinical Examination as a Certifying Examination in Pediatrics. Ann R Coll Physicians Surg Canada. 2000;33(4):222-228.
18.
Regehr G, MacRae H, Reznick R, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993-997.
19.
Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. doi:10.1177/1049732305276687.
20.
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research. Acad Med. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388.
21.
Bogetz JF, Bogetz AL, Gabhart JM, Bergman DA, Blankenburg RL, Rassbach CE. Continuing education needs of pediatricians across diverse specialties caring for children with medical complexity. Clin Pediatr (Phila). 2015;54(3):222-227. doi:10.1177/0009922814564049.
22.
Shah NH, Bhansali P, Barber A, et al. Children With Medical Complexity: A Web-Based Multimedia Curriculum Assessing Pediatric Residents Across North America. Acad Pediatr. 2018;18(1):79-85. doi:10.1016/j.acap.2017.08.008.
23.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S6318
7. doi:10.1097/00001888-199009000-00045. 24.
Hodges B. Validity and the OSCE. Med Teach. 2003;25(3):250-254. doi:10.1080/01421590310001002836.
25.
Cate O ten. Nuts and Bolts of Entrustable Professional Activities. J Grad Med Educ. 2013;5(1):157-158.
26.
Bandura A. Social Cognitive Theory: An Agentic Perspective. Annu Rev Psychol. 2001;52:1-26.
27.
Bandura A. Health Promotion by Social Cognitive Means. Heal Educ Behav. 2004;31(2):143-164. doi:10.1177/1090198104263660.
28.
Paes P, Leat D, Stewart J. Complex decision making in medical training : key internal and external in fl uences in developing practical wisdom. Med Educ. 2018:1-10. doi:10.1111/medu.13767.
29.
Sullivan G. Getting Off the “‘ Gold Standard ’”: Randomized Controlled Trials and Education Research. J Grad Med Educ. 2011:285-289.
19
Table 1: Resident characteristics (n=44) n (%) Study site Toronto
32 (73)
Ottawa
12 (27)
Female
29 (66)
Male
12 (27)
Not indicated
3 (7)
Gender
Age (years) 20-24
1 (2)
25-29
27 (61)
30-34
12 (27)
35-39
1 (2)
Not indicated
3 (7)
Postgraduate Year PGY 1
18 (40)
PGY 2
13 (30)
PGY 3
13 (30)
Completed clinical rotation in complex care? Yes
2 (5)
No
39 (89)
Not indicated
3 (7)
Received formal training in complex care? Yes
11 (25)
No
29 (66)
20
Not indicated
4 (9)
How often actively involved in the care of CMC? Inpatient Never
0 (0)
Rarely
0 (0)
Sometimes
12 (27)
Frequently
15 (34)
Very frequently
14 (32)
Not indicated
3 (7)
Outpatient Never
7 (16)
Rarely
19 (43)
Sometimes
11 (25)
Frequently
3 (7)
Very frequently
1 (2)
Not indicated
3 (7)
How comfortable are you caring for CMC? Not at all comfortable
1(2)
Slightly comfortable
13 (30)
Moderately comfortable
22 (50)
Quite comfortable
5 (11)
Extremely comfortable
0 (0)
Not indicated
3 (7)
21
Table 2: Emerging themes and exemplar quotations Theme
Exemplar quotation
Building a System of Care
“I felt this collaborative and team approach…really value the dietician… the physiotherapist who follows the patient, the respiratory therapist, the OT, all of these people who know the patient much more than you… these are the things that I take, like acknowledging other people’s roles and responsibilities.” (#38)
Collaboration with interdisciplinary team Communication of care plans Caregiver as resource
“We rely on the parents’ expertise... And becoming a lot more comfortable with that uncertainty… Like, we never really fully, fully understand maybe all of the things that are going on and… being OK with that.” (#35)
Navigating Uncertainty Recognizing knowledge gaps Feelings associated with complexity Challenges with decision-making
“No one person can take care of all of the complex needs… I think that really makes you, more, of a manager… not just as, someone who sees them for the acute issues but also, who follows them, and arranges their care through other people as well and I think that really... epitomizes a lot of pediatrics.” (#16)
Professional Identity Formation Developing approach to care Clinical exposure Value of longitudinal relationships
Table 3: Educational needs identified by residents Educational needs Structured, formal teaching in complex care Complex care training earlier in residency Approaches to troubleshooting common medical issues Care of CMC in the home and community Active participation in interdisciplinary team member clinical encounters Hands-on medical technology management Longitudinal clinical care
22
Figure 1: Flow diagram of resident participation in CMC curriculum (intervention) versus usual training (control).
23
Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial Kathleen Huth MD MMSc , Tobey Audcent MD , Sara Long-Gagne MD MSc , Anne Marie Sbrocchi MD , Natalie Weiser MA , Doug Miller , Danielle Arje , Derek Stephens MSc , Nathalie Major MD , Kheirie Issa MD , Eyal Cohen MD MSc , Julia Orkin MD MSc PII: DOI: Reference:
S1876-2859(19)30385-7 https://doi.org/10.1016/j.acap.2019.09.002 ACAP 1409
To appear in:
Academic Pediatrics
Received date: Accepted date:
3 May 2019 7 September 2019
Please cite this article as: Kathleen Huth MD MMSc , Tobey Audcent MD , Sara Long-Gagne MD MSc , Anne Marie Sbrocchi MD , Natalie Weiser MA , Doug Miller , Danielle Arje , Derek Stephens MSc , Nathalie Major MD , Kheirie Issa MD , Eyal Cohen MD MSc , Julia Orkin MD MSc , Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial, Academic Pediatrics (2019), doi: https://doi.org/10.1016/j.acap.2019.09.002
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier Inc. on behalf of Academic Pediatric Association
Evaluating curricular modules in the care of children with medical complexity: A mixed-methods randomized controlled trial
Kathleen Huth MD MMSc1, Tobey Audcent MD2, Sara Long-Gagne MD MSc3, Anne Marie Sbrocchi MD3, Natalie Weiser MA4, Doug Miller5, Danielle Arje4, Derek Stephens MSc4, Nathalie Major MD2, Kheirie Issa MD2, Eyal Cohen MD MSc4,5, Julia Orkin MD MSc4,5.
Boston Children’s Hospital, Boston, MA; 2Children’s Hospital of Eastern Ontario, University of Ottawa, Ottawa, ON; 3Montreal Children’s Hospital, McGill University, Montreal, QC; 4 Paediatric Medicine, The Hospital for Sick Children, Toronto, ON; 5 Department of Paediatrics, University of Toronto, Toronto, ON. 1
Corresponding author Kathleen Huth Boston Children’s Hospital 300 Longwood Ave, Boston MA, 02115 Tel: 617-355-6162 Email: [email protected]
Keywords Complex care, residency education, postgraduate medical education, pediatrics, self-efficacy.
Running title Mixed-methods evaluation of a complex care curriculum
Funding sources This work was supported by the Norman Saunders Complex Care Initiative and the Paediatric Consultant’s Education Development and Innovation Grants, The Hospital for Sick Children, Toronto Ontario, Canada.
Conflicts of interest The authors have no conflicts of interest to disclose.
What’s New We performed a randomized control trial of a standardized complex care curriculum for pediatric residents using quantitative and qualitative methods. Our findings provide insight into effective educational strategies for increasing resident self-efficacy in the care of children with medical complexity.
ABSTRACT
Objective: Children with medical complexity (CMC) are a growing population, yet training in complex care varies across pediatric residency programs. The purpose of this study was 1) to evaluate the effectiveness of a curriculum for pediatric residents in improving performance in a simulated clinical scenario; and 2) to explore residents’ perceived self-efficacy in caring for CMC.
Methods: A randomized controlled trial was conducted supplemented by qualitative inquiry. Pediatric residents from two residency programs were randomly assigned to participate in interactive modules on: (a) clinical assessment, care planning and technological dependency or (b) non-complex care topics. The primary outcome was mean score on an Observed Structured Clinical Examination (OSCE) of tracheostomy care. Semi-structured interviews were conducted post-intervention and analyzed using qualitative content analysis.
Results: Ninety-four eligible residents were randomized. Residents who attended all modules and the OSCE and consented to participate [intervention (n=20) and control (n=24)] were included in the final analysis. At baseline, few (9%) reported being comfortable caring for CMC. There was no significant difference in mean OSCE score between intervention and control groups (39.0±1.1 vs. 38.0±1.0, p=0.48). Qualitative analysis revealed three emerging themes related to resident self-efficacy: building a system of care, navigating uncertainty and professional identity formation.
2
Conclusion: A standardized complex care curriculum delivered in a classroom setting did not lead to improved performance in an OSCE station despite increased resident-reported self-efficacy in approaching care for CMC. These findings highlight the need for multidimensional educational interventions and assessments in complex care.
INTRODUCTION Children with medical complexity (CMC) are characterized by chronic medical conditions and functional limitations with dependence on medical technology.1 Due to their unique and specialized care needs, CMC are at increased risk of medical error and care fragmentation, and therefore require care that is comprehensive, coordinated, and proactive.2 The prevalence of CMC is increasing due to improved medical technologies and advances in care.3 General pediatricians have taken on a larger role in the management of their chronic multisystem diseases and medical technology, both in hospital and in community practice.2
Currently, pediatric residency programs across North America offer variable clinical exposure and educational opportunities in complex care. As complex care is an evolving field of practice, structured training programs are lacking despite recognition of increasing need.4–6 Pediatric residents may have different levels of expertise and comfort in the care of CMC depending on their motivation to seek additional training in complex care at their academic center. Numerous publications have highlighted the need for formal training during pediatric residency to support the delivery of safe, evidence-based care for CMC.4,5,7 Equipping future pediatricians with clinical skills in complex care is essential for the delivery of comprehensive care for CMC in their community, which may help optimize outcomes including improved patient/family experience, quality and safety performance measures and reduced healthcare 3
utilization.8–11
Responding to the educational gaps in complex care, members of the study team (KH, SLG, AMS) designed a training experience for pediatric residents to develop competence in caring for CMC.12,13 Curriculum development and evaluation was guided by self-efficacy as a theoretical framework, which indicates that an expectation of personal efficacy in a particular area determines what actions will be taken, what goals are made, how much effort is expended, and what outcomes are expected.14 We hypothesized that an educational experience focused on standard clinical approaches in complex care would impact residents’ clinical encounters with patients/families as well as enhance confidence in their ability to provide high-quality care for CMC as future pediatricians.
The overarching study aim was to identify whether a standardized complex care curriculum improves pediatric residents’ clinical performance and self-efficacy. Specific study objectives were: 1. To determine the comparative effectiveness of participation in a standardized curriculum on performance on an objective structured clinical exam (OSCE) station of tracheostomy care, relative to residents receiving an educational session unrelated to complex care, evaluated by total mean score on an objective structured clinical exam (OSCE) of tracheostomy care. 2. To explore changes in residents' self-efficacy in caring for CMC following participation in the curriculum.
4
METHODS Study Participants The study was conducted at The Hospital for Sick Children and the Children’s Hospital of Eastern Ontario in Ontario, Canada. The pediatric residency programs consisted of 41 (Ottawa) and 72 (Toronto) learners across four postgraduate years and both had weekly academic half-day teaching. All pediatrics residents enrolled at the study sites were eligible for participation. Exclusion criteria were enrolment in subspecialty residency training.
Study Design We conducted a randomized controlled trial (RCT) of a standardized curriculum. This was a mixedmethods study with sequential explanatory design, wherein quantitative data collection was followed by qualitative study to better understand the findings from the first phase. Residents were randomized (allocation ratio 1:1, random block size stratified by site) to attend either a standardized complex care curriculum or regular academic sessions unrelated to complex care. One month after collection of outcome measurements, each group received the alternative training. A survey was administered at baseline including level of training, previous complex care experience, and comfort in caring for CMC.
Curriculum Development We used a systematic approach to curriculum development to meet training needs in complex care for the graduating pediatrician, with learning objectives aligned with the Canadian CanMEDS Physician Competency Framework.15,16 The intervention is a set of four curricular modules focused on key areas of competence in complex care: understanding the needs of CMC; clinical assessment; advocacy, leadership
5
and care planning; and care of the child with technology dependence; two of which are publicly available on MedEdPORTAL with associated formative assessments.12,13 The modules focused on skill development in approaching history-taking and physical examination for a child with medical complexity using a clinical case, developing proactive action plans with families, and routine care and management of common tracheostomy, feeding tube, and central line issues. The modules are organized into 60-minute sessions consisting of interactive case-based activities with accompanying facilitator guides. The curriculum was piloted with pediatrics residents in another Canadian program not participating in the study and revised based on participant and facilitator feedback.
Outcomes Our primary outcome was the score obtained on a scenario on history-taking for a child with tracheostomy in the Spring 2017 national pediatric in-training OSCE, two months after participation in the curriculum. The OSCE examiners received standard faculty development currently offered in each center for OSCE evaluation. They were provided with a standardized checklist for scoring data gathering, data interpretation and management of a child with blood-tinged secretions from his tracheostomy, in addition to a global assessment rating, for a total score of 50. OSCE examiners were blinded to group assignment.
A secondary outcome of the study was resident self-efficacy in the care of CMC, explored in semistructured interviews. A purposive sample of residents representing different levels of training and clinical exposure to CMC were approached by a research assistant (RA) one to three months after curriculum delivery, for voluntary participation in a semi-structured interview. Interviews were conducted over the telephone by a RA with experience in qualitative interviewing, and explored residents’ perceived capability in caring for CMC. The interview guide was pilot-tested and adapted iteratively. 6
Written informed consent was obtained from all participants. The study was approved by the institutional research ethics boards at both study sites. No financial incentives were provided for any part of the study. ClinicalTrials.gov Identifier: NCT03349541.
Data Analysis Quantitative Analysis Descriptive statistics were used to summarize demographics, survey responses and OSCE scores. Mean OSCE scores from the intervention and control groups were compared using a two-way ANOVA, with an interaction between group (intervention and control) and PGY group (junior and senior). Based on a prior study reporting mean and standard deviation for OSCE scores across all years of training,17 in order to detect a difference of one standard deviation in OSCE scores between intervention and control groups with 80% power at the 5% level of significance, a minimum sample size of 18 participants in each group was required. Aggregate OSCE scores from prior years at one of the study sites showed that the total mean score increased by 5-10% with each PGY, so this was thought to represent a clinically important difference.17,18
Qualitative Analysis Interviews were analyzed using conventional content analysis.19 Interviews were audio-recorded, transcribed, and de-identified. Transcripts were uploaded to qualitative data analysis software (NVivo 10). An immersive reading of each transcript was performed independently by three researchers (KH, 7
NW, JO) to identify themes emerging from the data. The researchers collaborated to discuss initial interpretations and develop a preliminary coding scheme. Transcripts were re-analyzed using this coding scheme to refine concepts. Data collection and analysis occurred iteratively, with sampling until the point of thematic saturation. A checklist was used to ensure we adhered to appropriate standards for qualitative research.20
RESULTS Quantitative Ninety-four residents across both residency programs were eligible for participation and randomized to the intervention or control groups (Figure 1). Forty-four residents participated in all curricular modules, attended the OSCE and consented to release their scores. Demographics of the study sample are described in Table 1. Residents were in PGY1 (40%), PGY2 (30%) and PGY3 (30%). Few residents (9%) reported being quite or extremely comfortable caring for CMC at baseline. Two percent had completed a complex care clinical rotation prior to the training session. Residents described themselves as being frequently/very frequently involved in inpatient care of CMC (66%); fewer residents described this level of involvement in outpatient care (9%).
Senior residents (PGY2-3) had significantly higher mean OSCE scores than junior residents (PGY1) (p=0.02). There was no significant difference in mean OSCE score between intervention and control groups (39.0±1.1 vs. 38.0±1.0, p=0.48). There was no significant interaction effect for PGY group (p=0.62).
8
Qualitative We conducted 13 interviews. Analysis revealed three emergent themes in relation to the care of CMC: 1) building a system of care, 2) navigating uncertainty and 3) professional identity formation (Table 2). Residents provided feedback on the educational experience, describing overall satisfaction with the modules, appreciation for dedicated complex care training, and increased comfort in approaching care for CMC. Residents also identified perceived learning gaps in complex care and next steps for curriculum development.
Building a system of care Residents recognized the importance of a collaborative approach to care for CMC within a larger system of care. Many described challenges with clinical decision-making in cases where they had limited understanding of the roles within the care team: I think the hardest part with complex care kids is the number of people involved in their care and figuring out who the appropriate person to contact is... you want to change something but you might not understand the context in which the original decisions were made for that child. So I think that’s the most difficult part from a medical standpoint for them. (#21)
Residents identified the need for various resources to inform care decisions for children with high healthcare utilization, including the interdisciplinary team, the shared care plan and the caregiver. Leveraging resources and recognizing the team as a support system was recognized as critical in approaching the care of CMC: Relying on staff, again relying on parents, like I said, is a huge one and then, like, relying on, like, the expertise of the complex care team. I think the more heads in the game… The better. It helps, like, alleviate some of that feeling of overwhelming. Kind of like sharing it. (#35).
9
Residents noted that the approach to complex care that they learned involved collaboration within a broader system of care, and this increased their comfort in caring for CMC.
Navigating uncertainty Residents described experiences navigating uncertainty as being central to their care of CMC. They identified knowledge gaps and limitations in the care of this specialized population compared to their relative knowledge caring for patients with simple or single clinical issues. Residents described the nature of complexity as involving multiple interdependent issues: …my concern is sometimes if I take the traditional way or the way I know how to correct a problem, is that, does that have contraindications to their other issues?... is that gonna then send, you know, something else way out of whack?... You know how to tackle all their issues individually, but then once you put them in the same kid… that makes it more difficult. (#52)
Residents described having less confidence in their clinical skills due to a lack of predictability in the care of CMC. …we can’t anticipate the way that they’re going to respond to an intervention, like I feel that we can with a lot of other children. And so that’s, that’s just hard because you’re kind of in this unknown realm where you might try something but the reaction you might get from the child is going to be not, completely not what you expected. (#14)
A number of residents described feeling more comfortable in the face of uncertainty, often through partnership with the child’s caregiver. They recognized the importance of acknowledging limitations and exploring this uncertainty with caregivers.
10
Professional identity formation Developing a professional identity as a future pediatrician caring for CMC was a predominant theme. Residents described a transition from seeing acute medical issues “with a narrow-minded focus” to “thinking more holistically”, recognizing the pediatrician’s role in working with caregivers toward a “bigger picture” as it was modelled by curriculum facilitators.
Some residents described feeling anxious when collaborating with caregivers in their child’s care, particularly if the conversation was related to a more chronic issue. One resident said they had felt “really uncomfortable or inadequate… you’re supposed to be this expert”. With further training and clinical exposure, residents re-envisioned their relationship with the caregiver as more of a partnership, rather than as a threat to their role: these parents both know their kids and usually know like why or how it’s related to their problem so it’s kind of like you have a partner working with you so I… find it really helpful. (#11)
Residents described developing their clinical skills through repeated experiences with CMC. Providing continuity of care and serving as a manager or coordinator of care, beyond episodic management of acute issues, was identified as being central to the pediatrician’s role. Residents were motivated to be able to oversee the care of CMC as a future pediatrician, with the goal being “I’m kind of like the go-to person for that family”.
Educational experience Residents provided feedback on aspects of the complex care curriculum that contributed to their comfort in caring for CMC, as well as ongoing training needs (Table 3). For almost all residents, this intervention 11
was their first formal training experience in complex care despite frequently providing clinical care for this population: it’s hard because we’re expected to care for these patients but we don’t actually get any, formal teaching in, in caring, or I guess, formal clinical experience in caring for them, except for now that they’re introduced this half-day, so that’s been really, really helpful. (#14)
Residents appreciated having an approach to clinical assessment of CMC as well as understanding more about life outside of the hospital: the [curriculum] provided us with kind of an approach on how to take an adequate history or what resources are out there, when you’re managing a complex care patient… what the home environment might look like… And it also provided context for, why families might be tired or annoyed… what their day-to-day might look like… definitely helped build my comfort in managing these patients... (#7)
Educational gaps identified included lack of exposure to home and community care of CMC, gaining deeper insight into interdisciplinary roles and need for regular clinical teaching in different care settings throughout training. Residents advocated for classroom-based teaching about care of CMC in the home and community to be accompanied by meaningful clinical experiences: I just wish I had more experience with seeing them out in the community… that’s a burden of the family of getting them ready in the morning, getting to the appointment, dealing with whatever happens in the appointment… That was something that was highlighted [in the curriculum] and so I think that needs to be reinforced by, like, us going and seeing. Like, if I ask, you know, Johnny to come in for an appointment, what is involved in getting him there and getting him home? (#18)
The complex care curriculum allowed residents to recognize educational gaps, and motivated them to seek immersive training experiences to increase comfort in the care of CMC.
12
DISCUSSION This multi-site mixed-methods RCT evaluated the impact of participation in a standardized curriculum on measures of pediatric residents’ knowledge, skills and attitudes in the care of CMC, providing insight into effective strategies and outstanding needs in complex care education. Participation in a standardized curriculum in a classroom setting did not lead to improved performance on an OSCE station on historytaking for a child with tracheostomy. However, following participation in the curriculum, residents described increased comfort in caring for CMC and recognizing their role in a care team. Our study responds to an unmet educational need in pediatrics, intending to prepare future pediatricians to provide high-quality care for a vulnerable pediatric population.
We found that pediatric residents described challenges, discomfort and educational gaps in caring for CMC at baseline, similar to those described by trainees and practicing pediatricians in the literature.4,7,21 Prior work in complex care education has focused on hospital-based care and expertise in the management of medical technology. Shah et al. evaluated web-based training in complex care and reported knowledge acquisition and improved verbal handoff of a simulated patient with complex care needs.22 Yet there is a lack of dedicated training on the lives of CMC outside of a hospital setting, and previous study describes residents’ lack of comfort in identifying appropriate supports and care plans for these patients and families.4 Pediatric residency training typically emphasizes time on inpatient wards and in acute care, and the time that is dedicated to ambulatory care is often focused on addressing a specific chief complaint. Many residency programs do not have a dedicated complex care clinical rotation, and existing rotations are often elective and primarily inpatient. Additional educational strategies are needed to foster competence in managing common and challenging issues in complex care, including simulation, direct observation of clinical encounters and feedback throughout residency training.
13
We found no significant change in mean OSCE score for residents who participated in the complex care curriculum. Our negative RCT highlights limitations in one-off training sessions without accompanying longitudinal clinical experiences and limitations in existing assessments of competence in complex care. Our finding may provide insight into the “dose” of training necessary to see an effect on clinical skills in complex care. The lack of an effect of the training session on the primary outcome may also be explained by discordance between training on a comprehensive clinical approach emphasizing team communication, and an assessment focused on knowledge-based expertise for a specific medical technology. We used an OSCE score as our primary outcome measure to evaluate knowledge transfer to a simulated clinical setting. While the OSCE does not directly translate to workplace performance, it does fall relatively high on Miller’s framework for assessing levels of competence,23 testing behavior which can show how individuals may apply knowledge in the future. Yet performance in a simulated setting may not predict future clinical practice. The validity of the OSCE as an assessment tool varies on the measures used, the context in which the examination is administered and the characteristics of the learners.24 There is no validity data in our particular context and population of interest, in particular with the ability of the OSCE to measure communication, collaboration and advocacy skills, all of which are important domains of physician performance.16 Evaluation of competence in complex care must systematically address these roles across multiple and distinctive clinical encounters and practice settings. Multidimensional assessments including direct observation and multisource feedback would be more appropriate methods of establishing competence in this diverse field. Next steps in complex care training and assessment should include defining entrustable professional activities for complex tasks such as care coordination and team management, which would provide clear markers of progressive skill development and translation of complex care competencies into clinical practice.25
Qualitative analysis enabled us to gain a deeper understanding of the impact of dedicated training on a holistic approach to the care of CMC, and to generate hypotheses about how new insights about complex 14
care may influence clinical practice. We identified emerging themes related to residents’ self-efficacy in the care of CMC. The provided training experience acknowledges the inherent uncertainty in clinical decision-making for CMC and models problem-solving for common challenges, fostering trainees’ ability to address these challenges. Bandura’s social cognitive theory described factors that determine motivation and behavior, and the cognitive processes that allow people to achieve their goals.14 Beliefs in one’s efficacy is at the center of what actions one is motivated to take.26 People with high self-efficacy view challenges as possible to overcome with perseverance and improved skills.27 In medical education, selfefficacy has been recognized as an enabler of complex decision-making and development of “practical wisdom” through experience.28 In our study, following the training residents described increased comfort in the approach to clinical assessment of CMC, management of chronic and interdependent issues, and motivation to collaborate with families and an interdisciplinary team. A pediatric resident’s sense of selfefficacy, or perceived capability, in caring for CMC, is important to foster for pediatricians to navigate challenges, partner with families and ultimately provide high-quality care for CMC.
We used a randomized controlled trial design with blinded outcome assessments to minimize risk of bias, supplemented by rigorous qualitative analysis, but nevertheless this study had a number of important limitations. Similar to challenges with other educational RCTs, we had a small sample size, high dropout rate, variability in how the intervention is administered and its timing, and multiple contextual factors impacting the outcome.29 Though 94 residents were eligible, only about half participated in all aspects of the educational intervention and the OSCE evaluation. A similar proportion of resident attrition has been reported in the literature,22 though our study had a smaller sample size, which may have contributed to the lack of our ability to detect the effect of training on clinical skills. There were no large differences in resident characteristics including PGY and gender between those that were included in the analysis and those that were not. Factors that affected participation in all curricular modules included vacation, illness, post-call, or off-site rotation, all of which should be considered when designing a curriculum evaluation. 15
We are not able to determine what changes in knowledge and skill in complex care may be expected over the course of residency training, and we were not powered to identify differences in knowledge gain by PGY, though previous studies have shown larger gains for more junior trainees.22 The OSCE focused on medical technology management, which did not allow us to adequately identify improved clinical skills related to other curriculum objectives. The OSCE may not be the optimal method to assess competency in complex care, and future research should consider using direct observation in clinical care of CMC as an assessment tool.
The study was performed in two large pediatric residency programs at academic institutions with established complex care programs in North America, which may limit the generalizability of these results to other settings and countries. Although both sites were tertiary-care referral centers with many patients defined as CMC, there may have been differences in the number or nature of CMC seen by each learner across sites. Our study did not evaluate long-term impact of training or direct impact on care of CMC. We anticipate that our findings will inform future study of longitudinal educational interventions in complex care incorporating direct observation and evaluation of clinical outcomes.
CONCLUSIONS Residents’ simulated performance in an OSCE station on tracheostomy care did not improve with dedicated complex care training, however residents reported increased self-efficacy in approaching care of CMC and developing their role within a care team. The complex care curriculum was recognized as a valuable educational experience that motivated trainees to seek opportunities to care for CMC. Our findings support the development of integrative and longitudinal clinical learning opportunities and multidimensional assessments of competence in complex care.
16
ACKNOWLEDGEMENTS This study was supported by the Norman Saunders Complex Care Initiative and the Paediatric Consultant’s Education Development and Innovation Grants at The Hospital for Sick Children. The authors would like to thank the Complex Care Special Interest Group of the Canadian Paediatric Society, the Canadian Pediatric Program Directors, and particularly Dr. Hilary Writer (University of Ottawa) and Dr. Adelle Atkinson (University of Toronto) for their support of this work.
References 1.
Cohen E, Kuo DZ, Agrawal R, et al. Children with medical complexity: An emerging population for clinical and research initiatives. Pediatrics. 2011;127(3):529-538. doi:10.1542/peds.20100910.
2.
Dewan T, Cohen E. Children with medical complexity in Canada. Paediatr Child Health. 2013;18(10):518-522.
3.
Tennant PW, Pearce MS, Bythell M, Rankin J. 20-year survival of children born with congenital anomalies: a population-based study. Lancet. 2010;375(9715):649-656. doi:10.1016/S01406736(09)61922-X.
4.
Nazarian BL, Glader L, Choueiri R, Shipman DL, Sadof M. Identifying what pediatric residents are taught about children and youth with special health care needs and the medical home. Pediatrics. 2010;126 Suppl(December):S183-9. doi:http://dx.doi.org/10.1542/peds.2010-1466O.
5.
Sbrocchi AM, Millar C, Henin C, Allard M, Patel H. Complex care core curriculum for pediatric post-graduate trainees: Results of a North American needs assessment. J Pediatr Rehabil Med. 2015;8(2):141-146. doi:10.3233/PRM-150327.
6.
Hewitt-Taylor J. Caring for children with complex needs: Staff education and training. J Child Heal Care. 2005;9(1):72-86. doi:10.1177/1367493505050506.
7.
Bogetz JF, Bogetz AL, Rassbach CE, Gabhart JM, Blankenburg RL. Caring for children with medical complexity: Challenges and educational opportunities identified by pediatric residents. Acad Pediatr. 2015;15(6):621-625. doi:10.1016/j.acap.2015.08.004.
8.
Turchi R, Berhane Z, Bethell C, Pomponio A, Antonelli R, Minkovitz C. Care coordination for CSHCN: Associations with family-provider relations and family/child outcomes. Pediatrics. 2009;124 Suppl:S428-34. doi:10.1542/peds.2009-1255O. 17
9.
Cohen E, Lacombe-Duncan A, Spalding K, et al. Integrated complex care coordination for children with medical complexity : A mixed-methods evaluation of tertiary care-community collaboration. BMC Health Serv Res. 2012;12:366.
10.
Liptak GS, Burns CM, Davidson PW, McAnarney ER. Effects of providing comprehensive ambulatory services to children with chronic conditions. Arch Pediatr Adolesc Med. 1998;152:1003-1008. doi:10.1001/archpedi.152.10.1003.
11.
Antonelli R, Stille C, Antonelli D. Care coordination for children and youth with special health care needs: A descriptive, multisite study of activities, personnel costs, and outcomes. Pediatrics. 2008;122(1):E209-E216.
12.
Huth K, Long-Gagne S, Mader J, Sbrocchi A. Understanding the needs of children with medical complexity. MedEdPORTAL. 2018;14(10709).
13.
Huth K, Long-Gagne S, Mader J, Sbrocchi A. Approach to clinical assessment of children with medical complexity. MedEdPORTAL. 2018;14(10765).
14.
Bandura A. Self-efficacy: Toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191-215. doi:10.1037/0033-295X.84.2.191.
15.
Kern D, Thomas PA, Hughes MT. Curriculum Development for Medical Education : A Six-Step Approach. 2nd ed. Baltimore, MD: Johns Hopkins University Press; 2009.
16.
Frank J, Snell L, Sherbino J, eds. The Draft CanMEDS 2015 Physician Competency framework— Series IV. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2015.
17.
HIlliard R, Tallett S, Tabak D. Use of an Objective Structured Clinical Examination as a Certifying Examination in Pediatrics. Ann R Coll Physicians Surg Canada. 2000;33(4):222-228.
18.
Regehr G, MacRae H, Reznick R, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993-997.
19.
Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. doi:10.1177/1049732305276687.
20.
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research. Acad Med. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388.
21.
Bogetz JF, Bogetz AL, Gabhart JM, Bergman DA, Blankenburg RL, Rassbach CE. Continuing education needs of pediatricians across diverse specialties caring for children with medical complexity. Clin Pediatr (Phila). 2015;54(3):222-227. doi:10.1177/0009922814564049.
22.
Shah NH, Bhansali P, Barber A, et al. Children With Medical Complexity: A Web-Based Multimedia Curriculum Assessing Pediatric Residents Across North America. Acad Pediatr. 2018;18(1):79-85. doi:10.1016/j.acap.2017.08.008.
23.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S6318
7. doi:10.1097/00001888-199009000-00045. 24.
Hodges B. Validity and the OSCE. Med Teach. 2003;25(3):250-254. doi:10.1080/01421590310001002836.
25.
Cate O ten. Nuts and Bolts of Entrustable Professional Activities. J Grad Med Educ. 2013;5(1):157-158.
26.
Bandura A. Social Cognitive Theory: An Agentic Perspective. Annu Rev Psychol. 2001;52:1-26.
27.
Bandura A. Health Promotion by Social Cognitive Means. Heal Educ Behav. 2004;31(2):143-164. doi:10.1177/1090198104263660.
28.
Paes P, Leat D, Stewart J. Complex decision making in medical training : key internal and external in fl uences in developing practical wisdom. Med Educ. 2018:1-10. doi:10.1111/medu.13767.
29.
Sullivan G. Getting Off the “‘ Gold Standard ’”: Randomized Controlled Trials and Education Research. J Grad Med Educ. 2011:285-289.
19
Table 1: Resident characteristics (n=44) n (%) Study site Toronto
32 (73)
Ottawa
12 (27)
Female
29 (66)
Male
12 (27)
Not indicated
3 (7)
Gender
Age (years) 20-24
1 (2)
25-29
27 (61)
30-34
12 (27)
35-39
1 (2)
Not indicated
3 (7)
Postgraduate Year PGY 1
18 (40)
PGY 2
13 (30)
PGY 3
13 (30)
Completed clinical rotation in complex care? Yes
2 (5)
No
39 (89)
Not indicated
3 (7)
Received formal training in complex care? Yes
11 (25)
No
29 (66)
20
Not indicated
4 (9)
How often actively involved in the care of CMC? Inpatient Never
0 (0)
Rarely
0 (0)
Sometimes
12 (27)
Frequently
15 (34)
Very frequently
14 (32)
Not indicated
3 (7)
Outpatient Never
7 (16)
Rarely
19 (43)
Sometimes
11 (25)
Frequently
3 (7)
Very frequently
1 (2)
Not indicated
3 (7)
How comfortable are you caring for CMC? Not at all comfortable
1(2)
Slightly comfortable
13 (30)
Moderately comfortable
22 (50)
Quite comfortable
5 (11)
Extremely comfortable
0 (0)
Not indicated
3 (7)
21
Table 2: Emerging themes and exemplar quotations Theme
Exemplar quotation
Building a System of Care
“I felt this collaborative and team approach…really value the dietician… the physiotherapist who follows the patient, the respiratory therapist, the OT, all of these people who know the patient much more than you… these are the things that I take, like acknowledging other people’s roles and responsibilities.” (#38)
Collaboration with interdisciplinary team Communication of care plans Caregiver as resource
“We rely on the parents’ expertise... And becoming a lot more comfortable with that uncertainty… Like, we never really fully, fully understand maybe all of the things that are going on and… being OK with that.” (#35)
Navigating Uncertainty Recognizing knowledge gaps Feelings associated with complexity Challenges with decision-making
“No one person can take care of all of the complex needs… I think that really makes you, more, of a manager… not just as, someone who sees them for the acute issues but also, who follows them, and arranges their care through other people as well and I think that really... epitomizes a lot of pediatrics.” (#16)
Professional Identity Formation Developing approach to care Clinical exposure Value of longitudinal relationships
Table 3: Educational needs identified by residents Educational needs Structured, formal teaching in complex care Complex care training earlier in residency Approaches to troubleshooting common medical issues Care of CMC in the home and community Active participation in interdisciplinary team member clinical encounters Hands-on medical technology management Longitudinal clinical care
22
Figure 1: Flow diagram of resident participation in CMC curriculum (intervention) versus usual training (control).
23