- Email: [email protected]
A mock job interview to assess an interprofessional education program
Journal of Interprofessional Education & Practice 15 (2019) 94–99
Contents lists available at ScienceDirect
Journal of Interprofessional Education & Practice journal homepage: www.elsevier.com/locate/jiep
A mock job interview to assess an interprofessional education program a,∗
a
Jodi Polaha , Ivy Click , Brian Cross a b
a,b
b
b
, Adam Welch , Rick Hess , Jessica Burchette
b
T
ETSU Department of Family Medicine, Quillen College of Medicine, PO Box 70621, Johnson City, TN, 37614, USA Department of Pharmacy Practice, Gatton College of Pharmacy, PO Box 70657, Johnson City, TN, 37614, USA
ARTICLE INFO
ABSTRACT
Keywords: Interprofessional education Competencies Measurement
Background: Evaluation of interprofessional education (IPE) has been limited to students’ self-report and rarely assessed content validity using real-world input. Purpose: This study compared students who did and did not complete an IPE curriculum using a mock interview. Aims were to compare them: 1) in terms of competencies around team based care, and 2) as rated by clinicians providing team based care. Method: Students participated in a mock field placement interview. Study staff rated transcripts on IPE competencies. Clinicians rated and ranked students in terms of their knowledge/values and preference for hiring. Discussion.: IPE students had higher ratings on seven of eleven competences than non-IPE students. Clinical experts rated IPE students higher and ranked them as more preferable. Conclusions: This study demonstrates the efficacy and validity of foundational IPE beyond self-report. IPE impacted students’ responses in a real-world scenario in a way that discriminated them from their peers.
1. Introduction Collaborative practice, or team-based care, is increasingly common in health care, supported by specific policies and evidence of improved patient outcomes.1–4 In response, health professions’ training programs have invested resources into Interprofessional Education (IPE) and their accrediting bodies now require IPE as part of the curricula.5 A growing evidence base shows IPE has an impact on a range of outcomes including self-reported attitudes about team care and IPE,6 satisfaction with and self-reported increased knowledge (Polaha, Schetzina, & Baker, 2016), team communication,8 and patient outcomes (e.g., Ref. 10. Measuring IPE outcomes is challenging, however, particularly for programs that target beginning learners.11 While such students may be ready to learn core interprofessional concepts, they are not yet engaged in clinical environments. Thus, there are few opportunities to measure the impact of IPE on clinical outcomes (e.g., collaborative behavior or patient response), though it is a recommended area of focus for IPE research.12 A review of IPE research showed the majority of studies (77%) measured attitudes and knowledge, and most (76%) were assessed through self-report using surveys.14 These targets and methods are at the lowest level of rigor, according to the Kirkpatrick-Barr Framework.16 Barr et al. and others recommend more rigor in measuring IPE outcomes by focusing on methods that assess student behavior
∗
change (Level 4; Kirkpatrick-Barr16; and evaluate the effectiveness of IPE using more sophisticated methods to include curriculum-based approaches that engage a mixed methods approach (10, 12. In addition to weaknesses in measuring IPE outcomes, the field has struggled to align IPE objectives with the needs of real-world clinical settings.12 The critical link between health professions' training and the needs evolving in the dynamic healthcare industry is a topic of significance across all training aspects including IPE (The Lancet Commissions, 2010). The conceptual model for researching IPE proposed by the12 laid out key relationships between the developmental progression of training among health professions’ students, the continuum of IPE outcomes (from student reactions/attitudes to performance in practice) and its inter-relationship with the needs of health care environments. The present study of beginning learners was designed to measure IPE outcomes in a way that addresses present weaknesses by: 1) moving beyond self-reported attitudes/knowledge, 2) utilizing a mixedmethods approach, and 3) connecting the curriculum content with the needs of real-world interprofessional care using team care experts. To do so, a novel paradigm was used: a mock interview for an advanced training experience, to evaluate students' values and knowledge around team care. We conducted interviews with second year medical and pharmacy students, half of whom had completed a formalized IPE curriculum and half of whom had not. Importantly, the students were unaware of the true purpose of the interview, in that they were told the
Corresponding author. E-mail address: [email protected] (J. Polaha).
https://doi.org/10.1016/j.xjep.2019.03.001 Received 6 April 2018; Received in revised form 5 February 2019; Accepted 5 March 2019 2405-4526/ © 2019 Elsevier Inc. All rights reserved.
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
interview was to “see what you've learned” in their program to date and “get feedback on your interview skills.” In other words, we designed the study to assess whether students in the IPE curriculum spontaneously incorporated knowledge and values around team-based care into an interview that was not marked for that purpose. Health care based (non-academic) clinicians with expertise in team care, also unaware of the true purpose of the study, provided a measure of external validity. Aim 1. The first aim of this study was to compare the two groups’ (IPE vs. non-IPE) responses in terms of knowledge and values about team-based care in terms of the competencies taught in the formalized IPE curricula. In other words, the first aim sought to answer the question, “In an interview, do IPE students spontaneously incorporate the content taught in the IPE curriculum, and does that differentiate them from non-IPE students?” Aim 2. The second aim was to compare the two groups’ (IPE vs. nonIPE) responses as rated by clinicians who provide team based care. This aim sought to answer the question, “Does what we are teaching, when expressed by the learner in an interview, discriminate them from their peers and appeal to real-world clinicians?”
important aspect to this was to avoid eliciting a social desirability effect or participant bias by eliminating any cues that might prompt them to believe the research was designed to study IPE. In other words, the goal was for the students to respond as they might in a “real” job interview. Thus, procedures were designed to eliminate all cues around IPE programming at ETSU. Students were recruited at the beginning of medical and pharmacy classes during the last month of their second year, three months after all IPE programming was completed. Recruitment was conducted by study staff who were not associated with IPE programming. An announcement was made to the entire class that a study was being conducted to evaluate what they had learned in their general program by having them participate in a 15–20 min mock clerkship (medical student) or fieldwork (pharmacy student) interview. They were told that they would have an opportunity to practice their interview skills and gain feedback. Study staff unassociated with teaching in the IPE curriculum conducted an informed consent process in which the students were told that the study was being conducted to “evaluate students’ learning” using a mock interview. After consent, students read an “advertisement” of the training opportunity for which they would be interviewing. The job was worded vaguely, but included an embedded reference to a “multidisciplinary team” and “team care opportunities” (See Fig. 1). Next, staff accompanied students to a conference room, where an actor wearing a white lab coat introduced herself as Dr. Woodward. Staff started a digital recorder and a 20-min timer, and left the room. At 5 min remaining, the staff knocked on the door as an alert. At 20 min, he or she interrupted to conclude the interview. The interviewer followed a semi-structured script, which included a brief introduction followed by three questions increasing in specificity and team-care/IPE content as follows: “Thank you for your interest in this advanced training position. As you know from our ad, this is a highly selective placement so we take our time in these interviews to really get to know how our applicants’ interests and skills match with our work.
2. Methods 2.1. Participants Students. Twenty (20) students participated in the study. All 20 students were completing their second year in either the Pharmacy (N = 10) or Medicine (N = 10) programs at East Tennessee State University (ETSU). Half of the students (5 Pharm, 5 Med) had completed ETSU's formalized IPE curriculum, which was in a pilot stage and offered to students by random selection at the beginning of their first year in the program. Half of the students (5 Pharm, 5 Med) did not complete the IPE program. Expert Clinicians. Twelve (12) clinicians identified as “experts” in team based care participated in the study. There were four clinicians each from medicine, pharmacy, and psychology. Study staff identified clinicians using collegial networks. Clinicians, known to study staff as champions of team-based health care practice but not involved in IPE program development, were targeted for study participation. All of the clinicians reported that they spend at least 20% of their time in teambased care and have been in practice for a range of 4–28 years. ETSU clinicians or colleagues, familiar with the IPE program, were excluded.
1) What interests you about this position and what unique skills or training do you bring to us? Probe: if they only answer one of these two (e.g., only tell interests or only list skills/training), go back and ask about the other. 2) You may have read in our ad that we are an award winning healthcare system staffed by a coordinated multidisciplinary team. What do you know about team-based care? Probe: Can you tell me more? Probe: Can you give me an example? 3) What training have you received in interprofessional or team based care and how might that make you a stronger candidate for this position? Probe: Can you give me an example? Probe: Can you tell me more?”
2.2. Setting ETSU's IPE programming incorporates students from all five Colleges in its Academic Health Sciences Center including Medicine, Pharmacy, Nursing, Public Health and the College of Clinical Rehabilitative Sciences as well as students from the Departments of Social Work and Psychology. The IPE students who participated in this study were part of a 60-student cohort who met with 12 interprofessional faculty for four full-day workshops themed to address each of the core IPE competencies (i.e., Interprofessional Communications, Values and Ethics, Teams and Teamwork, Roles and Responsibilities; IPEC, 2016). Workshops occurred over 18 months early in their respective curricula. Workshops included creative and experiential activities, didactics, and role-play with standardized patients or professionals. Workshop content is based on the Preceptors in the Nexus Toolkit disseminated by the National Center for Interprofessional Practice and Education.17 The activities of each day focused on 3–5 key competencies as appear in Table 2.
At the end of the interview, students were thanked for their time and digital recordings were transcribed, with all identifying information redacted. 2.4. Curriculum review In order to address Aim 1, the extent to which students integrated content specifically taught in the IPE curriculum into their interview responses, three competency raters (first author, Polaha, second author, Click, and a research assistant, Tolliver) scored each transcript. Specifically, raters evaluated transcripts by looking for content around each of 11 competencies and rating each on a scale from 1 (student did not mention any content consistent with this topic/competency) to 5 (student provided data, definitions, specific skills or knowledge on this topic.) The process for scoring was initiated by the three raters scoring one transcript together and creating specific decision rules, scoring two transcripts separately and comparing results to fine tune decision rules,
2.3. Procedure Mock Job Interview. The purpose of the mock job interview was to assess whether students spontaneously integrated knowledge and values from the IPE curriculum into their response to the interviewer. An 95
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Do you have a strong desire to expand your clinical skills and gain invaluable real world experience? Interested in an advanced training opportunity in your ideal geographic location? Goodbet Health Services, a non-profit integrated health care delivery system has an advanced training opportunity for medical/pharmacy students. Our award-winning health system provides exceptional primary care and a range of specialty services throughout the region. All of our clinics are staffed by a coordinated multidisciplinary team. The network serves approximately 300,000 residents in your favorite geographic region. Our skilled providers provide a complete range of services and have a passion for building healthier communities through world-class, compassionate health care. Currently, we have a highly competitive ADVANCED PRACTICE EXPERINCE for your last year of training in the location you prefer! This experience will include: Attractive stipend Housing for duration of experience Excellent team care opportunities Real world clinical experiences Recruitment for this position is organized by Goodbet Health Services, an integrated health care delivery system in your favorite State. Fig. 1. “Advertisement” for student training opportunity.
competency rating. Two-way analysis of variance (ANOVA) tests were conducted to compare differences in competency ratings between medicine and pharmacy students and IPE and non-IPE students and to assess for interactions. Intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals were calculated using SPSS statistical package version 25 (IBM SPSS Inc., Chicago, IL) based on a mean rating (k = 3), consistency, 2-way random-effects model. Inter-rater reliability was considered good to excellent, average measure ICC = 0.935 (0.850-0.975, p < .001). To evaluate the effect of the small sample size, skewness and kurtosis values as well as boxplots were obtained to examine the distributions for all outcome variables. Cases identified as outliers were removed from the sample and inferential statistics conducted again. No differences were noted when including or removing outliers. Aim 2: Clinical Experts Evaluation of Students' Responses. Two-way ANOVAs were conducted to assess the influence of student program (medicine, pharmacy) and curriculum (IPE, non-IPE) on expert ratings of students' values around IPE, values around team-based care, and knowledge and skills in team based care. Chi-square tests were used to assess for differences in experts’ rankings of students, as well as whether experts felt the student had specialized training in IPE. Statistical analyses used SPSS version 25 and the level of statistical significance was set at p ≤ .05.
and finally scoring the remaining transcripts separately. 2.5. Expert clinician review To address Aim 2, comparing the groups from the clinicians' perspectives, each clinician expert completed reviews of four students' transcripts. Transcripts were randomly assigned so that each reviewer had one IPE medical student, one IPE pharmacy student, one non-IPE medical student, and one non-IPE pharmacy student in varying order of appearance. Importantly, clinicians were unaware of the true purpose of the study and told only that it aimed to evaluate students’ knowledge and attitudes about team-based care. They did not know that some students had IPE. Experts received instructions via email, which guided them through a series of sequential steps designed so that later questions would not influence earlier responses. In this way, each expert: 1. Read the students' responses in the transcripts, highlighting phrases that suggested the student had training, values, or skills in teambased care. 2. Rated students on three questions including: values around IPE, values around team based care, and knowledge and skills in team based care as expressed in the interview using a 5 point rating scale in which 1 = “this student did not mention IPE/team based care”, 3 = “this student discussed IPE/team based care and its value to some extent” and 5 = “this student provided specifics/details around IPE/team based care”. 3. Responded “yes” or “no” to the question: Do you feel this student had specialized training in interprofessional work, collaborative practice models, or team-based care? 4. Ranked the four students in terms of which they would most like to hire as part of their team-based practice using 1 (most preferred) through 4 (least preferred) giving rationale.
3. Results Of the 20 participant interviews, three were unusable due to inconsistencies with the interviewer question order. An additional transcript was not returned by a clinician expert, thus 8 IPE (4 Med, 4 Pharm) and 8 non-IPE (4 Med, 4 Pharm) students’ transcripts were used. Aim 1: IPE Content Reflected in Interview. The results of analyses regarding the core areas represented by the IPE competencies as reflected in the interview transcripts can be found in Table 1. Results showed raters scored IPE students' transcripts as significantly higher in terms of Interprofessional Communications (p = .01), Teams and Teamwork (p = .01), and Roles and Responsibilities (p < .001) competencies, as well as overall competency scores (p < .001) than non-IPE students. There was no significant difference between IPE and non-IPE students’ Values and Ethics core competency ratings (p = .133). There was no
Clinical experts returned responses via email and were compensated with a small stipend for the time they contributed to this work. 2.6. Data analysis Aim 1: IPE Content Reflected in Interview. Rating means were calculated for each of the four core competencies as well as an overall 96
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Table 1 Competency ratings in core areas (n = 16)a. Competency
Interprofessional Communication Values and Ethics Teams and Teamwork Roles and Responsibilities Overall a
Program
Curriculum
Medicine (n = 8)
Pharmacy (n = 8)
IPE (n = 8)
Non-IPE (n = 8)
2.12 2.21 2.63 2.75 2.47
2.19 1.75 2.19 2.40 2.15
2.67 ± 0.19 2.29 ± 0.69 3.00 ± 0.80 3.45 ± 0.70 2.91 ± 0.49
1.76 ± 0.69† 1.79 ± 0.51 2.00 ± 0.49† 2.60 ± 0.93‡ 1.88 ± 0.21‡
± ± ± ± ±
0.92 0.73 0.75 0.91 0.65
± ± ± ± ±
0.52 0.36 0.86 0.99 0.61
All values expressed as M ± SD. †p < .05; ‡p < .01.
significant difference between Medical and Pharmacy students in any core or overall competency ratings. There were no significant interactions between student program and curriculum in any core or overall competency ratings. Analyses of individual competencies are found in Table 2. Results showed that the IPE students had significantly higher scores for content consistent with: linking team care with improved patient outcomes (C1, p = .05), identifying one's own personal and professional values and how these relate to values and ethics in team care (V-1, p = .003), and explaining specific team processes (T-2, p = .05) but were no different than non-IPE students on other competencies within the domains of Communications, Values, and Teams. In the Roles Domain, the IPE students had higher ratings on all four competencies (R-1, p = 045; R-2, p = .002; R-3, p = .001; R-4, p = .001). Pharmacy and Medical students demonstrated differing reflection of two competencies: Medical students scored significantly higher in discussing health care policy and best practices in the context of values in team care (V-2, p = .039), and describing the flexible application of roles within the healthcare team (R-2, p = .018). Finally, results showed one significant interaction between program and curriculum, indicating that IPE Medical students were able to describe more team processes/infrastructure that allow for better-coordinated care such as team huddles and/or hand-offs (T-2, p = .041) than were IPE Pharmacy students or Non-IPE students. Aim 2: Clinical Experts' Evaluation of Students’ Responses. Interview
transcripts from all 16 students, evenly split between medicine and pharmacy, were rated by 12 clinician experts. Each transcript was rated up to four times (n = 48). IPE students were rated significantly higher by experts in values around IPE (p = .006), values around team based care (p = .02), and knowledge and skills in team based care (p = .007) than were non-IPE students. (Table 3). Expert raters predicted 83.3% of IPE students received specialized IPE training versus 45.8% of non-IPE students (p = .007). With regard to the question, which student would you hire to be part of your team-based primary care practice? Experts ranked IPE students as significantly more preferred (higher, where “1” is most preferred) than non-IPE students (p = .02; Fig. 2). There were no significant differences between Medical and Pharmacy students in terms of expert ratings or rankings, and no significant interactions between program and curriculum (p > .05). 4. Discussion The present study is one of the first to measure the outcomes of IPE for beginning learners in a real-world situation: a mock advanced placement interview, conducted three months after the completion of an 18-month IPE curriculum. Participants were unaware of the true objectives of the study, and presumably answered open-ended questions about their skills/interest and experience in team-based care as they
Table 2 Ratings for each competencya. Competency
Program
Workshop 1: Interprofessional Communication C-1. To describe the relevance of team communication to improved patient safety and outcomes. C-2. To demonstrate essential communication competencies around crucial conversations.
Curriculum
Medicine
Pharmacy
IPE
Non-IPE
2.44 ± 1.13 1.81 ± 0.67
2.43 ± 0.99 1.96 ± 0.91
3.05 ± 0.78 2.29 ± 0.85
1.96 ± 0.99† 1.56 ± 0.53
3.04 ± 1.05
2.90 ± 1.15
3.81 ± 0.88
2.33 ± 0.67‡
2.07 ± 0.95 1.52 ± 1.08
1.14 ± 0.38† 1.19 ± 0.18
1.43 ± 0.63 1.62 ± 1.22
1.85 ± 1.03 1.19 ± 0.18
3.04 ± 0.77
2.38 ± 1.04
3.00 ± 1.07
2.56 ± 0.82
2.22 ± 1.38
2.00 ± 0.77
3.00 ± 1.12
1.44 ± 0.50†§
3.26 ± 0.78 2.96 ± 1.10 2.44 ± 1.34 2.33 ± 1.07
2.76 ± 0.46 2.00 ± 0.92† 2.29 ± 1.58 2.57 ± 1.37
3.43 3.38 3.57 3.43
2.74 1.89 1.44 1.67
Workshop 2: Values and Ethics V-1. To identify one's own personal and professional values and how these relate to values and ethics in team care. V-2. To discuss health care policy and best practices in the context of values in team care. V-3/4. To explain what social determinants of health are, why they are part of ethical patient-centered care, and how to address them as a team. Workshop 3: Teams and Teamwork T-1. To discuss aspects to team work in any setting (even non-clinical) that impact team functioning and outcomes. T-2. To describe team processes/infrastructure that allow for better-coordinated care such as team huddles and/or hand-offs. Workshop 4: Roles and Responsibilities R-1. To discuss various team members' roles and training. R-2. Describe and discuss the flexible application of roles within the healthcare team. R-3. Discuss leadership as a team role (guild vs. team). R-4. Practice the skill of articulating your role as a member of the healthcare team. a
All values expressed as M ± SD. †p < .05; ‡p < .01; §significant interaction, p < .05. 97
± ± ± ±
0.71 0.91 1.15 0.99
± ± ± ±
0.52† 0.76‡ 0.69‡ 0.55‡
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Table 3 Clinical experts evaluation of students’ responsesa. Question
Program
Values around IPE Values around team based care Knowledge and skills in team based care a
Curriculum
Medicine
Pharmacy
IPE
Non-IPE
3.42 ± 1.32 3.67 ± 1.05 3.38 ± 1.01
3.42 ± 1.25 3.46 ± 1.06 3.33 ± 1.05
3.92 ± 1.02 3.92 ± 1.06 3.75 ± 0.94
2.92 ± 1.32‡ 3.21 ± 0.93† 2.96 ± 0.95‡
All values expressed as M ± SD. †p < .05; ‡p < .01.
might in an actual interview.
from non-IPE peers who did not mention the content at all or only in vague terms. The lack of specificity in their answers could be due to the lapse in time between this interview and the IPE workshops, which were held three months (Roles), seven months (Teams), fifteen months (Values), and eighteen months (Communications) prior. The time lapse may also explain why IPE students scored higher on all Roles competencies than their non-IPE peers: it was the most recent content taught. Another explanation for more generalized responding is that, while the curricula is quite targeted in terms of the specific competencies, there are no tests or other opportunities to rehearse content to fluency or commit to memory. Moreover, because these were beginning learners, they did not have exposure to clinical contexts in which they might observe varied examples of the targeted material. It is important to consider the developmental sequela of IPE competencies, understanding that in order to develop very specific skills of fluency, students will need practice and exposure to wide-ranging examples in practice environments. One observation made was that, in many cases, IPE students discussed content they had learned in the curriculum in response to the interviewer's first and most general question: “What interests you about this position and what unique skills or training do you bring to us?” Thus, after two years of coursework in medicine or pharmacy, IPE students prioritized content learned in the IPE training workshops above other discipline-specific content when listing their own selfperceived assets. It seemed they had assimilated the content into their professional identity and/or understood that the IPE training was unique and gave them an advantage in an interview. That both groups conveyed the value of team based care may reflect that fact that ETSU has developed a culture that supports both interprofessional practice and education in classroom and clinic settings. Indeed both groups of students were exposed to the topic of interprofessional communication in a required introductory course called
4.1. IPE students talked about IPE competencies Regarding Aim 1, results showed that the transcripts of medical and pharmacy students who participated in ETSU's IPE curricula contained significantly more competency-related content than did non-IPE students. Specifically, IPE students scored higher than non-IPE students on seven out of the eleven competencies addressed in the curriculum. Thus, the IPE students were able to incorporate curriculum-based knowledge into their answers regarding why they are a good fit for the position and their experiences in team based care. These findings reflect16 Levels 3 (change in knowledge) and 4 (change in behavior), a shift in rigor over prior studies. Examples of language reflecting competency knowledge include: “(team care) provides more complete care for the patient and I think results from studies show that it provides the best care for patients,” and “… instead of the physician does this, the pharm student does this, the nursing student does this and that, we were supposed to integrate with each other and try to say, ‘hey what do you know that's different from what I learned in my school … ’ like try to apply different … perspectives to the same patient's problem. What we found is that this can actually make a pretty big difference in care.” Such content was not found, or was less specific, in non-IPE transcripts. It is important to note that while IPE students consistently scored higher than non-IPE students, their average scores in each domain were just above the mid-point on a 5 point scale (ranged from 3.75 to 3.92). Anchors for this scale were: 3 = “student mentioned this competency or related content but did not elaborate” and 5 = “student provided data, definitions, specific skills or knowledge on this topic.” Thus, while IPE students were articulating content learned, they did not appear to use the detail or specificity with which it was taught. Rather, they seemed to talk about the content in general terms and differentiated themselves 50.0%
Percent of Students Ranked
45.0% 40.0%
IPE Non-IPE
35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0%
First
Second
Third
Fourth
Ranking (First is Most Preferrable) Fig. 2. Expert clinicians' rankings of students by preference to hire for team based care. 98
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Communication Skills for Health Professionals, which includes students from nursing, psychology, pharmacy and medicine. That said, overall scores for the Values and Ethics domain were the lowest out of all the of the core competency areas. It could be that our curricula lacks traction in this area, or that the interview did not elicit content from this domain. Alternatively, it could be that this core area is the most intangible, both to teach and to measure, a concept that has long been a challenge in the social sciences.18
competency raters who were familiar with the curriculum and could distinguish IPE from non-IPE students by transcript answers, although transcripts were deidentified. 5. Conclusions The results of this study showed that students who participated in an IPE curriculum provided answers in a mock job interview that were seen as preferable by team-based clinicians. This study advanced the rigor evaluation of foundational IPE curricula by targeting behavior (interview content) rather than self-report. Moreover, it is one of the first to connect foundational competencies with what is valued by clinicians in real-world team-based care. Future research should continue to develop sophisticated methods to demonstrate the connection between the classroom/theory based learning of foundational learners and meaningful outcomes across the training trajectory.
4.2. IPE students were judged more positively In addition to documenting competencies, the present study demonstrates the validity of the IPE content learned by asking team care experts from pharmacy, medicine, and psychology to rate and rank the students. Results from Aim 2 showed that clinical experts, unaware of the true purpose of the study or that the sample included “two groups,” rated the IPE students higher in terms of their knowledge of team care, values around team care, and values for IPE. They were able to discriminate which students had IPE from those who had not. When ranking the students in terms of preference to work in team care, they consistently ranked the IPE students as more preferable. The significance of these findings is underscored by the charge to align learning outcomes with the skills and knowledge needed for later application in health care settings.19,20 Evidence that these clinicians were unaware of the purpose of the study and susceptible to participant bias/social desirability is found in their responses to a follow-up question asking them what they thought the purpose to be. None of them guessed that the study included two groups. Rather, responses were “to find out what values or skills are important for professionals in different disciplines,” “how well the students are learning team-based care,” or “different forms of teambased training and education.” One clinician summed up her experience of participating with the comment that she preferred students whose transcript showed they “were able to talk in a way that showed she had actually internalized, valued, and accepted the essence of team based care … instead of focusing on her clinical skills, she focused more on the key ingredients to team based work; likeability, flexibility, openness, enthusiasm, willingness to adapt/work/learn/relearn, communicate easily and effectively … you can train the other skills much more easily than you can help someone become a fluid part of a working team.” Taken together, the results of this study show that ETSU's foundational IPE program effectively teaches key core competencies around team care, at least to the point of students' incorporating that knowledge into their interview answers. Considering learning outcomes across the entire trajectory of training is important in IPE evaluation.19 One interpretation of these findings is that our foundational learners have learned to articulate relatively theoretical and classroom-based IPE material in a meaningful way, and that they value this knowledge. Future research should evaluate how that skill and knowledge base translates to their clinic training experiences in their later in the curriculum. Are they perceived as more prepared by members of the clinic team? Does that preparation translate to accelerated uptake of teambased skills in the clinic setting? Are students who had foundational IPE still discriminable from non-IPE students after a team-based clinic training experience? These questions, as well as questions about their performance further into the future, are all critical questions for further research.
Acknowledgements The authors wish to acknowledge the contributions of the following: Sarah Tolliver (assistance with data entry and analysis); Wendy Guinn (assistance with data collection), Dr. Tese Stephens (assistance with data collection and early conceptualization); and Marty Woodward, who performed the role of the interviewer for all participants. In addition, we would like to thank Dr. Wilsie Bishop, Vice President for Health Affairs, who has championed IPE at ETSU and who funded this study through the Academic Health Sciences Center. References 1. Beacham AO, Kinman C, Harris JG, Masters KS. The patient-centered medical home: unprecedented workforce growth potential for professional psychology. Prof Psychol Res Pract. 2012;43(1):17–23. 2. Kwan BM, Nease DE. The state of the evidence for integrated behavioral health in primary care. In: Talen MR, Valeras AB, eds. Integrated Behavioral Health in Primary Care. New York, NY: Springer; 2013:65–98. 3. Working Party Group on Integrated Behavioral Healthcare. The development of joint principles: integrating behavioral health care into the patient-centered medical home. Ann Fam Med. 2014;12(2):183. 4. Zeind CS, McCloskey WW. Pharmacists' role in the health care setting. Harvard Health Policy Review. 2006;7:147–154. 5. Zorek J, Raehl C. Interprofessional education accreditation standards in the USA: a comparative analysis. J Interprofessional Care. 2013;27(2):123–130. 6. Darlow B, Coleman K, McKinlay E, Donovan S, et al. The positive impact of interprofessional education: a controlled trial to evaluate a programme for health professional students. BMC Med Educ. 2015;15:98. 8. Nurok ML, Evans LA, Lipsitz S, Satwicz P, Kelly A, Frankel A. The relationship of the emotional climate of work and threat to patient outcome in a high-volume thoracis surgery operating room team. BMJ Qual Saf. 2011;20(3):237–242. 10. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013(3) https://doi.org/10.1002/14651858.CD002213.pub3 Art. No.: CD002213. 11. Blue AV, Chesluk BJ, Conforti LN, Holmboe ES. Assessment and evaluation in interprofessional education: exploring the field. J Allied Health. 2015;44(2):73–82. 12. IOM (Institute of Medicine). Measuring the Impact of Inteprofessional Education on Collaborative Practice and Patient Outcomes. Washington, DC: The National Academies Press; 2015. 14. Abu-Rish E, et al. Current trends in interprofessional education of health sciences students: a literature review. J Interprofessional Care. 2012;26:444–451. 16. Barr H, Koppel I, Reeves S, Hammick M, Freeth D. Effective interprofessional education: arguments, assumptions and avidence. In: Meads G, Ashcroft J, Barr H, Scott R, Wild A, eds. The Case for Interprofessional Collaboration. Oxford: Blackwell Publishing Ltd.; 2005. 17. Shrader S, Zaudke JK. Preceptors in the Nexus Toolkit. 2015; 2015 Retrieved from https://nexusipe.org/engaging/learning-system/preceptors-nexus-toolkit. 18. Wong G, Greenhalgh T, Westhorp G, Pawson R. Realist methods in medical education research: what are they and what can they contribute? Med Educ. 2012;26:89–96. 19. Anderson E, Smith R, Hammick M. Evaluating an interprofessional education curriculum: a theory-informed approach. Med Teach. 2016;38:385–394. 20. Thistlewaite JE, Moran M. Learning outcomes for interprofessioanl education (IPE): literature review and synthesis. J Interprofessional Care. 2010;24:503–513.
4.3. Limitations The small number of participants in this study and its evaluation of only one (ETSU) IPE training program limit the generalizability of findings. Another limitation is a potential lack of objectivity in
99
Contents lists available at ScienceDirect
Journal of Interprofessional Education & Practice journal homepage: www.elsevier.com/locate/jiep
A mock job interview to assess an interprofessional education program a,∗
a
Jodi Polaha , Ivy Click , Brian Cross a b
a,b
b
b
, Adam Welch , Rick Hess , Jessica Burchette
b
T
ETSU Department of Family Medicine, Quillen College of Medicine, PO Box 70621, Johnson City, TN, 37614, USA Department of Pharmacy Practice, Gatton College of Pharmacy, PO Box 70657, Johnson City, TN, 37614, USA
ARTICLE INFO
ABSTRACT
Keywords: Interprofessional education Competencies Measurement
Background: Evaluation of interprofessional education (IPE) has been limited to students’ self-report and rarely assessed content validity using real-world input. Purpose: This study compared students who did and did not complete an IPE curriculum using a mock interview. Aims were to compare them: 1) in terms of competencies around team based care, and 2) as rated by clinicians providing team based care. Method: Students participated in a mock field placement interview. Study staff rated transcripts on IPE competencies. Clinicians rated and ranked students in terms of their knowledge/values and preference for hiring. Discussion.: IPE students had higher ratings on seven of eleven competences than non-IPE students. Clinical experts rated IPE students higher and ranked them as more preferable. Conclusions: This study demonstrates the efficacy and validity of foundational IPE beyond self-report. IPE impacted students’ responses in a real-world scenario in a way that discriminated them from their peers.
1. Introduction Collaborative practice, or team-based care, is increasingly common in health care, supported by specific policies and evidence of improved patient outcomes.1–4 In response, health professions’ training programs have invested resources into Interprofessional Education (IPE) and their accrediting bodies now require IPE as part of the curricula.5 A growing evidence base shows IPE has an impact on a range of outcomes including self-reported attitudes about team care and IPE,6 satisfaction with and self-reported increased knowledge (Polaha, Schetzina, & Baker, 2016), team communication,8 and patient outcomes (e.g., Ref. 10. Measuring IPE outcomes is challenging, however, particularly for programs that target beginning learners.11 While such students may be ready to learn core interprofessional concepts, they are not yet engaged in clinical environments. Thus, there are few opportunities to measure the impact of IPE on clinical outcomes (e.g., collaborative behavior or patient response), though it is a recommended area of focus for IPE research.12 A review of IPE research showed the majority of studies (77%) measured attitudes and knowledge, and most (76%) were assessed through self-report using surveys.14 These targets and methods are at the lowest level of rigor, according to the Kirkpatrick-Barr Framework.16 Barr et al. and others recommend more rigor in measuring IPE outcomes by focusing on methods that assess student behavior
∗
change (Level 4; Kirkpatrick-Barr16; and evaluate the effectiveness of IPE using more sophisticated methods to include curriculum-based approaches that engage a mixed methods approach (10, 12. In addition to weaknesses in measuring IPE outcomes, the field has struggled to align IPE objectives with the needs of real-world clinical settings.12 The critical link between health professions' training and the needs evolving in the dynamic healthcare industry is a topic of significance across all training aspects including IPE (The Lancet Commissions, 2010). The conceptual model for researching IPE proposed by the12 laid out key relationships between the developmental progression of training among health professions’ students, the continuum of IPE outcomes (from student reactions/attitudes to performance in practice) and its inter-relationship with the needs of health care environments. The present study of beginning learners was designed to measure IPE outcomes in a way that addresses present weaknesses by: 1) moving beyond self-reported attitudes/knowledge, 2) utilizing a mixedmethods approach, and 3) connecting the curriculum content with the needs of real-world interprofessional care using team care experts. To do so, a novel paradigm was used: a mock interview for an advanced training experience, to evaluate students' values and knowledge around team care. We conducted interviews with second year medical and pharmacy students, half of whom had completed a formalized IPE curriculum and half of whom had not. Importantly, the students were unaware of the true purpose of the interview, in that they were told the
Corresponding author. E-mail address: [email protected] (J. Polaha).
https://doi.org/10.1016/j.xjep.2019.03.001 Received 6 April 2018; Received in revised form 5 February 2019; Accepted 5 March 2019 2405-4526/ © 2019 Elsevier Inc. All rights reserved.
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
interview was to “see what you've learned” in their program to date and “get feedback on your interview skills.” In other words, we designed the study to assess whether students in the IPE curriculum spontaneously incorporated knowledge and values around team-based care into an interview that was not marked for that purpose. Health care based (non-academic) clinicians with expertise in team care, also unaware of the true purpose of the study, provided a measure of external validity. Aim 1. The first aim of this study was to compare the two groups’ (IPE vs. non-IPE) responses in terms of knowledge and values about team-based care in terms of the competencies taught in the formalized IPE curricula. In other words, the first aim sought to answer the question, “In an interview, do IPE students spontaneously incorporate the content taught in the IPE curriculum, and does that differentiate them from non-IPE students?” Aim 2. The second aim was to compare the two groups’ (IPE vs. nonIPE) responses as rated by clinicians who provide team based care. This aim sought to answer the question, “Does what we are teaching, when expressed by the learner in an interview, discriminate them from their peers and appeal to real-world clinicians?”
important aspect to this was to avoid eliciting a social desirability effect or participant bias by eliminating any cues that might prompt them to believe the research was designed to study IPE. In other words, the goal was for the students to respond as they might in a “real” job interview. Thus, procedures were designed to eliminate all cues around IPE programming at ETSU. Students were recruited at the beginning of medical and pharmacy classes during the last month of their second year, three months after all IPE programming was completed. Recruitment was conducted by study staff who were not associated with IPE programming. An announcement was made to the entire class that a study was being conducted to evaluate what they had learned in their general program by having them participate in a 15–20 min mock clerkship (medical student) or fieldwork (pharmacy student) interview. They were told that they would have an opportunity to practice their interview skills and gain feedback. Study staff unassociated with teaching in the IPE curriculum conducted an informed consent process in which the students were told that the study was being conducted to “evaluate students’ learning” using a mock interview. After consent, students read an “advertisement” of the training opportunity for which they would be interviewing. The job was worded vaguely, but included an embedded reference to a “multidisciplinary team” and “team care opportunities” (See Fig. 1). Next, staff accompanied students to a conference room, where an actor wearing a white lab coat introduced herself as Dr. Woodward. Staff started a digital recorder and a 20-min timer, and left the room. At 5 min remaining, the staff knocked on the door as an alert. At 20 min, he or she interrupted to conclude the interview. The interviewer followed a semi-structured script, which included a brief introduction followed by three questions increasing in specificity and team-care/IPE content as follows: “Thank you for your interest in this advanced training position. As you know from our ad, this is a highly selective placement so we take our time in these interviews to really get to know how our applicants’ interests and skills match with our work.
2. Methods 2.1. Participants Students. Twenty (20) students participated in the study. All 20 students were completing their second year in either the Pharmacy (N = 10) or Medicine (N = 10) programs at East Tennessee State University (ETSU). Half of the students (5 Pharm, 5 Med) had completed ETSU's formalized IPE curriculum, which was in a pilot stage and offered to students by random selection at the beginning of their first year in the program. Half of the students (5 Pharm, 5 Med) did not complete the IPE program. Expert Clinicians. Twelve (12) clinicians identified as “experts” in team based care participated in the study. There were four clinicians each from medicine, pharmacy, and psychology. Study staff identified clinicians using collegial networks. Clinicians, known to study staff as champions of team-based health care practice but not involved in IPE program development, were targeted for study participation. All of the clinicians reported that they spend at least 20% of their time in teambased care and have been in practice for a range of 4–28 years. ETSU clinicians or colleagues, familiar with the IPE program, were excluded.
1) What interests you about this position and what unique skills or training do you bring to us? Probe: if they only answer one of these two (e.g., only tell interests or only list skills/training), go back and ask about the other. 2) You may have read in our ad that we are an award winning healthcare system staffed by a coordinated multidisciplinary team. What do you know about team-based care? Probe: Can you tell me more? Probe: Can you give me an example? 3) What training have you received in interprofessional or team based care and how might that make you a stronger candidate for this position? Probe: Can you give me an example? Probe: Can you tell me more?”
2.2. Setting ETSU's IPE programming incorporates students from all five Colleges in its Academic Health Sciences Center including Medicine, Pharmacy, Nursing, Public Health and the College of Clinical Rehabilitative Sciences as well as students from the Departments of Social Work and Psychology. The IPE students who participated in this study were part of a 60-student cohort who met with 12 interprofessional faculty for four full-day workshops themed to address each of the core IPE competencies (i.e., Interprofessional Communications, Values and Ethics, Teams and Teamwork, Roles and Responsibilities; IPEC, 2016). Workshops occurred over 18 months early in their respective curricula. Workshops included creative and experiential activities, didactics, and role-play with standardized patients or professionals. Workshop content is based on the Preceptors in the Nexus Toolkit disseminated by the National Center for Interprofessional Practice and Education.17 The activities of each day focused on 3–5 key competencies as appear in Table 2.
At the end of the interview, students were thanked for their time and digital recordings were transcribed, with all identifying information redacted. 2.4. Curriculum review In order to address Aim 1, the extent to which students integrated content specifically taught in the IPE curriculum into their interview responses, three competency raters (first author, Polaha, second author, Click, and a research assistant, Tolliver) scored each transcript. Specifically, raters evaluated transcripts by looking for content around each of 11 competencies and rating each on a scale from 1 (student did not mention any content consistent with this topic/competency) to 5 (student provided data, definitions, specific skills or knowledge on this topic.) The process for scoring was initiated by the three raters scoring one transcript together and creating specific decision rules, scoring two transcripts separately and comparing results to fine tune decision rules,
2.3. Procedure Mock Job Interview. The purpose of the mock job interview was to assess whether students spontaneously integrated knowledge and values from the IPE curriculum into their response to the interviewer. An 95
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Do you have a strong desire to expand your clinical skills and gain invaluable real world experience? Interested in an advanced training opportunity in your ideal geographic location? Goodbet Health Services, a non-profit integrated health care delivery system has an advanced training opportunity for medical/pharmacy students. Our award-winning health system provides exceptional primary care and a range of specialty services throughout the region. All of our clinics are staffed by a coordinated multidisciplinary team. The network serves approximately 300,000 residents in your favorite geographic region. Our skilled providers provide a complete range of services and have a passion for building healthier communities through world-class, compassionate health care. Currently, we have a highly competitive ADVANCED PRACTICE EXPERINCE for your last year of training in the location you prefer! This experience will include: Attractive stipend Housing for duration of experience Excellent team care opportunities Real world clinical experiences Recruitment for this position is organized by Goodbet Health Services, an integrated health care delivery system in your favorite State. Fig. 1. “Advertisement” for student training opportunity.
competency rating. Two-way analysis of variance (ANOVA) tests were conducted to compare differences in competency ratings between medicine and pharmacy students and IPE and non-IPE students and to assess for interactions. Intraclass correlation coefficient (ICC) estimates and their 95% confidence intervals were calculated using SPSS statistical package version 25 (IBM SPSS Inc., Chicago, IL) based on a mean rating (k = 3), consistency, 2-way random-effects model. Inter-rater reliability was considered good to excellent, average measure ICC = 0.935 (0.850-0.975, p < .001). To evaluate the effect of the small sample size, skewness and kurtosis values as well as boxplots were obtained to examine the distributions for all outcome variables. Cases identified as outliers were removed from the sample and inferential statistics conducted again. No differences were noted when including or removing outliers. Aim 2: Clinical Experts Evaluation of Students' Responses. Two-way ANOVAs were conducted to assess the influence of student program (medicine, pharmacy) and curriculum (IPE, non-IPE) on expert ratings of students' values around IPE, values around team-based care, and knowledge and skills in team based care. Chi-square tests were used to assess for differences in experts’ rankings of students, as well as whether experts felt the student had specialized training in IPE. Statistical analyses used SPSS version 25 and the level of statistical significance was set at p ≤ .05.
and finally scoring the remaining transcripts separately. 2.5. Expert clinician review To address Aim 2, comparing the groups from the clinicians' perspectives, each clinician expert completed reviews of four students' transcripts. Transcripts were randomly assigned so that each reviewer had one IPE medical student, one IPE pharmacy student, one non-IPE medical student, and one non-IPE pharmacy student in varying order of appearance. Importantly, clinicians were unaware of the true purpose of the study and told only that it aimed to evaluate students’ knowledge and attitudes about team-based care. They did not know that some students had IPE. Experts received instructions via email, which guided them through a series of sequential steps designed so that later questions would not influence earlier responses. In this way, each expert: 1. Read the students' responses in the transcripts, highlighting phrases that suggested the student had training, values, or skills in teambased care. 2. Rated students on three questions including: values around IPE, values around team based care, and knowledge and skills in team based care as expressed in the interview using a 5 point rating scale in which 1 = “this student did not mention IPE/team based care”, 3 = “this student discussed IPE/team based care and its value to some extent” and 5 = “this student provided specifics/details around IPE/team based care”. 3. Responded “yes” or “no” to the question: Do you feel this student had specialized training in interprofessional work, collaborative practice models, or team-based care? 4. Ranked the four students in terms of which they would most like to hire as part of their team-based practice using 1 (most preferred) through 4 (least preferred) giving rationale.
3. Results Of the 20 participant interviews, three were unusable due to inconsistencies with the interviewer question order. An additional transcript was not returned by a clinician expert, thus 8 IPE (4 Med, 4 Pharm) and 8 non-IPE (4 Med, 4 Pharm) students’ transcripts were used. Aim 1: IPE Content Reflected in Interview. The results of analyses regarding the core areas represented by the IPE competencies as reflected in the interview transcripts can be found in Table 1. Results showed raters scored IPE students' transcripts as significantly higher in terms of Interprofessional Communications (p = .01), Teams and Teamwork (p = .01), and Roles and Responsibilities (p < .001) competencies, as well as overall competency scores (p < .001) than non-IPE students. There was no significant difference between IPE and non-IPE students’ Values and Ethics core competency ratings (p = .133). There was no
Clinical experts returned responses via email and were compensated with a small stipend for the time they contributed to this work. 2.6. Data analysis Aim 1: IPE Content Reflected in Interview. Rating means were calculated for each of the four core competencies as well as an overall 96
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Table 1 Competency ratings in core areas (n = 16)a. Competency
Interprofessional Communication Values and Ethics Teams and Teamwork Roles and Responsibilities Overall a
Program
Curriculum
Medicine (n = 8)
Pharmacy (n = 8)
IPE (n = 8)
Non-IPE (n = 8)
2.12 2.21 2.63 2.75 2.47
2.19 1.75 2.19 2.40 2.15
2.67 ± 0.19 2.29 ± 0.69 3.00 ± 0.80 3.45 ± 0.70 2.91 ± 0.49
1.76 ± 0.69† 1.79 ± 0.51 2.00 ± 0.49† 2.60 ± 0.93‡ 1.88 ± 0.21‡
± ± ± ± ±
0.92 0.73 0.75 0.91 0.65
± ± ± ± ±
0.52 0.36 0.86 0.99 0.61
All values expressed as M ± SD. †p < .05; ‡p < .01.
significant difference between Medical and Pharmacy students in any core or overall competency ratings. There were no significant interactions between student program and curriculum in any core or overall competency ratings. Analyses of individual competencies are found in Table 2. Results showed that the IPE students had significantly higher scores for content consistent with: linking team care with improved patient outcomes (C1, p = .05), identifying one's own personal and professional values and how these relate to values and ethics in team care (V-1, p = .003), and explaining specific team processes (T-2, p = .05) but were no different than non-IPE students on other competencies within the domains of Communications, Values, and Teams. In the Roles Domain, the IPE students had higher ratings on all four competencies (R-1, p = 045; R-2, p = .002; R-3, p = .001; R-4, p = .001). Pharmacy and Medical students demonstrated differing reflection of two competencies: Medical students scored significantly higher in discussing health care policy and best practices in the context of values in team care (V-2, p = .039), and describing the flexible application of roles within the healthcare team (R-2, p = .018). Finally, results showed one significant interaction between program and curriculum, indicating that IPE Medical students were able to describe more team processes/infrastructure that allow for better-coordinated care such as team huddles and/or hand-offs (T-2, p = .041) than were IPE Pharmacy students or Non-IPE students. Aim 2: Clinical Experts' Evaluation of Students’ Responses. Interview
transcripts from all 16 students, evenly split between medicine and pharmacy, were rated by 12 clinician experts. Each transcript was rated up to four times (n = 48). IPE students were rated significantly higher by experts in values around IPE (p = .006), values around team based care (p = .02), and knowledge and skills in team based care (p = .007) than were non-IPE students. (Table 3). Expert raters predicted 83.3% of IPE students received specialized IPE training versus 45.8% of non-IPE students (p = .007). With regard to the question, which student would you hire to be part of your team-based primary care practice? Experts ranked IPE students as significantly more preferred (higher, where “1” is most preferred) than non-IPE students (p = .02; Fig. 2). There were no significant differences between Medical and Pharmacy students in terms of expert ratings or rankings, and no significant interactions between program and curriculum (p > .05). 4. Discussion The present study is one of the first to measure the outcomes of IPE for beginning learners in a real-world situation: a mock advanced placement interview, conducted three months after the completion of an 18-month IPE curriculum. Participants were unaware of the true objectives of the study, and presumably answered open-ended questions about their skills/interest and experience in team-based care as they
Table 2 Ratings for each competencya. Competency
Program
Workshop 1: Interprofessional Communication C-1. To describe the relevance of team communication to improved patient safety and outcomes. C-2. To demonstrate essential communication competencies around crucial conversations.
Curriculum
Medicine
Pharmacy
IPE
Non-IPE
2.44 ± 1.13 1.81 ± 0.67
2.43 ± 0.99 1.96 ± 0.91
3.05 ± 0.78 2.29 ± 0.85
1.96 ± 0.99† 1.56 ± 0.53
3.04 ± 1.05
2.90 ± 1.15
3.81 ± 0.88
2.33 ± 0.67‡
2.07 ± 0.95 1.52 ± 1.08
1.14 ± 0.38† 1.19 ± 0.18
1.43 ± 0.63 1.62 ± 1.22
1.85 ± 1.03 1.19 ± 0.18
3.04 ± 0.77
2.38 ± 1.04
3.00 ± 1.07
2.56 ± 0.82
2.22 ± 1.38
2.00 ± 0.77
3.00 ± 1.12
1.44 ± 0.50†§
3.26 ± 0.78 2.96 ± 1.10 2.44 ± 1.34 2.33 ± 1.07
2.76 ± 0.46 2.00 ± 0.92† 2.29 ± 1.58 2.57 ± 1.37
3.43 3.38 3.57 3.43
2.74 1.89 1.44 1.67
Workshop 2: Values and Ethics V-1. To identify one's own personal and professional values and how these relate to values and ethics in team care. V-2. To discuss health care policy and best practices in the context of values in team care. V-3/4. To explain what social determinants of health are, why they are part of ethical patient-centered care, and how to address them as a team. Workshop 3: Teams and Teamwork T-1. To discuss aspects to team work in any setting (even non-clinical) that impact team functioning and outcomes. T-2. To describe team processes/infrastructure that allow for better-coordinated care such as team huddles and/or hand-offs. Workshop 4: Roles and Responsibilities R-1. To discuss various team members' roles and training. R-2. Describe and discuss the flexible application of roles within the healthcare team. R-3. Discuss leadership as a team role (guild vs. team). R-4. Practice the skill of articulating your role as a member of the healthcare team. a
All values expressed as M ± SD. †p < .05; ‡p < .01; §significant interaction, p < .05. 97
± ± ± ±
0.71 0.91 1.15 0.99
± ± ± ±
0.52† 0.76‡ 0.69‡ 0.55‡
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Table 3 Clinical experts evaluation of students’ responsesa. Question
Program
Values around IPE Values around team based care Knowledge and skills in team based care a
Curriculum
Medicine
Pharmacy
IPE
Non-IPE
3.42 ± 1.32 3.67 ± 1.05 3.38 ± 1.01
3.42 ± 1.25 3.46 ± 1.06 3.33 ± 1.05
3.92 ± 1.02 3.92 ± 1.06 3.75 ± 0.94
2.92 ± 1.32‡ 3.21 ± 0.93† 2.96 ± 0.95‡
All values expressed as M ± SD. †p < .05; ‡p < .01.
might in an actual interview.
from non-IPE peers who did not mention the content at all or only in vague terms. The lack of specificity in their answers could be due to the lapse in time between this interview and the IPE workshops, which were held three months (Roles), seven months (Teams), fifteen months (Values), and eighteen months (Communications) prior. The time lapse may also explain why IPE students scored higher on all Roles competencies than their non-IPE peers: it was the most recent content taught. Another explanation for more generalized responding is that, while the curricula is quite targeted in terms of the specific competencies, there are no tests or other opportunities to rehearse content to fluency or commit to memory. Moreover, because these were beginning learners, they did not have exposure to clinical contexts in which they might observe varied examples of the targeted material. It is important to consider the developmental sequela of IPE competencies, understanding that in order to develop very specific skills of fluency, students will need practice and exposure to wide-ranging examples in practice environments. One observation made was that, in many cases, IPE students discussed content they had learned in the curriculum in response to the interviewer's first and most general question: “What interests you about this position and what unique skills or training do you bring to us?” Thus, after two years of coursework in medicine or pharmacy, IPE students prioritized content learned in the IPE training workshops above other discipline-specific content when listing their own selfperceived assets. It seemed they had assimilated the content into their professional identity and/or understood that the IPE training was unique and gave them an advantage in an interview. That both groups conveyed the value of team based care may reflect that fact that ETSU has developed a culture that supports both interprofessional practice and education in classroom and clinic settings. Indeed both groups of students were exposed to the topic of interprofessional communication in a required introductory course called
4.1. IPE students talked about IPE competencies Regarding Aim 1, results showed that the transcripts of medical and pharmacy students who participated in ETSU's IPE curricula contained significantly more competency-related content than did non-IPE students. Specifically, IPE students scored higher than non-IPE students on seven out of the eleven competencies addressed in the curriculum. Thus, the IPE students were able to incorporate curriculum-based knowledge into their answers regarding why they are a good fit for the position and their experiences in team based care. These findings reflect16 Levels 3 (change in knowledge) and 4 (change in behavior), a shift in rigor over prior studies. Examples of language reflecting competency knowledge include: “(team care) provides more complete care for the patient and I think results from studies show that it provides the best care for patients,” and “… instead of the physician does this, the pharm student does this, the nursing student does this and that, we were supposed to integrate with each other and try to say, ‘hey what do you know that's different from what I learned in my school … ’ like try to apply different … perspectives to the same patient's problem. What we found is that this can actually make a pretty big difference in care.” Such content was not found, or was less specific, in non-IPE transcripts. It is important to note that while IPE students consistently scored higher than non-IPE students, their average scores in each domain were just above the mid-point on a 5 point scale (ranged from 3.75 to 3.92). Anchors for this scale were: 3 = “student mentioned this competency or related content but did not elaborate” and 5 = “student provided data, definitions, specific skills or knowledge on this topic.” Thus, while IPE students were articulating content learned, they did not appear to use the detail or specificity with which it was taught. Rather, they seemed to talk about the content in general terms and differentiated themselves 50.0%
Percent of Students Ranked
45.0% 40.0%
IPE Non-IPE
35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0%
First
Second
Third
Fourth
Ranking (First is Most Preferrable) Fig. 2. Expert clinicians' rankings of students by preference to hire for team based care. 98
Journal of Interprofessional Education & Practice 15 (2019) 94–99
J. Polaha, et al.
Communication Skills for Health Professionals, which includes students from nursing, psychology, pharmacy and medicine. That said, overall scores for the Values and Ethics domain were the lowest out of all the of the core competency areas. It could be that our curricula lacks traction in this area, or that the interview did not elicit content from this domain. Alternatively, it could be that this core area is the most intangible, both to teach and to measure, a concept that has long been a challenge in the social sciences.18
competency raters who were familiar with the curriculum and could distinguish IPE from non-IPE students by transcript answers, although transcripts were deidentified. 5. Conclusions The results of this study showed that students who participated in an IPE curriculum provided answers in a mock job interview that were seen as preferable by team-based clinicians. This study advanced the rigor evaluation of foundational IPE curricula by targeting behavior (interview content) rather than self-report. Moreover, it is one of the first to connect foundational competencies with what is valued by clinicians in real-world team-based care. Future research should continue to develop sophisticated methods to demonstrate the connection between the classroom/theory based learning of foundational learners and meaningful outcomes across the training trajectory.
4.2. IPE students were judged more positively In addition to documenting competencies, the present study demonstrates the validity of the IPE content learned by asking team care experts from pharmacy, medicine, and psychology to rate and rank the students. Results from Aim 2 showed that clinical experts, unaware of the true purpose of the study or that the sample included “two groups,” rated the IPE students higher in terms of their knowledge of team care, values around team care, and values for IPE. They were able to discriminate which students had IPE from those who had not. When ranking the students in terms of preference to work in team care, they consistently ranked the IPE students as more preferable. The significance of these findings is underscored by the charge to align learning outcomes with the skills and knowledge needed for later application in health care settings.19,20 Evidence that these clinicians were unaware of the purpose of the study and susceptible to participant bias/social desirability is found in their responses to a follow-up question asking them what they thought the purpose to be. None of them guessed that the study included two groups. Rather, responses were “to find out what values or skills are important for professionals in different disciplines,” “how well the students are learning team-based care,” or “different forms of teambased training and education.” One clinician summed up her experience of participating with the comment that she preferred students whose transcript showed they “were able to talk in a way that showed she had actually internalized, valued, and accepted the essence of team based care … instead of focusing on her clinical skills, she focused more on the key ingredients to team based work; likeability, flexibility, openness, enthusiasm, willingness to adapt/work/learn/relearn, communicate easily and effectively … you can train the other skills much more easily than you can help someone become a fluid part of a working team.” Taken together, the results of this study show that ETSU's foundational IPE program effectively teaches key core competencies around team care, at least to the point of students' incorporating that knowledge into their interview answers. Considering learning outcomes across the entire trajectory of training is important in IPE evaluation.19 One interpretation of these findings is that our foundational learners have learned to articulate relatively theoretical and classroom-based IPE material in a meaningful way, and that they value this knowledge. Future research should evaluate how that skill and knowledge base translates to their clinic training experiences in their later in the curriculum. Are they perceived as more prepared by members of the clinic team? Does that preparation translate to accelerated uptake of teambased skills in the clinic setting? Are students who had foundational IPE still discriminable from non-IPE students after a team-based clinic training experience? These questions, as well as questions about their performance further into the future, are all critical questions for further research.
Acknowledgements The authors wish to acknowledge the contributions of the following: Sarah Tolliver (assistance with data entry and analysis); Wendy Guinn (assistance with data collection), Dr. Tese Stephens (assistance with data collection and early conceptualization); and Marty Woodward, who performed the role of the interviewer for all participants. In addition, we would like to thank Dr. Wilsie Bishop, Vice President for Health Affairs, who has championed IPE at ETSU and who funded this study through the Academic Health Sciences Center. References 1. Beacham AO, Kinman C, Harris JG, Masters KS. The patient-centered medical home: unprecedented workforce growth potential for professional psychology. Prof Psychol Res Pract. 2012;43(1):17–23. 2. Kwan BM, Nease DE. The state of the evidence for integrated behavioral health in primary care. In: Talen MR, Valeras AB, eds. Integrated Behavioral Health in Primary Care. New York, NY: Springer; 2013:65–98. 3. Working Party Group on Integrated Behavioral Healthcare. The development of joint principles: integrating behavioral health care into the patient-centered medical home. Ann Fam Med. 2014;12(2):183. 4. Zeind CS, McCloskey WW. Pharmacists' role in the health care setting. Harvard Health Policy Review. 2006;7:147–154. 5. Zorek J, Raehl C. Interprofessional education accreditation standards in the USA: a comparative analysis. J Interprofessional Care. 2013;27(2):123–130. 6. Darlow B, Coleman K, McKinlay E, Donovan S, et al. The positive impact of interprofessional education: a controlled trial to evaluate a programme for health professional students. BMC Med Educ. 2015;15:98. 8. Nurok ML, Evans LA, Lipsitz S, Satwicz P, Kelly A, Frankel A. The relationship of the emotional climate of work and threat to patient outcome in a high-volume thoracis surgery operating room team. BMJ Qual Saf. 2011;20(3):237–242. 10. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013(3) https://doi.org/10.1002/14651858.CD002213.pub3 Art. No.: CD002213. 11. Blue AV, Chesluk BJ, Conforti LN, Holmboe ES. Assessment and evaluation in interprofessional education: exploring the field. J Allied Health. 2015;44(2):73–82. 12. IOM (Institute of Medicine). Measuring the Impact of Inteprofessional Education on Collaborative Practice and Patient Outcomes. Washington, DC: The National Academies Press; 2015. 14. Abu-Rish E, et al. Current trends in interprofessional education of health sciences students: a literature review. J Interprofessional Care. 2012;26:444–451. 16. Barr H, Koppel I, Reeves S, Hammick M, Freeth D. Effective interprofessional education: arguments, assumptions and avidence. In: Meads G, Ashcroft J, Barr H, Scott R, Wild A, eds. The Case for Interprofessional Collaboration. Oxford: Blackwell Publishing Ltd.; 2005. 17. Shrader S, Zaudke JK. Preceptors in the Nexus Toolkit. 2015; 2015 Retrieved from https://nexusipe.org/engaging/learning-system/preceptors-nexus-toolkit. 18. Wong G, Greenhalgh T, Westhorp G, Pawson R. Realist methods in medical education research: what are they and what can they contribute? Med Educ. 2012;26:89–96. 19. Anderson E, Smith R, Hammick M. Evaluating an interprofessional education curriculum: a theory-informed approach. Med Teach. 2016;38:385–394. 20. Thistlewaite JE, Moran M. Learning outcomes for interprofessioanl education (IPE): literature review and synthesis. J Interprofessional Care. 2010;24:503–513.
4.3. Limitations The small number of participants in this study and its evaluation of only one (ETSU) IPE training program limit the generalizability of findings. Another limitation is a potential lack of objectivity in
99