Learning crisis resource management: Practicing versus an observational role in simulation training – a randomized controlled trial

Accepted Manuscript Title: Learning Crisis Resource Management: Practicing versus an Observational Role in Simulation Training – A Randomized Controlled Trial Authors: Anita Lai MD, FRCPC Alexis Haligua MD, FRCPC M. Dylan Bould MB ChB, MEd Tobias Everett MD, FRCA Mark Gale MD, FRCPC Ashlee-Ann Pigford MSc Sylvain Boet MD, PhD PII: DOI: Reference:

S2352-5568(16)30004-2 http://dx.doi.org/doi:10.1016/j.accpm.2015.10.010 ACCPM 125

To appear in: Received date: Revised date: Accepted date:

25-3-2015 15-10-2015 19-10-2015

Please cite this article as: Anita LaiAlexis HaliguaM. Dylan BouldTobias EverettMark GaleAshlee-Ann PigfordSylvain Boet Learning Crisis Resource Management: Practicing versus an Observational Role in Simulation Training – A Randomized Controlled Trial (2016), http://dx.doi.org/10.1016/j.accpm.2015.10.010 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

*Title page

Title: Learning Crisis Resource Management: Practicing versus an Observational Role in Simulation Training – A Randomized Controlled Trial

ip t

Author(s): Anita Lai, MD, FRCPC1, Alexis Haligua, MD, FRCPC2, M Dylan Bould MB ChB, MEd3,4, Tobias Everett, MD, FRCA5, Mark Gale, MD, FRCPC6, Ashlee-Ann Pigford MSc7, Sylvain Boet MD, PhD3,7

te

d

M

an

us

cr

Affiliations: 1. Department of Emergency Medicine, Alberta Health Services, University of Calgary, Room C231, 1403 – 29 Street NW, Calgary, AB T2N 2T9, Canada 2. Department of Emergency Medicine, The Montfort Hospital, University of Ottawa, 713, Montreal Road Ottawa, ON, K1K 0T2, Canada 3. The Department of Innovation in Medical Education of the Faculty of Medicine & The University of Ottawa Skills and Simulation Centre, University of Ottawa, Roger Guidon Hall Room 2211, 451 Smyth Rd. Ottawa, ON, K1H 85M, Canada 4. Department of Anesthesiology, The Children’s Hospital of Eastern Ontario, University of Ottawa, 401 Smyth Road, Ottawa, ON, K1H 8L1, Canada 5. Department of Anesthesiology, The Hospital for Sick Children, University of Toronto, 555 University Avenue, Room 2303 Toronto, ON M5G 1X8, Canada 6. Department of Anesthesiology, Alberta Children’s Hospital, University of Calgary, 2888 Shaganappi Trail NW Calgary, AB, T3B 6A8, Canada 7. Department of Anesthesiology & Department of Innovation in Medical Education, The Ottawa Hospital, University of Ottawa, The Ottawa Hospital Civic Campus, 1053 Carling Ave., Box 106, Ottawa, ON, K1Y 4E9, Canada

Ac ce p

Corresponding Author: Dr. Sylvain Boet Department of Anesthesiology & Department of Innovation in Medical Education The Ottawa Hospital, 1053 Carling Ave., Box 106, Ottawa, ON, K1Y 4E9, Canada Phone: (+1) 613-737-8187 Fax: (+1) 613-737-8189 Email: [email protected] Sources of Support: Royal College of Physician and Surgeons of Canada - Medical Education Research Grant University of Ottawa - Educational Initiatives in Residency Education Grant Word Count: 238 (abstract); 2649 (paper)

Page 1 of 26

*Manuscript (no author information)

Learning Crisis Resource Management: Practicing versus an Observational Role in Simulation Training – A Randomized Controlled Trial

ip t

Anita Lai, MD, FRCPC1, Alexis Haligua, MD, FRCPC2, M Dylan Bould MB ChB, MEd3,4, Tobias Everett, MD, FRCA5, Mark Gale, MD, FRCPC6, Ashlee-Ann Pigford MSc7, Sylvain Boet MD, PhD3,7

te

d

M

an

us

cr

Affiliations: 1. Department of Emergency Medicine, Alberta Health Services, University of Calgary, Room C231, 1403 – 29 Street NW, Calgary, AB T2N 2T9, Canada 2. Department of Emergency Medicine, The Montfort Hospital, University of Ottawa, 713, Montreal Road Ottawa, ON, K1K 0T2, Canada 3. The Department of Innovation in Medical Education of the Faculty of Medicine & The University of Ottawa Skills and Simulation Centre, University of Ottawa, Roger Guidon Hall Room 2211, 451 Smyth Rd. Ottawa, ON, K1H 85M, Canada 4. Department of Anesthesiology, The Children’s Hospital of Eastern Ontario, University of Ottawa, 401 Smyth Road, Ottawa, ON, K1H 8L1, Canada 5. Department of Anesthesiology, The Hospital for Sick Children, University of Toronto, 555 University Avenue, Room 2303 Toronto, ON M5G 1X8, Canada 6. Department of Anesthesiology, Alberta Children’s Hospital, University of Calgary, 2888 Shaganappi Trail NW Calgary, AB, T3B 6A8, Canada 7. Department of Anesthesiology & Department of Innovation in Medical Education, The Ottawa Hospital, University of Ottawa, The Ottawa Hospital Civic Campus, 1053 Carling Ave., Box 106, Ottawa, ON, K1Y 4E9, Canada Corresponding Author: Dr. Sylvain Boet Department of Anesthesiology & Department of Innovation in Medical Education The Ottawa Hospital, 1053 Carling Ave., Box 106, Ottawa, ON, K1Y 4E9, Canada

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Phone: (+1) 613-737-8187 Fax: (+1) 613-737-8189 Email: [email protected]

Sources of Support: Royal College of Physician and Surgeons of Canada - Medical Education Research Grant University of Ottawa - Educational Initiatives in Residency Education Grant

1 Page 2 of 26

Abstract Aim: Simulation training has been shown to be an effective way to teach crisis resource management (CRM) skills. Deliberate practice theory states that learners need to actively

ip t

practice so that learning is effective. However, many residency programs have limited

opportunities for learners to be “active” participants in simulation exercises. This study compares

us

observer participant in simulation followed by a debriefing.

cr

the effectiveness of learning CRM skills when being an active participant versus being an

Methods: Participants were randomized to two groups: active or observer. Active participants

an

managed a simulated crisis scenario (pre-test) while paired observer participants viewed the scenario via video transmission. Then, a trained instructor debriefed participants on CRM

M

principles. On the same day, each participant individually managed another simulated crisis

d

scenario (post-test) and completed a post-test questionnaire. Two independent, blinded raters

te

evaluated all videos using the Ottawa Global Rating Scale (GRS). Results: Thirty-nine residents were included in the analysis. Normally distributed data were analyzed using paired and unpaired t-tests. Inter-rater reliability was 0.64. Active participants

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

significantly improved from pre-test to post-test (p=0.015). There was no significant difference between the post-test performance of active participants compared to observer participants (p=0.12).

Conclusion: We found that learning CRM principles was not superior when learners were active participants compared to being observers followed by debriefing. These findings challenge the deliberate practice theory claiming that learning requires active practice. Assigning residents as observers in simulation training and involving them in debriefing is still beneficial. Key words: Crisis Resource Management; Debriefing; Inter-professional care; Teamwork; Communication; Dyad Learning

2 Page 3 of 26

1. Introduction

1.1 Literature Foundation

ip t

Acute care, including emergency medicine, critical care, and anaesthesiology, is a dynamic

environment where physicians can be exposed to patient crises at any time. Non-technical skills

cr

for crisis resource management (CRM) such as task management, teamwork, situational

us

awareness and decision-making are crucial to ensure patient safety when managing crises.[1] Over recent years, high-fidelity simulation training has been demonstrated to be an effective tool

M

an

in learning CRM, and to be more effective than didactic teaching.[2-5] In addition to increased

d

knowledge and skills,[3, 6] learning CRM in the simulator appears to transfer to patient care.[4]

te

Simulation-based education is supported by the theory of deliberate practice which states that learners need to actively practice in order to be effective.[7]

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Due to the financial costs of simulation sessions, limited time availabilities and an increasing demand for simulation training, there may be limited opportunities for each learner to be the active participant in simulated scenarios. As a result, simulation instructors often designate an active participant who is given the opportunity to practice their clinical skills in a simulated scenario while other trainees are assigned to an observational role outside the simulation room.

The importance of actively participating is emphasized in both Kolb’s experiential learning cycle[8], which describes the importance of experimentation through actively participating and

3 Page 4 of 26

then reflecting on the experience and Ericsson’s theory of deliberate practice[7] which describes the importance of actively participating and obtaining feedback to become expert in a field. Following these theories, one hypothesizes that learning by actively participating would allow

ip t

for more effective learning than being simply an observer, but this is unproven.[7] Currently, we are unaware if observer participants improve their skills at the same rate as their active

cr

colleagues. In order for educators to best prioritize resource allocation in simulation, it is

us

imperative to gain a better understanding of the importance of actively practicing on learning CRM in simulation training. Once this issue has been better addressed, educators will be better

M

observational roles while in simulation sessions.

an

equipped to make best practice decisions regarding scheduling trainees for active versus

d

1.2 Study objective

te

The aim of this study was to compare the effectiveness of learning CRM principles when being an active participant in simulation-based education versus being an observer participant. We hypothesized that active participants would improve their CRM skills more than observer

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

participants.

4 Page 5 of 26

2. Materials and Methods 2.1 Participants The Ottawa Hospital Research Ethics Board granted approval to this study (20120008-01H). All

ip t

emergency medicine residents in postgraduate years 1–5 in both the Royal College of Physicians and Surgeons and College of Family Physician programs at the University of Ottawa were

cr

invited to participate in the study. The decision to participate was voluntary and had no impact

us

on residents program. Prior to acceptance into an emergency medicine residency program in Ottawa, Advanced Cardiac Life Support (ACLS) certification is required; therefore all the

an

participants were ACLS certified. For the past five years, a standardized curriculum of simulation training in emergency medicine has been the norm in Ottawa. On average, each

M

resident is assigned two simulation sessions per academic year. Informed consent and

d

confidentiality agreements were obtained for each participant to prevent details pertaining to the

te

clinical scenarios from being disseminated before the end of the study.

2.2 Study Design and Intervention

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

For this prospective randomized controlled study (ClinicalTrials.gov ID: NCT01653704), participants were randomized to one of two groups: active group or observer group (Figure 1). The study took place at The University of Ottawa Skills and Simulation Centre of The Ottawa Hospital and The University of Ottawa. All participants completed a demographic questionnaire. Assignment of each participant into each group was done through computer randomization. Each participant of the active group was randomly paired with one of the observer group participants. Active group participants individually managed a simulated crisis scenario (pre-test) while the paired observer participant observed the scenario from outside of the simulation room using a

5 Page 6 of 26

video transmission system. Immediately afterwards, both participants were debriefed together on CRM by a trained instructor using an advocacy-inquiry model.[9] The instructor reviewed video footage from the simulation scenario and both participants were encouraged to actively

ip t

participate in the debriefing. The objective of the debriefing was to discuss and reflect on

resident’s performance. The debriefing was guided by the concept of debriefing with good

cr

judgment that includes an advocacy-inquiry model.[9] Following the debriefing, both the active

us

and observer participant individually managed a simulated crisis scenario (post-test). The posttest occurred on the same day as the pre-test. The perceived usefulness of learning CRM and the

an

perceived stress of the two scenarios (pre-test and post-test) were collected in a post-scenario survey using a six point scale. Participants independently completed a survey, which consisted of

M

a 7 point Likert scale ranging from not comfortable/useful to extremely comfortable/useful after

d

the completion of each scenario (Appendix A). Two independent raters, blinded to the

te

participating resident’s experience and training levels and unaware of the research question, rated CRM performances of all videos in a random order using the Ottawa Global Rating Scale (GRS).[10]

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Both pre- and post-tests scenarios involved the application of CRM principles and were piloted to ensure that they were of equal difficulty (Appendix B). Simulation faculty, with experience in emergency medicine, approved the scenarios for content and also to further ensure equality of level of difficulty. Two confederates played scripted roles for one nurse and one resident. The confederates were instructed to perform tasks when directed but not to give advice on the management of the simulated scenario. Each scenario was ten minutes long and sealed envelopes opened by the investigators prior to each simulation session decided the scenario order. Each

6 Page 7 of 26

scenario was video recorded with multi-screen video, including the simulated patient’s vital signs. The SimMan 3G (Norway, Laerdal) was used for the entire study. The participants were

ip t

expected to properly and effectively manage the crises by utilizing CRM principles.

2.3 Measurement Instruments and Outcomes

cr

The Ottawa GRS scale is a reliable and valid scale used to measure CRM performance in high

us

fidelity simulation previously used in several studies.[10, 11] The scale assesses five main CRM categories: leadership, problem solving, situational awareness, resource allocation, and

an

communication skills. In addition to these five categories, there is a category evaluating the overall performance of the participant. Each category is measured on a 7 point Likert scale with

M

descriptive anchors to provide guidelines on alternating points along the scale. The categorical

2.4 Statistical Analysis

te

performance).[10]

d

scores are summed to give a total score of 5 (poor performance) to 35 (high level

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Statistical analysis was performed using SPSS 17.0 (SPSS, Chicago, IL). A two-tailed p value of .05 was considered significant for all analyses. Questionnaire data were analyzed using the chisquare and Mann-Whitney test as appropriate.[12] Inter-rater reliability was assessed using the intra-class correlation coefficient for the total Ottawa GRS score.

Our primary outcome was the performance of participants of the two groups in their post-test (total Ottawa GRS score). Our secondary outcomes were the change in performance between their pre and post-tests of the active participants (total Ottawa GRS score) and the comparison of

7 Page 8 of 26

the post-test performance of the active versus observer groups at the Ottawa GRS score category levels. Although the individual Likert scales that make up the GRS scores are ordinal in nature, it has been suggested that these scores behave empirically as a parametric variable.[13] Following

ip t

a Kolmogorov–Smirnov test to assess normality of data, the outcomes were analyzed using a

cr

paired and unpaired t-test as appropriate. Results are expressed with mean ± SD.

us

Sample size was calculated based on our primary outcome of the post-test performance comparison using the G*Power software (version 3.12, Düsseldorf, Germany).[14] Since we

an

have no previous data on CRM performance of emergency residents measured by the Ottawa GRS score (expected mean and SD), we elected to base our sample size calculation on the

M

expected effect size (Cohen's d). Because the cost required to make all learners active

d

participants in simulation is substantial, we used an expected effect size of >1.0 as large gains in

te

learning would better justify this extra cost. Therefore, assuming an effect size of 1.0 and a power of 0.8, we calculated a total sample size of 34 subjects ( = .05 two-tailed). Allowing for technical failures, we planned to recruit 20 subjects per group.

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

8 Page 9 of 26

3. Results 3.1 Demographics Forty-two participants enrolled in the study and forty participants completed the study (Figure 2).

ip t

Thirty-nine participants were included in the final analysis as one outlier from the active group was excluded. This participant was a first year resident and scored exceptionally high on one of

cr

the scenarios compared to the other and this was unexpected based on his level of training.

us

During the debriefing, the participant revealed that he received teaching on the topic of the scenario just prior to entering the study thus we thought that his performance was not reflective

an

of his usual baseline performance and did not help us answer our study question. The male:female ratio of the participants was 22:18. The mean age was 29.9 with a range from 26 to

d

3.2 Inter-Rater Reliability

M

43 years old.

te

The overall inter-rater reliability for the total GRS score level was substantial: ICC=0.64 (p<0.001). Due to the agreement between expert raters, we elected to use the mean score as the

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

expert score for analysis.

3.3 Primary and Secondary Outcomes Our primary outcome was the performance of participants of the two groups in their post-test (total Ottawa GRS score). There was no significant difference between the performance in active compared to observer participants in their total Ottawa GRS post-test scores (24.3±5.4 vs. 21.8±4.5; p=0.22). Active participants significantly improved from pre-test to post-test in their total Ottawa GRS scores (20.7±4.9 vs. 24.3±5.4; p=0.015; Cohens d=0.50) (Figure 3).

9 Page 10 of 26

Results for the 5 specific categories of the Ottawa GRS scale and the overall item are reported in Table 1. There was no difference in the performance of the participants in the active group compared to those in the observer group for their post-tests at the level of each category of non-

ip t

technical skills from CRM. There was no significant difference between the overall performance

cr

in active compared to observer participants in their post-test (4.7±1.2 vs. 4.1±1.2; p=0.16).

us

There were no differences in comfort level with the assigned role (p=0.35), how useful participants found the first scenario for CRM learning (p=0.29), how useful participants found

an

the second scenario for CRM learning (p=0.67) between groups, and if participants thought the scenario was helpful in learning CRM (p=0.32). The active participants perceived the first

M

scenario to be more stressful than the observer participants (p=0.005) with the majority of

d

observer participants rating themselves as “not stressed.” However, for the second scenario

(p=0.83).

te

where both groups were ‘active’ there was no perceived difference in stress level between groups

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

10 Page 11 of 26

4. Discussion This study confirms that active participants improve their CRM skills from participating in a scenario and debriefing afterwards. The effectiveness of being an active participant in a

ip t

simulated crisis scenario followed by a debriefing is in agreement with previous literature.[2, 6, 15, 16] Our study adds to the existing literature demonstrating that learning as an active

cr

participant followed by a debriefing is not largely superior to being an observer participant when

us

followed by a debriefing.

an

There are several possible explanations for our findings. Learning by observation may be supported by the presence of mirror neurons in the brain, discovered by a group of researchers at

M

the University of Parma in the 1990s, which are a subset of neurons that respond equally when

d

an action is performed and when an action is observed.[17] Mirror neurons have been considered

te

key in post-stroke rehabilitation using mental practice and mental imagery. [18, 19] Mental practice is defined as the ‘cognitive rehearsal of a skill in the absence of an overt physical movement’, which is a type of self-observation.[20-22] However, a recent study showed that

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

mental practice does not seem to improve CRM skills in simulated crisis scenarios.[11] Therefore, reflection that occurs during debriefing might be the crucial element for learning and the type of experience, active or observer, may be of secondary importance. Although learning by observation may be effective due to these neurons, our findings do challenge previous learning theories in simulation.

Ericsson’s educational theory of deliberate practice[7] and Kolb’s experiential learning cycle[8] both emphasize the importance of active participation for learning in simulation. Kolb’s

11 Page 12 of 26

experiential learning cycle describes learning through concrete experimentation; the process of actively practicing and then reflecting on this practice to conceptualize new concepts to apply to practice.[8] Our finding that learning is achievable through observing challenges this theory of

ip t

the necessity of active participation, and may broaden the means by which ‘concrete experimentation’ is obtained with respect to non-technical skills. As Gardner wrote recently, the

cr

“sentiment shared among simulation educators is that simulation is a good excuse to

us

debrief”.[23] On the same line, we would specify that “simulation” could represent either an active involvement or an observation of a simulation scenario. The key element to learning non-

an

technical skills for CRM in simulation may be in the act of reflection through debriefing and not particularly in actively participating. It is important to note that both groups in our study had an

M

experience (active or observer) that debriefing could build on. One cannot extrapolate the

d

learning effect of no experience at all and simply reflecting during a debriefing. Practically

te

speaking, assigning learners as observer participants may be useful for learning CRM.

The majority of literature in simulation considers the participant in the active role. A multitude of

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

studies on topics varying from the retention of procedural skills to the relative effectiveness of different debriefing techniques have been performed, all of which consider the trainee as an active participant in the simulation session and draw conclusions on the proposed learning interventions based on that structure.[2, 13, 16, 24] In the current reality of limited resources, not all participants are afforded the opportunity to actively participate. Our findings validate the current and widely practiced simulation model, where some participants are active and others are simply observers, before reflecting together during debriefing. Shanks et al found that learning procedural skills as pairs, called dyads, may be as effective as self-directed learning, suggesting

12 Page 13 of 26

that it may be useful to schedule more learners per simulation session to optimize resources and learning.[25] Furthermore, our findings may be encouraging for people who naturally shy away from participating in simulation training due to personal preference, and my provide them with a

ip t

more comfortable method of learning from simulation. Additional research to examine retention of technical and non-technical skills in the active versus observer participants may also be

us

cr

necessary.

There were a number of limitations to the study. First, the research question itself prevented us

an

from assessing the pre-test performance of the observer group. Therefore, we had to assume that there was a non-significant difference in the baseline level in crisis management between the two

M

groups. Furthermore, the study was not designed to quantify the long-term impact of the active

d

group being exposed to two hands-on scenarios compared to the observer group, which only had

te

one hands-on experience. All of the participants enrolled were emergency medicine residents and thus potentially a homogenous study population. Another limitation was that the sample size was calculated based on a large effect size. Although our study did not find a statistically significant

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

benefit to being in the active group, the observed benefit corresponds to a Cohen’s d of 0.5. Thus, while we have ruled out a large effect, we cannot rule out the possibility that there may have been a moderate benefit for the active arm. This is important, because if our findings are considered from a patient safety lens even a moderate improvement in performance could be crucial, making it important consider small differences. From a simulation education perspective this effect size, may be perceived to be significant; however, we caution this interpretation because the study was not powered to detect a small or moderate effect size. Since simulationbased education is resource intensive one may require any teaching intervention to have a large

13 Page 14 of 26

effect size to justify cost associated with it. Larger studies including different specialties and disciplines may be required to determine the generalizability of results to other health professionals and the relative effectiveness of being active versus observer participants in

ip t

simulation education. Further studies may be useful to study the optimal number of observers

te

d

M

an

us

cr

and active participants per session.

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

14 Page 15 of 26

5. Conclusion We found that learning CRM principles was not largely superior when learners were active participants in a simulated scenario compared to being observers followed by debriefing. These

ip t

findings challenge the experiential learning theory claiming that learning non-technical CRM skills requires active practice. Debriefing, potentially via reflection, after either practice or

cr

observation, seems to be the main element for learning. In times of limited resources, assigning

us

residents as observers in simulation training and involving them in debriefing is still beneficial. This study provides a foundation for an evidence-based approach to scheduling trainees for

an

simulation education.

d

M

Conflict of Interest: The authors have no conflicts of interest to declare.

te

Acknowledgements: This project was supported in part by a Medical Education Research Grant from the Royal College of Physician and Surgeon of Canada and the Educational Initiatives in Residency Education Grant from the University of Ottawa. It was also supported by the

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Anesthesiology Department of the University of Ottawa. The authors wish to thank: Sue Humphrey Murto for her help defining the research question and improving the methods; Timothy Wood for his help defining the sample size, improving the methods and answering statistical queries; Sean Moore and Brian Weitzman for re-organizing resident schedules to participate in the study; and Stan Hamstra for his assistance in defining the research question and assisting in the statistical analysis.

15 Page 16 of 26

References:

te

d

M

an

us

cr

ip t

[1] Gaba DM FK, Howard SK. Crisis Management in Anesthesiology. Philidelphia: Churchill Livingstone; 1994. [2] Boet S, Bould MD, Bruppacher HR, Desjardins F, Chandra DB, Naik VN. Looking in the mirror: Self-debriefing versus instructor debriefing for simulated crises. Crit Care Med. 2011;39:1377-81. [3] Yee B, Naik VN, Joo HS, Savoldelli GL, Chung DY, Houston PL, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology. 2005;103:241-8. [4] Bruppacher HR, Alam SK, LeBlanc VR, Latter D, Naik VN, Savoldelli GL, et al. Simulation-based training improves physicians' performance in patient care in high-stakes clinical setting of cardiac surgery. Anesthesiology. 2010;112:985-92. [5] Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42:128-33. [6] Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technologyenhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306:978-88. [7] Ericsson K. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med [Review] 2004;79(10 Suppl):S70-81. [8] Kolb DA. Experiential learning: Experience as the source of learning and development: Prentice-Hall Englewood Cliffs, NJ; 1984. [9] Rudolph JW, Simon R, Dufresne RL, Raemer DB. There's no such thing as" nonjudgmental" debriefing: a theory and method for debriefing with good judgment. Simul Healthc 2006;1:49-55. [10] Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med. 2006;34:2167-74. [11] Hayter MA, Bould MD, Afsari M, Riem N, Chiu M, Boet S. Does warm-up using mental practice improve crisis resource management performance? A simulation study. Br J Anaesth. 2012;110:299-304. [12] Erdfelder E, Faul F, Buchner, A. GPOWER A general power analysis program. Behavior Research Methods, Instruments & Computers 1996;28:1-11. [13] Boet S, Borges BC, Naik VN, Siu LW, Riem N, Chandra D, et al. Complex procedural skills are retained for a minimum of 1 yr after a single high-fidelity simulation training session. Br J Anaesth. 2011;107:533-9. Epub 2011/06/11. [14] Faul F, Erdfelder E, Lang A-G, Buchner A. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007;39:175-91. [15] Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. Med Teach. 2013;35:e844-75. Epub 2012/09/04.

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

16 Page 17 of 26

te

d

M

an

us

cr

ip t

[16] Savoldelli GL, Naik VN, Park J, Joo HS, Chow R, Hamstra SJ. Value of debriefing during simulated crisis management: oral versus video-assisted oral feedback. Anesthesiology. 2006;105:279-85. [17] Gallese V, Fadiga L, Fogassi L, Rizzolatti G. Action recognition in the premotor cortex. Brain. 1996;119:593-609. [18] Müller K, Bütefisch C, Seitz RJ, Hömberg V. Mental practice improves hand function after hemiparetic stroke. Restor Neurol Neurosci 2007;25:501-11. [19] Garrison KA, Winstein CJ, Aziz-Zadeh L. The mirror neuron system: a neural substrate for methods in stroke rehabilitation. Neurorehabilitation and neural repair. 2010;24(5:404-12. [20] Driskell JE, Copper C, Moran A. Does mental practice enhance performance? J Applied Psychology. 1994;79:481. [21] Weinberg R. Does imagery work? Effects on performance and mental skills. J Imagery Res Sport Physical Activity. 2008;3(1). [22] Epstein ML. The relationship of mental imagery and mental rehearsal to performance of a motor task. J Sport Psychol . 1980;2:1-220. [23] Gardner R, editor. Introduction to debriefing. Seminars in perinatology; 2013: Elsevier. [24] Welke TM, LeBlanc VR, Savoldelli GL, Joo HS, Chandra DB, Crabtree NA, et al. Personalized Oral Debriefing Versus Standardized Multimedia Instruction After Patient Crisis Simulation. Anesth Analg. 2009;109:183-9. [25] Shanks D, Brydges R, den Brok W, Nair P, Hatala R. Are two heads better than one? Comparing dyad and self‐regulated learning in simulation training. Med Educ. 2013;47:1215-22.

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

17 Page 18 of 26

Active (Mean ± SD)

Observer (Mean ± SD)

Leadership

5.2 ± 1.2

4.9 ± 0.9

Problem solving

4.3 ± 1.3

3.7 ± 1.4

Situational awareness

4.6 ± 1.4

3.6 ± 1.7

0.07

Resource utilization

4.9 ± 1.1

4.4 ± 0.9

0.14

Communication

5.4 ± 1.1

us

Table 1: Post-Test performance after experiencing active or observational simulation training

5.2 ± 0.7

0.63

Total GRS Score

24.3 ± 5.4

21.8 ± 4.5

0.22

ip t

cr

an

P value 0.32 0.16

te

d

M

GRS Skills Category

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

18 Page 19 of 26

ip t

Figure 1: Methods Flow Diagram

te

d

M

an

us

cr

Figure 2: Flow diagram of participants

Ac ce p

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Figure 3: Average total GRS pre- and post-test performance scores for the active and observer groups

* Denotes significant difference within the active group

19 Page 20 of 26

ip t cr

Appendix A: Post-scenario survey Please Circle the appropriate answer Role in 1st Scenario 1. How comfortable were you with this role?

0 Not useful 0 Not useful

ce pt

0 Not stressed

5. What was your perceived stress level for 2nd scenario? 6. Was the debrief helpful in learning CRM

Active Participant 4 Very comfortab le 4 Very useful 4 Very useful

2 Somewhat comfortable

3 Moderately comfortable

1

2 Somewhat useful

3 Moderately useful

1

2 Somewhat useful

3 Moderately useful

1

2 Somewhat stressed

3 Moderately stressed

4 Very stressed

2 Somewhat stressed 2 Somewhat helpful

3 Moderately stressed 3 Moderately helpful

4 Very stressed 4 Very helpful

M an

1

ed

3. How useful did you find the 2nd scenario in learning about crisis resource management? 4. What was your perceived stress level for 1st scenario?

0 Not comfortable

us

Observer Participant

2. How useful did you find the 1st scenario in learning about crisis resource management?

Ac

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

0 Not stressed

1

0 Not helpful

1

5

5

6 Extremely comfortabl e 6 Extremely useful

5

6 Extremely useful

5

6 Extremely stressed

5

5

6 Extremely stressed 6 Extremely helpful

21 Page 21 of 26

ip t

Case I: Lidocaine Toxicity Timeline Status of SimMan

Learning outcomes or actions desired and Triggers Learner Actions: o Primary Assessment o Primary Survey o Obtain further history from nurse o Proper initial management of toxic ingestion and orders appropriate initial investigations (glucose, salicylates, acetaminophen, ethanol levels, electrocardiogram) o Recognize lidocaine toxicity o Calls for help (“resident” available)

2. 3-5 minutes

Further decrease in level of consciousness, heart rate increases.

Learner Actions: o Continue fluids and oxygen (supportive management) o Definitive airway management o Calls for help (resident, anaesthesia, intensive care unit, poison control)

3. 6-10minitues

Ventricular tachycardia pulseless

Operator: o Moans and groans, Vitals HR: 124, BP: 102/60 RR: 18 O2 sat:

ed

M an

us

Glasgow Coma Scale 8 (eyes close, moans and groans, localizes pain) Multiple sutured lacerations on body

ce pt

1. 1- 2 minutes

cr

Appendix B: Crisis Scenarios

Ac

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

Learner Actions: o Initiate Advanced Cardiac Life Support Pulseless Ventricular Tachycardia algorithm

93%

Crash cart outside of room Teaching Points: o Initiate clear primary assessment o Recognition of lidocaine toxicity/ toxic dose/ central nervous system depression o Crisis resource management skills Triggers: -At 1.5 minutes overhead announcement “porter to Obs for patient transport, porter to Obs please” -At 2 minutes - If does not recognize lidocaine toxicity prompt participant by reminding that med student took long time suturing Operator: o Increase HR to 167. BP: 75/40, decrease O2 sat to 85% but increase back up to 90% if bag or intubate Teaching Points: o Initiate appropriate supportive therapy o Consult poison control o Calls for extra help o Call for intralipid – if asked say it is coming Triggers: -3 minutes overhead announcement “Code Blue 4 East, Code Bleu 4 Est.” -At 3 minutes - prompt participant by stating so many bottles of lidocaine used if not recognizing toxicity -4 minutes - investigations back --5.5 minutes - overhead announcement “Code Trauma emergency, Code 1 Urgence” Operator:  Ventricular tachycardia converts following 2st defibrillation Teaching Points: o Crisis resource management principals Triggers: -Regains pulse after 1 cycle of cardiopulmonary resuscitation with total 2 defibrillations - if does not defibrillate by 9 minutes nurse will tell them 2 minutes of CPR has elapsed and insist on 2nd defibrillation.

HR heart rate, BP blood pressure, RR respiratory rate, O2 sat oxygen saturation 22 Page 22 of 26

ip t Learning outcomes or actions desired and Triggers Learner Actions: o Primary Assessment o Primary Survey– including definitive airway o Obtain further history o Proper initial management of toxic ingestion and orders appropriate initial investigations (glucose, salicylates, acetaminophen, ethanol levels, electrocardiogram) o Fluids, oxygen o Calls for help (“resident” available)

2. 3- 5 min

Investigations return

Learner Actions: o Urine alkalinization o Calls nephrology service for dialysis o Calls intensive care unit and poison control o More fluids and correct potassium depletion

3. 6-10 min

Pulseless Electrical Activity arrest

Operator:

HR: 130, BP: 100/70 RR: 40 O2: 91%, temp 38.5 Glucose 5.0, crackles bilaterally in lungs

M an

us

Confused, Glasgow Coma Scale 8 (eyes close, moans and groans, localizes pain)

ce pt

ed

1. 1-2 minutes

cr

Case II: Salicylates Toxicity Status of SimMan Timeline

Learner Actions: o Initiate Advanced Cardiac Lie Support Pulseless Electrical Activity Algorithm

Ac

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

Teaching Points: o Initiate clear primary assessment o Recognition of salicylate toxicity o Crisis resource management skills Triggers: -At 1.5 minutes - overhead announcement “Dr Kay for line one for outside call” Operator: HR 160, BP 72/30, RR 44, temp 38.5, O2sat 86% If bagging or intubated – O2 sat 90% Teaching Points: o Management of acid base disorder from salicylate toxicity o Recognize need for dialysis o Asks for help on management of patient Triggers: -3 minutes - overhead announcement “Stroke code emergency, code stroke urgence” -If call nephrology – they call back and want to talk to them on phone -4 minutes – investigations back -5.5 minutes - overhead announcement “code STEMI heart institute, code STEMI ” Operator: o Pulseless electrical activity reverts to sinus after epinephrine given Teaching Points: o Pulseless electrical activity most likely due to acidosis continue treatment of acidosis o Crisis resource management principles Triggers: -Pulse will return after epinephrine given -If at 9 minutes no epinephrine given, nurse will insist on giving it then pulse returns

HR heart rate, BP blood pressure, RR respiratory rate, O2 sat oxygen saturation 23 Page 23 of 26

Ac ce p

te

d

M

an

us

cr

ip t

Figure 1

Page 24 of 26

Ac ce p

te

d

M

an

us

cr

ip t

Figure 2

Page 25 of 26

ip t

Revised Figure3

us

cr

35

M an

30

ed

20

10

5

Post test

ce pt

15

Pre-test

Ac

Mean Total Ottawa GRS± SD

25

0

Active Group *

Observer Group

Page 26 of 26