- Email: [email protected]
Medical Errors Education for Students of Surgery: A Pilot Study Revealing the Need for Action
ORIGINAL REPORTS
Medical Errors Education for Students of Surgery: A Pilot Study Revealing the Need for Action James H. Paxton, MD, and Ilan S. Rubinfeld, MD Department of Surgery, Henry Ford Hospital, Detroit, Michigan BACKGROUND: Medical errors training is an important yet often overlooked aspect of surgical education. In response to a perceived deficiency in medical errors training at our institution, we implemented an educational session on medical errors concepts for the benefit of rotating medical students. MATERIALS AND METHODS: Medical students com-
pleted the same 12-question test before and after the 90minute educational session. Pretest and posttest scores were compared for evidence of enhanced understanding. No personal identifiers were used, and students were provided unlimited time to complete the tests. Six groups of medical students (who ranged from 5 to 8 students per session) completed the educational session. All sessions were moderated by the same surgical resident and attending surgeon, who used a standard slide presentation. Test scores were analyzed with SPSS statistical software (version 14.0; SPSS Inc., Cary, NC), which employed the paired samples t-test (alpha ⫽ 0.05). RESULTS: Test scores increased significantly from a pretest mean of 27.3% correct (3.28 of 12 possible, SD ⫽ 1.57) to a mean posttest score of 70.1% (8.41, SD ⫽ 1.52) (p ⬍ 0.001). CONCLUSIONS: This retrospective pilot study demonstrated that a brief educational intervention led to statistically significant improved performance on a general understanding of medical errors. The study also revealed the dearth of baseline knowledge in our participating medical students on the subject. We believe that these results underscore the need for action in providing improved and ongoing education in medical errors concepts to enhance medical student awareness and proactive handling of medical errors. (J Surg 66:20-24. © 2009 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
Correspondence: Inquiries to James H. Paxton, MD, MBA, Department of Surgery, Henry Ford Hospital, 2799 West Grand Boulevard, Detroit, MI 48202; fax: (313) 916-8007; e-mail: [email protected]
20
KEYWORDS: medical errors, medical student education,
patient safety COMPETENCIES: Practice-Based Learning and Improve-
ment, Systems-Based Practice, Interpersonal and Communication Skills, Professionalism
INTRODUCTION Medical errors are responsible for an estimated 44,000 to 98,000 patient deaths each year in the United States, which costs the American health-care system more than $37 billion annually.1 Moreover, the Institute of Medicine has identified “faulty systems, processes, and conditions” within America’s health-care institutions as the underlying cause of most medical errors.1 Recognizing the enormity of this problem and the impact of medical errors on patient safety at our own institution, we were concerned about a perceived lack of education on medical errors topics among medical students who rotate on our surgical services. After all, we believe that a basic understanding of medical errors concepts is crucial to the detection and prevention of medical errors. The value of discussion on the subject of medical errors with both medical students and residents has been well described in the literature, especially by facilitating active learning from past medical errors to prevent future errors.2-6 Learners of medicine appreciate and benefit from opportunities to discuss medical errors that they have witnessed with medical colleagues and supervisory physicians in an open, reassuring, and nonthreatening environment.4 Although the role of resident physicians in the prevention of medical errors has received considerable attention, recent studies have shown that medical students can also play a significant role in the prevention of medical errors.7,8 The Accreditation Council of Graduate Medical Education and the American Board of Medical Specialties have identified competency-based education as the means to improve quality and safety in health care. Safety is often considered the “seventh competency” rolled into the PracticeBased Learning and Improvement or Systems-Based Practice
Journal of Surgical Education • © 2009 Association of Program Directors in Surgery Published by Elsevier Inc. All rights reserved.
1931-7204/09/$30.00 doi:10.1016/j.jsurg.2008.07.007
competencies. Error disclosure and communication issues broaden this topic to include the Interpersonal and Communication Skills competency as well as Professionalism. However, few validated educational or assessment tools exist to pursue these competencies as they pertain to medical errors. In light of this deficiency, we aimed to develop a brief, highly informative educational session to introduce learners of surgery to the key concepts, theories, and preventive strategies relevant to the study of medical errors. Although these core concepts are by no means specific to surgery, we integrated specific examples of surgical errors into the discussion, which aided our students in applying these fundamental principles to the patient-care activities that they witness daily in the wards and in the operating room.
MATERIALS AND METHODS During the fall of 2005, a team of surgical residents and attending staff developed a Microsoft PowerPoint (Microsoft Corporation, Redmond, Washington) presentation that consisted of approximately 45 slides covering 6 major
TABLE 1. Subjects Addressed in MEs’ Education Session 1. Terminology (T)1 Definitions of error/ME Definition of adverse event Preventable versus unpreventable medical errors Definitions of slip/lapse/mistake Definitions of negligence/malpractice 2. Active versus Latent Errors (ALE)1 Definitions of active error/latent error Root cause analysis 3. Incidence (I)5,9-12 Institute of medicine statistics (cost/morbidity/mortality) Incidence studies—Harvard Medical Practice Study/Virginia CWS Resident self-reporting Joint Commission on Accreditation of Healthcare Organizations Sentinel event definition/statistics Medication errors/“unsafe” abbreviations 4. Error Theory (ET)1,13,14 Chains of error/aviation industry “Swiss cheese model” (J Reason) Human Factors Analysis and Classification System framework “Spectrum of defense” (individual vs device/systemic improvements) 5. Error Disclosure (ED)5,15-20 American Medical Association Code of Medical Ethics/ American College of Physicians Ethics Manual Full disclosure versus nondisclosure Barriers to full disclosure Disclosure—why and how? 6. Legal Considerations (LC)21-24 Mandatory error reporting Litigation statistics—“litigation lottery?” Why do patients sue? Medical malpractice in Michigan ME, medical error.
medical errors subjects: terminology (T), active versus latent errors (ALE), incidence (I), error theory (ET), error disclosure (ED), and legal considerations (LC). These subjects are recorded in greater detail in Table 1.5,9-24 Although terminology was an important component of this session, most educational content was framed in layman’s terms, which focused on conceptual understanding rather than on rote memorization of research terminology. A single surgical resident and attending staff surgeon were selected to moderate all planned educational sessions to provide the highest possible level of consistency between sessions. We included only medical students who rotated on the general surgery service in these sessions, with the goal of refining the session for future use with surgical residents. All medical students who rotated in the department of surgery from December 2005 to December 2006 participated in the educational session as a component of their scheduled lectures for the 2-month rotation. A total of 6 bimonthly sessions were organized; the number of participants ranged from 5 to 8 students per session. Sessions were formatted as a 2-hour small group discussion, using the PowerPoint presentation to facilitate discussion and focus attention on important learning points. Frequent breaks throughout the session allowed students the opportunity to describe their own experiences with medical errors using session content as a framework for their responses. Students were asked to complete an identical 12-question pretest and posttest before and immediately after the session. Content and face validity of the presentation and testing instrument were evaluated by a multidisciplinary team, which included senior surgical staff from the divisions of general surgery, trauma surgery, vascular surgery, and surgical critical care; an associate program director in general surgery; a surgical resident; and a doctoral-level education specialist with 10 years of graduate medical education experience in psychometrics, which includes research and development of surgical resident curriculum. This team evaluated 20 questions for content validity and settled on 12 questions that were felt to reflect session content adequately with appropriate depth of knowledge for an audience of medical students. Students were provided unlimited time to complete the pretest and posttest, although all students completed the test within 15 minutes. Students identified themselves on the examination by session month and a randomly assigned number (eg, December 1, December 2, and so on) as well as by year of training. Pretest and posttest results were originally obtained strictly for use in refining the course, as we did not intend to publish the results of this pilot study. However, after review of the summary results for our pilot study, we applied for and received institutional review board (IRB) approval for retrospective analysis and publication of the testing results. The pretest and posttest consisted of the same 12 questions in multiple-choice format with a brief question stem and 5 foils. These questions are provided in abbreviated form
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
21
TABLE 2. Questions Included in Pretest and Posttest Examinations Question (Subject Area)
Mean Pretest Correct (%)
Mean Posttest Correct (%)
% Change (p value)
1. Define adverse event (T) 2. Vignette, identify mistake (T) 3. Vignette, identify slip (T) 4. Active versus latent error (ALE) 5. Vignette, identify latent error (ALE) 6. Incidence of deaths attributable to MEs (I) 7. Medication errors, stage in prescribing (I) 8. Most effective way of reducing errors (ET) 9. Disclosure and reporting requirements (ED) 10. Definition of malpractice (LC) 11. Odds of being sued in the United States for ME (LC) 12. Identify high-risk specialties (LC) Average
18.75 15.63 15.63 31.25 59.38 43.75 31.25 9.38 6.25 3.13 28.13 62.50 27.34
59.38 59.38 71.88 84.38 81.25 84.38 93.75 71.88 34.38 21.88 81.25 96.88 70.05
40.63 (0.003) 43.75 (0.017) 56.25 (⬍0.001) 53.13 (0.008) 21.88 (0.017) 40.63 (0.137) 62.50 (0.002) 62.50 (0.008) 28.13 (0.022) 18.75 (0.038) 53.13 (⬍0.001) 34.38 (0.006) 42.71 (⬍0.001)
ME, medical error.
in Table 2. Students who failed to answer all 12 questions on both tests were excluded from the study. Student scores were not used in determination of a student’s grade on the clerkship or released to the medical student program director. Pretest and posttest scores for individual students were analyzed using the paired samples t-test, as were pretest and posttest outcomes for individual questions.
RESULTS A total of 36 students [2 second-year medical students (MS2), 33 MS-3 students, and 1 MS-4 student] participated in the educational sessions. Of these, 32 students (88.9%) were included in the analysis, with 4 students excluded for failure to answer all questions (2 students failed to answer 1 question, 1 student failed to answer 2 questions, 1 student failed to answer 3 questions). Among these 32 students, test scores showed statistically significant improvement from a pretest mean of 27.3% correct (3.28 of 12 possible, SD ⫽ 1.57) to a mean posttest score of 70.1% correct (8.41, SD ⫽ 1.52) (p ⬍ 0.001). The 2 MS-2 students earned lower scores on the pretest than the MS-3 students (average 16.7% vs 28.2%), but both groups scored similarly on the posttest (average 70.8% for MS-2 students vs 69.5% for MS-3 students). Alternatively, the single MS-4 student’s pretest score was on par with those of the MS-3 students (average 25% vs 28.2% for MS-3 students), but his posttest score was substantially higher than the MS-3 average (average 83.3% vs 69.5% for MS-3 students). Of the 12 questions used in the testing instrument, all questions achieved a statistically significant level of improvement with posttesting, except for question 6 (p ⫽ 0.137). The individual p-values for each question are recorded in Table 2. 22
DISCUSSION This retrospective pilot study demonstrated that a brief educational intervention led to improved medical student performance in relation to a general understanding of medical errors. Although these educational gains were statistically significant, our retrospective review has identified many session areas in need of improvement. Certain questions, particularly questions 9 and 10, were associated with extremely low scores on the posttest and have already been clarified with noted improvement in the last 2 sessions. We also excluded 4 students from the study for failure to complete all test questions, which represents a sizeable percentage (11.1%) of our student population. After completion of a few sessions, we learned to ask the students to double check their work prior to handing in the assigned pretest and posttests. Consequently, none of the students in the last 4 sessions failed to complete the testing instrument, and we plan to continue this practice to help increase the power of future studies. Eleven of 12 questions showed statistically significant improvement on posttest, with the 1 exception being question 6 (p ⫽ 0.137). One explanation may be that this question showed a decline in the number of correct answers in a single cohort (December 2006, from 3 correct on pretest to 1 correct on posttest). Excluding this cohort from analysis, question 6 also demonstrates statistically significant improvement (p ⫽ 0.031). Although our educational session may not have been the first exposure to medical errors concepts for these students, we were consistently dismayed at how little baseline knowledge they brought to the session. Whether this dearth of knowledge is because of lack of exposure, we believe that these results reveal a need for action in providing improved and ongoing education to medical students to enhance awareness and facilitate their critical analysis of medical errors. Early and frequent attempts to educate young clinicians prior to the outset of their career hold the potential for more
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
substantial and enduring change.2,4-6 An exposed medical student will become a more receptive resident and attending physician. Few validated tools exist for education in this realm, and we seek to begin the creation of this essential toolbox at our institution. Future directions include a prospective analysis of student pretest and posttest performance using a refined version of the current educational session and test instrument. This proposed study has received IRB approval at our institution and will require student consent to participate. We expect to enroll a larger sample size in the prospective study to provide adequate power, and we will enroll a control group of additional medical students from the same medical school who will not participate in the educational session. Unlike our rotating medical students, who are primarily third-year students, the control arm will include more MS-1, MS-2, and MS-4 students to assess whether significant differences exist between different medical school classes in terms of baseline medical errors knowledge. We also intend to measure retention of subject matter at 1–12 months posttesting with the next iteration of this study, to ascertain whether the shortterm gains translate to enhanced, long-term understanding of the material. We believe that this project represents a competency education tool, which is currently focused on medical student education in our pilot study and future planned sessions, but it may be similarly applied to resident levels. The relevant competencies include Practice-Based Learning and Improvement, Systems-Based Practice, Interpersonal and Communication Skills, and Professionalism. This work represents the beginning of a process to create validated methods of teaching and assessing competency education in this elusive area related to both patient safety and health-care quality.
4. Fischer MA, Mazor KM, Baril J, Alper E, DeMarco D,
Pugnaire M. Learning from mistakes. Factors that influence how students and residents learn from medical errors. J Gen Intern Med. 2006;21:419-423. 5. Wu AW, Folkman S, McPhee SJ, Lo B. Do house of-
ficers learn from their mistakes? JAMA. 1991;265: 2089-2094. 6. White AA, Gallagher TH, Krauss MJ, et al. The atti-
tudes and experiences of trainees regarding disclosing medical errors to patients. Acad Med. 2008;83:250256. 7. Seiden SC, Galvan C, Lamm R. Role of medical students
in preventing patient harm and enhancing patient safety. Qual Saf Health Care. 2006;15:272-276. 8. Borenstein SH, Choi M, Gerstle JT, Langer JC. Errors
and adverse outcomes on a surgical service: what is the role of residents? J Surg Res. 2004;122:162-166. 9. McGuire HH, Horsley JS, Salter DR, Sobel M. Mea-
suring and managing quality of surgery: statistical vs. incidental approaches. Arch Surg. 1992;127:733-737. 10. http://www.jointcommission.org/SentinelEvents/Statistics/.
Accessed December 10, 2005. 11. Bates DW, Leape LL, Cullen DJ, et al. Effect of comput-
erized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998; 280:1311-1316. 12. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug
events in ambulatory care. N Engl J Med. 2003;348: 1556-1564. 13. Reason J. Human Error. New York: Cambridge Univer-
ACKNOWLEDGMENTS
sity Press; 1990.
We would like to acknowledge the contributions of Joseph L. Musial, PhD, in revising the testing instrument used for this study, and Sarah Whitehouse, MAW, for her editorial assistance.
14. Shappell S, Wiegmann D. The Human Factors Analysis
REFERENCES
15. American Medical Association. AMA Code of Medical
and Classification System (HFACS). Federal Aviation Administration, Office of Aviation Medicine Report No. DOT/ FAA/AM-00/7. Washington, DC: Office of Aviation Medicine; 2000.
Health System. Washington, DC: National Academy Press; 1999.
Ethics: Current Opinions with Annotations. American Medical Association Council on Ethical and Judicial Affairs. Chicago, IL: American Medical Association; 2002-2003.
2. Madigosky WS, Headrick LA, Nelson K, Cox KR, Ander-
16. American College of Physicians. Ethics manual, 4th edi-
son T. Changing and sustaining medical students’ knowledge, skills, and attitudes about patient safety and medical fallibility. Acad Med. 2006;81:94-101.
17. Gallagher TH, Waterman AD, Ebers AG, Fraser VJ, Levin-
1. Institute of Medicine. To Err is Human: Building a Safer
3. Engel KG, Rosenthal M, Sutcliffe KM. Residents’ re-
sponses to medical error: coping, learning, and change. Acad Med. 2006;81:86-93.
tion. Ann Intern Med. 1998;128:576-594. son W. Patients’ and physicians’ attitudes regarding the disclosure of medical errors. JAMA. 2003;289:1001-1007. 18. Robinson AR, Hohmann KB, Rifkin JI, et al. Physician
and public opinions on quality of health care and the
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
23
problem of medical errors. Arch Intern Med. 2002; 162:2186-2190. 19. Leape LL. Error in medicine. JAMA. 1994;272:1851-
1857. 20. Hérbert PC, Levin AV, Robertson G. Bioethics for clini-
cians: 23. Disclosure of medical error. CMAJ. 2001;164:509.
22. Lown B. The Lost Art of Healing: Practicing Compassion in
Medicine. New York: Ballantine Books; 1996. 23. Hiatt HH, Barnes BA, Brennan TA, et al. A study of
medical injury and medical malpractice: an overview. N Engl J Med. 1989;321:480-484.
21. Blendon RJ, DesRoches CM, Brodie M, et al. Views of
24. Vincent C, Young M, Phillips A. Why do people sue
practicing physicians and the public on medical errors. N Engl J Med. 2002;347:1933-1940.
doctors? A study of patients and relatives taking legal action. Lancet. 1994;343:1609-1613.
24
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
Medical Errors Education for Students of Surgery: A Pilot Study Revealing the Need for Action James H. Paxton, MD, and Ilan S. Rubinfeld, MD Department of Surgery, Henry Ford Hospital, Detroit, Michigan BACKGROUND: Medical errors training is an important yet often overlooked aspect of surgical education. In response to a perceived deficiency in medical errors training at our institution, we implemented an educational session on medical errors concepts for the benefit of rotating medical students. MATERIALS AND METHODS: Medical students com-
pleted the same 12-question test before and after the 90minute educational session. Pretest and posttest scores were compared for evidence of enhanced understanding. No personal identifiers were used, and students were provided unlimited time to complete the tests. Six groups of medical students (who ranged from 5 to 8 students per session) completed the educational session. All sessions were moderated by the same surgical resident and attending surgeon, who used a standard slide presentation. Test scores were analyzed with SPSS statistical software (version 14.0; SPSS Inc., Cary, NC), which employed the paired samples t-test (alpha ⫽ 0.05). RESULTS: Test scores increased significantly from a pretest mean of 27.3% correct (3.28 of 12 possible, SD ⫽ 1.57) to a mean posttest score of 70.1% (8.41, SD ⫽ 1.52) (p ⬍ 0.001). CONCLUSIONS: This retrospective pilot study demonstrated that a brief educational intervention led to statistically significant improved performance on a general understanding of medical errors. The study also revealed the dearth of baseline knowledge in our participating medical students on the subject. We believe that these results underscore the need for action in providing improved and ongoing education in medical errors concepts to enhance medical student awareness and proactive handling of medical errors. (J Surg 66:20-24. © 2009 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
Correspondence: Inquiries to James H. Paxton, MD, MBA, Department of Surgery, Henry Ford Hospital, 2799 West Grand Boulevard, Detroit, MI 48202; fax: (313) 916-8007; e-mail: [email protected]
20
KEYWORDS: medical errors, medical student education,
patient safety COMPETENCIES: Practice-Based Learning and Improve-
ment, Systems-Based Practice, Interpersonal and Communication Skills, Professionalism
INTRODUCTION Medical errors are responsible for an estimated 44,000 to 98,000 patient deaths each year in the United States, which costs the American health-care system more than $37 billion annually.1 Moreover, the Institute of Medicine has identified “faulty systems, processes, and conditions” within America’s health-care institutions as the underlying cause of most medical errors.1 Recognizing the enormity of this problem and the impact of medical errors on patient safety at our own institution, we were concerned about a perceived lack of education on medical errors topics among medical students who rotate on our surgical services. After all, we believe that a basic understanding of medical errors concepts is crucial to the detection and prevention of medical errors. The value of discussion on the subject of medical errors with both medical students and residents has been well described in the literature, especially by facilitating active learning from past medical errors to prevent future errors.2-6 Learners of medicine appreciate and benefit from opportunities to discuss medical errors that they have witnessed with medical colleagues and supervisory physicians in an open, reassuring, and nonthreatening environment.4 Although the role of resident physicians in the prevention of medical errors has received considerable attention, recent studies have shown that medical students can also play a significant role in the prevention of medical errors.7,8 The Accreditation Council of Graduate Medical Education and the American Board of Medical Specialties have identified competency-based education as the means to improve quality and safety in health care. Safety is often considered the “seventh competency” rolled into the PracticeBased Learning and Improvement or Systems-Based Practice
Journal of Surgical Education • © 2009 Association of Program Directors in Surgery Published by Elsevier Inc. All rights reserved.
1931-7204/09/$30.00 doi:10.1016/j.jsurg.2008.07.007
competencies. Error disclosure and communication issues broaden this topic to include the Interpersonal and Communication Skills competency as well as Professionalism. However, few validated educational or assessment tools exist to pursue these competencies as they pertain to medical errors. In light of this deficiency, we aimed to develop a brief, highly informative educational session to introduce learners of surgery to the key concepts, theories, and preventive strategies relevant to the study of medical errors. Although these core concepts are by no means specific to surgery, we integrated specific examples of surgical errors into the discussion, which aided our students in applying these fundamental principles to the patient-care activities that they witness daily in the wards and in the operating room.
MATERIALS AND METHODS During the fall of 2005, a team of surgical residents and attending staff developed a Microsoft PowerPoint (Microsoft Corporation, Redmond, Washington) presentation that consisted of approximately 45 slides covering 6 major
TABLE 1. Subjects Addressed in MEs’ Education Session 1. Terminology (T)1 Definitions of error/ME Definition of adverse event Preventable versus unpreventable medical errors Definitions of slip/lapse/mistake Definitions of negligence/malpractice 2. Active versus Latent Errors (ALE)1 Definitions of active error/latent error Root cause analysis 3. Incidence (I)5,9-12 Institute of medicine statistics (cost/morbidity/mortality) Incidence studies—Harvard Medical Practice Study/Virginia CWS Resident self-reporting Joint Commission on Accreditation of Healthcare Organizations Sentinel event definition/statistics Medication errors/“unsafe” abbreviations 4. Error Theory (ET)1,13,14 Chains of error/aviation industry “Swiss cheese model” (J Reason) Human Factors Analysis and Classification System framework “Spectrum of defense” (individual vs device/systemic improvements) 5. Error Disclosure (ED)5,15-20 American Medical Association Code of Medical Ethics/ American College of Physicians Ethics Manual Full disclosure versus nondisclosure Barriers to full disclosure Disclosure—why and how? 6. Legal Considerations (LC)21-24 Mandatory error reporting Litigation statistics—“litigation lottery?” Why do patients sue? Medical malpractice in Michigan ME, medical error.
medical errors subjects: terminology (T), active versus latent errors (ALE), incidence (I), error theory (ET), error disclosure (ED), and legal considerations (LC). These subjects are recorded in greater detail in Table 1.5,9-24 Although terminology was an important component of this session, most educational content was framed in layman’s terms, which focused on conceptual understanding rather than on rote memorization of research terminology. A single surgical resident and attending staff surgeon were selected to moderate all planned educational sessions to provide the highest possible level of consistency between sessions. We included only medical students who rotated on the general surgery service in these sessions, with the goal of refining the session for future use with surgical residents. All medical students who rotated in the department of surgery from December 2005 to December 2006 participated in the educational session as a component of their scheduled lectures for the 2-month rotation. A total of 6 bimonthly sessions were organized; the number of participants ranged from 5 to 8 students per session. Sessions were formatted as a 2-hour small group discussion, using the PowerPoint presentation to facilitate discussion and focus attention on important learning points. Frequent breaks throughout the session allowed students the opportunity to describe their own experiences with medical errors using session content as a framework for their responses. Students were asked to complete an identical 12-question pretest and posttest before and immediately after the session. Content and face validity of the presentation and testing instrument were evaluated by a multidisciplinary team, which included senior surgical staff from the divisions of general surgery, trauma surgery, vascular surgery, and surgical critical care; an associate program director in general surgery; a surgical resident; and a doctoral-level education specialist with 10 years of graduate medical education experience in psychometrics, which includes research and development of surgical resident curriculum. This team evaluated 20 questions for content validity and settled on 12 questions that were felt to reflect session content adequately with appropriate depth of knowledge for an audience of medical students. Students were provided unlimited time to complete the pretest and posttest, although all students completed the test within 15 minutes. Students identified themselves on the examination by session month and a randomly assigned number (eg, December 1, December 2, and so on) as well as by year of training. Pretest and posttest results were originally obtained strictly for use in refining the course, as we did not intend to publish the results of this pilot study. However, after review of the summary results for our pilot study, we applied for and received institutional review board (IRB) approval for retrospective analysis and publication of the testing results. The pretest and posttest consisted of the same 12 questions in multiple-choice format with a brief question stem and 5 foils. These questions are provided in abbreviated form
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
21
TABLE 2. Questions Included in Pretest and Posttest Examinations Question (Subject Area)
Mean Pretest Correct (%)
Mean Posttest Correct (%)
% Change (p value)
1. Define adverse event (T) 2. Vignette, identify mistake (T) 3. Vignette, identify slip (T) 4. Active versus latent error (ALE) 5. Vignette, identify latent error (ALE) 6. Incidence of deaths attributable to MEs (I) 7. Medication errors, stage in prescribing (I) 8. Most effective way of reducing errors (ET) 9. Disclosure and reporting requirements (ED) 10. Definition of malpractice (LC) 11. Odds of being sued in the United States for ME (LC) 12. Identify high-risk specialties (LC) Average
18.75 15.63 15.63 31.25 59.38 43.75 31.25 9.38 6.25 3.13 28.13 62.50 27.34
59.38 59.38 71.88 84.38 81.25 84.38 93.75 71.88 34.38 21.88 81.25 96.88 70.05
40.63 (0.003) 43.75 (0.017) 56.25 (⬍0.001) 53.13 (0.008) 21.88 (0.017) 40.63 (0.137) 62.50 (0.002) 62.50 (0.008) 28.13 (0.022) 18.75 (0.038) 53.13 (⬍0.001) 34.38 (0.006) 42.71 (⬍0.001)
ME, medical error.
in Table 2. Students who failed to answer all 12 questions on both tests were excluded from the study. Student scores were not used in determination of a student’s grade on the clerkship or released to the medical student program director. Pretest and posttest scores for individual students were analyzed using the paired samples t-test, as were pretest and posttest outcomes for individual questions.
RESULTS A total of 36 students [2 second-year medical students (MS2), 33 MS-3 students, and 1 MS-4 student] participated in the educational sessions. Of these, 32 students (88.9%) were included in the analysis, with 4 students excluded for failure to answer all questions (2 students failed to answer 1 question, 1 student failed to answer 2 questions, 1 student failed to answer 3 questions). Among these 32 students, test scores showed statistically significant improvement from a pretest mean of 27.3% correct (3.28 of 12 possible, SD ⫽ 1.57) to a mean posttest score of 70.1% correct (8.41, SD ⫽ 1.52) (p ⬍ 0.001). The 2 MS-2 students earned lower scores on the pretest than the MS-3 students (average 16.7% vs 28.2%), but both groups scored similarly on the posttest (average 70.8% for MS-2 students vs 69.5% for MS-3 students). Alternatively, the single MS-4 student’s pretest score was on par with those of the MS-3 students (average 25% vs 28.2% for MS-3 students), but his posttest score was substantially higher than the MS-3 average (average 83.3% vs 69.5% for MS-3 students). Of the 12 questions used in the testing instrument, all questions achieved a statistically significant level of improvement with posttesting, except for question 6 (p ⫽ 0.137). The individual p-values for each question are recorded in Table 2. 22
DISCUSSION This retrospective pilot study demonstrated that a brief educational intervention led to improved medical student performance in relation to a general understanding of medical errors. Although these educational gains were statistically significant, our retrospective review has identified many session areas in need of improvement. Certain questions, particularly questions 9 and 10, were associated with extremely low scores on the posttest and have already been clarified with noted improvement in the last 2 sessions. We also excluded 4 students from the study for failure to complete all test questions, which represents a sizeable percentage (11.1%) of our student population. After completion of a few sessions, we learned to ask the students to double check their work prior to handing in the assigned pretest and posttests. Consequently, none of the students in the last 4 sessions failed to complete the testing instrument, and we plan to continue this practice to help increase the power of future studies. Eleven of 12 questions showed statistically significant improvement on posttest, with the 1 exception being question 6 (p ⫽ 0.137). One explanation may be that this question showed a decline in the number of correct answers in a single cohort (December 2006, from 3 correct on pretest to 1 correct on posttest). Excluding this cohort from analysis, question 6 also demonstrates statistically significant improvement (p ⫽ 0.031). Although our educational session may not have been the first exposure to medical errors concepts for these students, we were consistently dismayed at how little baseline knowledge they brought to the session. Whether this dearth of knowledge is because of lack of exposure, we believe that these results reveal a need for action in providing improved and ongoing education to medical students to enhance awareness and facilitate their critical analysis of medical errors. Early and frequent attempts to educate young clinicians prior to the outset of their career hold the potential for more
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
substantial and enduring change.2,4-6 An exposed medical student will become a more receptive resident and attending physician. Few validated tools exist for education in this realm, and we seek to begin the creation of this essential toolbox at our institution. Future directions include a prospective analysis of student pretest and posttest performance using a refined version of the current educational session and test instrument. This proposed study has received IRB approval at our institution and will require student consent to participate. We expect to enroll a larger sample size in the prospective study to provide adequate power, and we will enroll a control group of additional medical students from the same medical school who will not participate in the educational session. Unlike our rotating medical students, who are primarily third-year students, the control arm will include more MS-1, MS-2, and MS-4 students to assess whether significant differences exist between different medical school classes in terms of baseline medical errors knowledge. We also intend to measure retention of subject matter at 1–12 months posttesting with the next iteration of this study, to ascertain whether the shortterm gains translate to enhanced, long-term understanding of the material. We believe that this project represents a competency education tool, which is currently focused on medical student education in our pilot study and future planned sessions, but it may be similarly applied to resident levels. The relevant competencies include Practice-Based Learning and Improvement, Systems-Based Practice, Interpersonal and Communication Skills, and Professionalism. This work represents the beginning of a process to create validated methods of teaching and assessing competency education in this elusive area related to both patient safety and health-care quality.
4. Fischer MA, Mazor KM, Baril J, Alper E, DeMarco D,
Pugnaire M. Learning from mistakes. Factors that influence how students and residents learn from medical errors. J Gen Intern Med. 2006;21:419-423. 5. Wu AW, Folkman S, McPhee SJ, Lo B. Do house of-
ficers learn from their mistakes? JAMA. 1991;265: 2089-2094. 6. White AA, Gallagher TH, Krauss MJ, et al. The atti-
tudes and experiences of trainees regarding disclosing medical errors to patients. Acad Med. 2008;83:250256. 7. Seiden SC, Galvan C, Lamm R. Role of medical students
in preventing patient harm and enhancing patient safety. Qual Saf Health Care. 2006;15:272-276. 8. Borenstein SH, Choi M, Gerstle JT, Langer JC. Errors
and adverse outcomes on a surgical service: what is the role of residents? J Surg Res. 2004;122:162-166. 9. McGuire HH, Horsley JS, Salter DR, Sobel M. Mea-
suring and managing quality of surgery: statistical vs. incidental approaches. Arch Surg. 1992;127:733-737. 10. http://www.jointcommission.org/SentinelEvents/Statistics/.
Accessed December 10, 2005. 11. Bates DW, Leape LL, Cullen DJ, et al. Effect of comput-
erized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998; 280:1311-1316. 12. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug
events in ambulatory care. N Engl J Med. 2003;348: 1556-1564. 13. Reason J. Human Error. New York: Cambridge Univer-
ACKNOWLEDGMENTS
sity Press; 1990.
We would like to acknowledge the contributions of Joseph L. Musial, PhD, in revising the testing instrument used for this study, and Sarah Whitehouse, MAW, for her editorial assistance.
14. Shappell S, Wiegmann D. The Human Factors Analysis
REFERENCES
15. American Medical Association. AMA Code of Medical
and Classification System (HFACS). Federal Aviation Administration, Office of Aviation Medicine Report No. DOT/ FAA/AM-00/7. Washington, DC: Office of Aviation Medicine; 2000.
Health System. Washington, DC: National Academy Press; 1999.
Ethics: Current Opinions with Annotations. American Medical Association Council on Ethical and Judicial Affairs. Chicago, IL: American Medical Association; 2002-2003.
2. Madigosky WS, Headrick LA, Nelson K, Cox KR, Ander-
16. American College of Physicians. Ethics manual, 4th edi-
son T. Changing and sustaining medical students’ knowledge, skills, and attitudes about patient safety and medical fallibility. Acad Med. 2006;81:94-101.
17. Gallagher TH, Waterman AD, Ebers AG, Fraser VJ, Levin-
1. Institute of Medicine. To Err is Human: Building a Safer
3. Engel KG, Rosenthal M, Sutcliffe KM. Residents’ re-
sponses to medical error: coping, learning, and change. Acad Med. 2006;81:86-93.
tion. Ann Intern Med. 1998;128:576-594. son W. Patients’ and physicians’ attitudes regarding the disclosure of medical errors. JAMA. 2003;289:1001-1007. 18. Robinson AR, Hohmann KB, Rifkin JI, et al. Physician
and public opinions on quality of health care and the
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009
23
problem of medical errors. Arch Intern Med. 2002; 162:2186-2190. 19. Leape LL. Error in medicine. JAMA. 1994;272:1851-
1857. 20. Hérbert PC, Levin AV, Robertson G. Bioethics for clini-
cians: 23. Disclosure of medical error. CMAJ. 2001;164:509.
22. Lown B. The Lost Art of Healing: Practicing Compassion in
Medicine. New York: Ballantine Books; 1996. 23. Hiatt HH, Barnes BA, Brennan TA, et al. A study of
medical injury and medical malpractice: an overview. N Engl J Med. 1989;321:480-484.
21. Blendon RJ, DesRoches CM, Brodie M, et al. Views of
24. Vincent C, Young M, Phillips A. Why do people sue
practicing physicians and the public on medical errors. N Engl J Med. 2002;347:1933-1940.
doctors? A study of patients and relatives taking legal action. Lancet. 1994;343:1609-1613.
24
Journal of Surgical Education • Volume 66/Number 1 • January/February 2009