The Unmet Need for a National Surgical Quality Improvement Curriculum: A Systematic Review

ORIGINAL REPORTS

The Unmet Need for a National Surgical Quality Improvement Curriculum: A Systematic Review Rachel L. Medbery, MD,* Morgan M. Sellers, MD,† Clifford Y. Ko, MD, MS,‡ and Rachel R. Kelz, MD, MSCE† Department of Surgery, Emory University School of Medicine, Atlanta, Georgia; †Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; and ‡Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, Illinois *

INTRODUCTION: The Accreditation Council for Gradu-

CONCLUSION: Elements of QI curriculum for surgical

ate Medical Education Next Accreditation System will require general surgery training programs to demonstrate outstanding clinical outcomes and education in quality improvement (QI). The American College of SurgeonsNational Surgical Quality Improvement Project Quality InTraining Initiative reports the results of a systematic review of the literature investigating the availability of a QI curriculum.

education exist; however, comprehensive content is lacking. The American College of Surgeons-National Surgical Quality Improvement Project Quality In-Training Initiative will build on the high-quality components identified in our review and develop data-centered QI content to generate a comprehensive national QI curriculum for use in graduate C 2014 Association of surgical education. ( J Surg ]:]]]-]]]. J Program Directors in Surgery Published by Elsevier Inc. All rights reserved.)

METHODS: Using defined search terms, a systematic review

was conducted in Embase, PubMed, and Google Scholar (January 2000-March 2013) to identify a surgical QI curriculum. Bibliographies from selected articles and other relevant materials were also hand searched. Curriculum was defined as an organized program of learning complete with content, instruction, and assessment for use in general surgical residency programs. Two independent observers graded surgical articles on quality of curriculum presented. RESULTS: Overall, 50 of 1155 references had information regarding QI in graduate medical education. Most (n ¼ 24, 48%) described QI education efforts in nonsurgical fields. A total of 31 curricular blueprints were identified; 6 (19.4%) were specific to surgery. Targeted learners were most often post graduate year-2 residents (29.0%); only 6 curricula (19.4%) outlined a course for all residents within their respective programs. Plan, Do, Study, Act (n ¼ 10, 32.3%), and Root Cause Analysis (n ¼ 5, 16.1%) were the most common QI content presented, the majority of instruction was via lecture/didactics (n ¼ 26, 83.9%), and only 7 (22.6%) curricula used validated tool kits for assessment. Presented in part at the 2013 Surgical Forum during the 99th Annual Clinical Congress of the American College of Surgeons. Correspondence: Inquiries to Rachel R. Kelz, MD, MSCE, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104; fax: (215) 662-7476; e-mail: [email protected]

KEY WORDS: surgical education, quality improvement,

graduate medical education, curriculum COMPETENCIES: Patient Care, Medical Knowledge, Practice Based Learning and Improvement, Systems-based Practice, Professionalism, Interpersonal Skills and Communication

INTRODUCTION The Accreditation Council for Graduate Medical Education (ACGME) Core Competencies require residents to participate in systems-based practice and practice-based learning and improvement.1,2 Additionally, the Next Accreditation System Clinical Learning Environment Review (CLER) Program will focus on 6 areas important to the safety and quality of care provided in teaching hospitals and throughout a lifetime of practice after the completion of the residency training period.3,4 These initiatives have made resident education in quality improvement (QI) a requirement to maintain accreditation; thereby giving surgical residency programs an opportunity to teach residents the science of QI. Unfortunately, however, educational programs to teach this new material are sparse, and program directors struggle to make the material relevant and to

Journal of Surgical Education  & 2014 Association of Program Directors in Surgery Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2013.12.004

1

identify a critical mass of instructors with experience in this subject matter. The American College of Surgeons National Surgical Quality Improvement Program Quality In-Training Initiative (ACS-NSQIP QITI) is a collaborative of the ACS-NSQIP designed to integrate patient-centered outcomes into surgical training in QI and to unite teaching hospitals in the pursuit of optimal patient care.5-8 As a part of this initiative, we aim to facilitate the implementation of a broad-based QI curriculum designed for surgeons to facilitate resident engagement. The proper curriculum must also highlight key concepts of patient safety and value-based delivery of care. In so doing, we hope to enable programmatic success with the ACGME requirements in systems-based practice and practice-based learning and improvement, and ensure success in the CLER program. We present the results of a comprehensive systematic review of QI educational materials designed to identify a surgery-specific QI curriculum for use in graduate surgical education.

METHODS Focus Groups Before designing the systematic review, we convened several focus groups during the academic year 2012 to outline the ideal content for a QI curriculum in surgery. The groups generated an outline of desired content and the final list was unanimously agreed on. It was also agreed on that any QI curriculum would need regular revision to reflect the needs of the learners and society. Participants included a nationally representative cohort of program directors, quality champions, nurses, residents, and administrators. This information was used as the basis for assessment of the completeness of the QI content. Search Strategy A systematic review was conducted in Embase, PubMed, and Google Scholar to identify a surgical QI curriculum. Defined search terms were “surgical education”/“graduate medical education”/“resident education”/“curriculum”/ “general surgery” and “quality improvement”/“patient safety”/“quality.” The search was limited to Englishlanguage articles published between January 2000 and March 2013. We chose January 2000 given the publication of the Institute of Medicine’s To Err is Human report in 1999.9 To ensure all potential resources for use in a surgeryspecific curriculum were included, bibliographies from selected articles and relevant materials from professional societies (Association of American Medical Colleges, Association for Surgical Education [ASE], American College of Surgeons, and Association for Program Directors in Surgery 2

[APDS]), accreditation boards (ACGME, American Board of Surgery, American Board of Medical Specialties, and American Board of Internal Medicine), and other wellknow resources (Institute for Healthcare Improvement [IHI], SCORE General Surgery Resident Curriculum Portal, and Clinical Preventive Services Practice Improvement Module) were also hand searched. Selection Criteria To identify studies for inclusion, a single reviewer (R.L.M.) screened all titles for relevance. Articles were considered relevant if they contained information regarding a QI curriculum for use in graduate medical education. Two independent reviewers (R.L.M. and R.R.K.) read abstracts of all potentially eligible articles. For articles that could not be excluded based on title/abstract, we obtained and reviewed the full text, again independently and in duplicate. We resolved all disagreements by consensus. Interrater reliability was assessed using the Cohen’s kappa. Articles were excluded if they pertained only to education of other health care professionals, were designed for implementation outside the USA, or if the full-text article was unavailable. Articles targeted to undergraduate or continuing medical education or both were labeled as unrelated. Data Abstraction We abstracted information from each study using a standardized abstraction form. Title, authors, year of publication, aim, specialty, and target learners were abstracted from all full-text reviews. Two independent reviewers abstracted all information for which reviewer judgment was required, with disagreements resolved by discussion. Additional information was abstracted from articles that presented a QI curriculum. For the current study, curriculum was defined as an organized program of learning complete with content, instruction, and assessment for use in general surgical residency programs. We abstracted information from these peer reviewed articles on 4 of the 6 parameters noted in Kerns 6-step approach to curricular design—problem identification and general needs assessment, needs assessment for targeted learners, goals and objectives (aim), educational strategies (instruction, materials, time allocation, and cost), implementation, and evaluation and feedback (assessment, outcomes, and satisfaction)—whenever possible.10 We did not abstract the information on the needs assessment as it was not relevant to the stated goals of the study. Data Synthesis Owing to the considerable amount of heterogeneity among both the study designs and reported outcomes within the studies we reviewed, we did not attempt a quantitative analysis of our results. Rather, we present our findings using Journal of Surgical Education  Volume ]/Number ]  ] 2014

descriptive statistics to describe educational QI content, teaching and instructional methods, and processes for QI assessment within the curricula we reviewed. As all data are previously published and publically available, the current study did not meet criteria for submission to our institutional review boards for approval. Data were managed using Excel (Redmond, WA; 2007).

RESULTS We identified 1155 articles for consideration based on our search terms, with 1076 excluded by title alone. Of 80 abstracts reviewed, 30 were excluded (for irrelevant content, population outside of the United States, or a lack of full-text availability), leaving 50 full-text articles included in the review (Fig. 1). The Cohen’s kappa score for reviewer agreement was 0.95. Study Features Characteristics of all included articles (n ¼ 50) are shown in Table 1. More than 50% of the articles (n ¼ 26) were published after 2009, and the majority of them (48%, n ¼ 24) described QI education efforts in nonsurgical fields.

Only 24% (n ¼ 12) were specific to surgery.11-22 A total of 31 curricular blueprints were identified.11,12,14,16-18,23-47 The remaining articles were opinion pieces (n ¼ 9),13,15,19,20,48-52 systematic reviews of preexisting literature (n ¼ 5),53-57 consensus statements (n ¼ 3),21,22,58 evaluation tools (n ¼ 1),59 or needs assessments (n ¼ 1).60 Characteristics of the overall curriculum blueprints (n ¼ 31) are outlined in Table 2. The majority (51.6%, n ¼ 16) of the curricula presented were designed and implemented in primary care residencies.25,27-30,32-35,39-41,44-47 Methodology available for use in continuous QI in medicine was the most common content presented, with an emphasis on plan, do, study, act (PSDA; n ¼ 12, 38.7%),12,17,23-25,27,29,33,34,40,43,44 root cause analysis (RCA; n ¼ 5, 16.1%),14,18,26,45,47 and lean thinking (n ¼ 2, 6.5%).30,46 Patient safety and error prevention and management were also featured amongst the curricula (29% and 12.9%, respectively). The majority of QI instruction was via lecture and didactics (83.9%), with fewer curricula utilizing small groups (19.4%), web-based modules (19.4%), QI projects (19.4%), and experiential teaching (16.1%). Only 22.6% of curricula used validated tool kits such as QAIC/ QIKAT61 and QIPAT59 for assessment; most (58.1%) relied on direct observation and feedback. Targeted learners were most often PGY2 residents (29.0%), while only 6 curricula

FIGURE 1. Literature search and study selection process for identifying a surgical QI curriculum. Journal of Surgical Education  Volume ]/Number ]  ] 2014

3

TABLE 1. Characteristics of Included Articles (n ¼ 50) Date of Publication 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 Specialty Surgery Nonsurgical Primary care Rads Critical care ER Generic GME Type Curriculum blueprint Opinion paper Systematic review Consensus statement Evaluation tool Needs assessment

n

(%)

2 4 4 7 9 7 6 1 3 4 3

(4.0) (8.0) (8.0) (14.0) (18.0) (14.0) (12.0) (2.0) (6.0) (8.0) (6.0)

12 24 20 2 1 1 14

(24.0) (48.0) (40.0) (4.0) (2.0) (2.0) (28.0)

31 9 5 3 1 1

(62.0) (18.0) (10.0) (6.0) (2.0) (2.0)

(19.4%) outlined a course for all residents within their respective programs.

Surgery-Specific Initiatives

within general surgery residencies. In response to this clearly unmet need, professional societies and other educational resources have released educational material and individual components of QI curricula for program directors to help facilitate implementation of a QI curriculum in surgery (Table 5). Nonsurgical Initiatives Details of nonsurgical QI curriculum blueprints are shown in Table A1 of Appendix.23-47 Table A2 of Appendix outlines nonsurgical opinion pieces, consensus statements, and needs assessments regarding the importance of QI TABLE 2. Characteristics of Curriculum Blueprints (n ¼ 31) Specialty

n

(%)

Primary care (adult þ pediatric) General surgery GME (generic) Radiology Critical care Emergency medicine QI content* Continuous QI methods: Plan, Do, Study, Act (PDSA) Root Cause Analysis (RCA) Lean Methodology DMAIC and Six-Sigma Nonspecific† Patient safety Error prevention and management QI instruction* Lecture/didactics Small group sessions Online/web-based modules QI projects Experiential/bedside teaching Independent study Not specifically mentioned QI assessment Direct observation/feedback Validated tools (QIKAT, QIPAT, etc.) Simulation/OSCE Not specifically mentioned Target learners Interns (PGY1) PGY2 PGY3 Laboratory residents Chief residents Fellows Optional/elective All residents within specific program

16 6 5 2 1 1

(51.6) (19.4) (16.1) (6.5) (3.2) (3.2)

12 5 2 1 11 9 4

(38.7) (16.1) (6.5) (3.2) (35.5) (29.0) (12.9)

26 6 6 6 5 1 3

(83.9) (19.4) (19.4) (19.4) (16.1) (3.2) (9.7)

18 7 1 5

(58.1) (22.6) (3.2) (16.1)

3 9 3 2 1 1 6 6

(9.7) (29.0) (9.7) (6.5) (3.2) (3.2) (19.4) (19.4)

Details of the QI curriculum blueprints described for General Surgery are shown in Table 3.11,12,14,16-18 On review, each of them detailed a rigorous platform for QI education; however, independently each was missing key concepts required for a comprehensive QI curriculum as determined by our focus groups. Although each of the 6 blueprints contained specific aspects of QI methodology such as PDSA and root cause analysis, the content lacked materials on important aspects of health care policy, value, patient experience, and appropriateness of care. Furthermore, instruction was mostly limited to didactics and lectures, whereas assessment was limited to QI project completion by the residents. It should be additionally noted that although the time, cost, and materials needed to implement these curricula vary, so do their target learners. Unfortunately, some of the most comprehensive curriculum described in the literature actually outlined only a singleresident or elective (nonmandatory) experience. Table 4 summarizes some of the consensus statements and opinion pieces written by educational experts within the field of surgery who clearly address the need for a QI curriculum.13,15,19-22 These statements have documented the overarching need for a comprehensive QI curriculum

DMAIC ¼ define, measure, analyze, improve, control; OSCE ¼ objectively structured clinical examination; PGY ¼ post-graduate year; QI ¼ quality improvement; QIKAT ¼ quality improvement knowledge assessment tool; QIPAT ¼ quality improvement proposal assessment tool. *Multiple curricula mention more than one type of content and instruction; numbers will be 4100%. † Run charts, tests of change, AIM statements.

4

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Journal of Surgical Education  Volume ]/Number ]  ] 2014

TABLE 3. Details of General Surgery QI Curriculum Blueprints (n ¼ 6)

Title

AIM

Content Type

Complete Target Instruction Assess- QI CurriType ment Type culum? Learners

Brewster LP. To create an Management None or not Management educational of Adverse described of adverse module that surgical surgical integrates events events: a clinical and structured technical skills education and module for emphasizes residents. Am interpersonal J Surg. and 2005;190 communica(5):687-90. tion skills and professionalism using a standardized approach Canal DF. To develop QI Continuous QI Didactics/ Practice-based curriculum for and PDSA lectures learning and laboratoryimprovement: year residents a curriculum in continuous quality improvement for surgery residents. Arch Surg. 2007 May;142 (5):479-82. Kauffmann RM. To demonstrate Patient Safety None or not The use of a that a and Root described multidisciplinmultidisciplinCause ary morbidity ary M&M can Analysis and mortality satisfy conference to ACGME core incorporate competencies ACGME general competencies. J Surg Educ. 2011 Jul-

Materials Needed

Time Allocation

Costs

Outcomes/ Satisfaction/ Assessment

Global No performance ratings and debriefing

Second-year Standardized 6 months residents patients, OR simulation, and simulation laboratory

QI projects, Yes Pretests and Posttests, and Surveys

Laboratory residents

None

Research years Time and buy- Improvement in from 4 domains of faculty QI: knowledge, experience, efficacy, and interest

None or not No described

All residents None

Quarterly 90- Time and buy- Evaluations at minute in from the end of conference faculty conference with completed meetings to by attendees prepare

Paying SPs Well received and cost of by residents simulation tools

5

6

TABLE 3 (continued)

Title

AIM

Content Type

Complete Target Instruction Assess- QI CurriType ment Type culum? Learners

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Aug;68 (4):303-8. Morales C.S. To demonstrate QI related to Independent QI projects Yes Performance that a QI local study, improvement: resident outcomes meeting Gettingan champion data attendance, early start. yielded etc. (no Journal of significant formal Surgical improvements instruction) Education in patient care (2012) 69:6 (774-779). Introduction Continuous Didactics/ QI projects Yes O0 Connor “Developing a of a QI, PDSA lectures and Practice-Based comprehensurveys Learning and sive PBLI Improvement curriculum Curriculum for (clinical an Academic decision General making, Surgery individual Residency” learning plan, 2010 JACS and QI) within the context of a clinical rotation for PGY2 residents Rosenfeld JC. A description Patient safety; None or not PracticeNo Using the piece Root cause described based Morbidity and describing analysis improveMortality how to modify ment log; conference to the M&M reviewed teach and by faculty format to assess the teach ACGME ACGME core General competencies Competencies. Curr Surg. 2005;62 (6):664-9.

Materials Needed

Single None resident champion during dedicated research year

Time Allocation

Costs

Outcomes/ Satisfaction/ Assessment

Dedicated Time and Numerous QI year; weekly buy-in from projects meetings faculty successfully with SCRs, implemented SCs, and coresidents

14 PGY2 residents over 2 years

Majority of 3-week Time and Increase in all curriculum curriculum þ buy-in from measured web based QI project faculty elements of QI knowledge; 14 QI projects developed

All chief residents

None

2-week Time and No data interval buy-in from presented between faculty case and M&M presentation to allow time for adequate preparation

TABLE 4. Summary of General Surgery QI Consensus Statements and Opinion Pieces (n ¼ 6) Title Darosa DA. Error training: missing link in surgical education. Surgery. 2012 Feb;151(2):139-45. Mery CM, et al. Teaching and assessing the ACGME core competencies in surgical residency. Bulletin ACS. 2008. Sachdeva “Educating surgery residents in patient safety” Surgical Clinics of North America. 2004 Sachdeva “Patient safety curriculum for surgical residency programs: Results of a national consensus conference” Surgery. 2007 Sachdeva "Surgical education to improve the quality of patient care: the role of practice-based learning and improvement." JGIS 2007 Welling “Graduate Medical Education as the Driver for Quality Improvement and Patient Safety: A National Initiative of Independent Academic Medical Centers” JSE 2009

initiatives within graduate medical education.48-52,58,60 Preexisting systematic reviews of the literature regarding the topic are listed in Table A3 of Appendix.53-57

Aim To present a proposed curriculum on error prevention and patient safety consisting of skills laboratory, simulation, and experiential learning To suggest methods for teaching and evaluating core competencies in surgery residency programs To give examples of how to educate residents in patient safety including teaching and evaluation methods To present recommendations of ACS/ASE conference and initiate the development of patient safety curriculum for general surgery residents To describe the role of surgical education in improving quality of patient care; no specific curriculum described; also not specific to residents Description of national initiative of the alliance of independent academic medical centers (AIAMC); 21 different institutions focused on 3 areas of patient safety (handoffs, infection control and medication safety) over 18-month project period

Surgical education has changed substantially over the past decade. A national curriculum was recently made available for use across all residency programs.62 The amount of material that must be covered to adequately prepare surgeons for practice often seems overwhelming to educators and trainees alike. Now, since the introduction of the curriculum, regulatory bodies have released new milestones and requirements for accreditation to place a greater emphasis on quality science and patient safety as well as value-based delivery of surgical services. In this article, we present a systematic review of the literature to identify a national quality improvement curriculum for adoption by program directors in surgery. We found that the individual elements for a comprehensive curriculum exist; however, they have yet to be combined into a single package that could be easily implemented at the national level. The combined surgery-specific materials define the content well, but there was not a single curriculum that outlined a comprehensive substance to educate residents about QI. The instructional materials lacked faculty development tools to train the trainer. The assessment was limited to a single instrument that has not yet been adapted for use in surgical education with occasional topic-specific MCQs that have not been validated across programs. There are a substantial amount of general QI materials available through organizations and societies. The IHI has a complete program that requires hours of time on top of the

full surgical curriculum and often goes beyond the level of knowledge needed by an individual generic resident to develop proficiency with these skills.63 Similar materials have been developed by the American College of Physicians64 and the University Healthsystem Consortium.65 Most surgery-specific blueprints were single-institution programs designed to prepare residents to complete an individual or team-based QI project. There is no doubt regarding the value of ensuring that every surgeon can complete their own QI project; however, this goal may be impossible to achieve especially at smaller programs with fewer resources. Moreover, in isolation, this approach separates the residents’ initiatives from the organizational goals and in so doing, may not satisfy the goal of engaging residents in the organizational targets for quality outlined in the CLER program. There are several barriers to a universal QI curriculum. Current surgeons have not been formally trained in QI and therefore do not all feel confident in their ability to educate their residents in these topics. The QI materials need to be integrated into what we already do to care for patients rather than added on as an afterthought if we are going to satisfy the requirements of the CLER program. The level of proficiency across residents should be flexible with a minimum level of competence achieved by all. To address these barriers, the ACS-NSQIP QITI was created.5-8 The initiative was first launched in 2011 and is a proactive response to the need to engage surgical residents in quality improvement. Through the collaboration of ACSNSQIP teaching hospitals, the QITI set forth to enable the easy manipulation of outcomes data to provide standardized resident-specific reports with benchmarking, to develop a surgery-specific QI curriculum, and to deliver faculty development opportunities to meet the challenges of the changing health care landscape. The multidisciplinary QITI team includes surgeons, nurses, and surgical residents from a diverse group of teaching hospitals. As such, the intent is to

Journal of Surgical Education  Volume ]/Number ]  ] 2014

7

DISCUSSION

8

TABLE 5. Summary of Society and Educational Resources with QI Education Materials Title

Source

Topic

Curriculum Type

Instruction Type

Assessment Type

Complete curriculum?

Online module; independent study and DVD

n/a

No

No method for assessment other than module completion

Online module; independent study n/a

n/a

No

n/a

No

Greenfield Ch 15 (Birkmeyer) QA in Surgery; QA Module x2; No assessment in QI Via ASE: Surgical Educator Handbook (2001); nothing specific about QI

Journal of Surgical Education  Volume ]/Number ]  ] 2014

ACS Division of Education

Society

Surgery

SCORE

Curriculum Portal

Surgery

APDS: Association for Program Directors in Surgery ASE: Association for Surgical Education SSH: Society for Simulation in Healthcare

Society

Surgery

QI, Patient Safety, and Error Prevention QI/QA; Six Sigma, RCA, and PDSA n/a

Society

Surgery

n/a

n/a

n/a

No

Society

General GME

n/a

Simulation

Video recordings; Debrief

No

Society

General GME

Self-study modules

n/a

No

University Chicago

General GME

QI and patient safety modules General QI methodology

Didactics/ lectures; independent study

Projects/ QIKAT

Yes

IHI: Institute for Healthcare Improvement QAIC/QIKAT Toolkit

Notes

Surgical Educator Handbook (2001); nothing specific about QI Unable to access complete website as a nonmember ($$$) but did not see anything about specific QI content No method for assessment other than module completion Complete curriculum that could be adapted for use in General Surgery residencies

produce a quality improvement curriculum to integrate surgical residents into local organizational QI efforts while teaching the principles that trainees would need following the transition to independent practice. The initiative is designed to equip all surgical residents with the skills needed to address QI needs of surgical patients to ensure that surgical outcomes continue to improve across generations. The ACS-NSQIP QITI aspires to develop a flexible patient safety and quality improvement curriculum that incorporates the sharing of hospital, resident, and teamspecific outcomes data with surgical residents and amongst participant hospitals enrolled in the QITI. Through the implementation of the QI curriculum, general surgery residents would learn how to interpret and use applicable outcomes data, participate in QI projects, and become engaged within their own institutions. As such, in combination with their other clinical skills, they would have the skills to improve the care provided to patients while in residency and be able to evaluate and improve the quality of care provided on transitioning to independent practice.

Journal of Surgical Education  Volume ]/Number ]  ] 2014

CONCLUSION The ACS-NSQIP QITI will build on the high-quality components identified in instruction and assessment and develop data-centered QI content to generate a comprehensive QI curriculum for use in graduate surgical education on a national level. The structure will permit the system to incorporate the knowledge into the existing educational platform and the process of caring for patients. Through these efforts and others, we hope to provide the substrate for a culture of continuous QI across all surgical disciplines.

ACKNOWLEDGMENT We would like to acknowledge all of the participants of the QITI for their tireless work in pursuit of a culture of continuous quality improvement.

9

10

TABLE A1. Details of Nonsurgical QI Curriculum Blueprints (n ¼ 6)

Title

Specialty

Buchmann R.F. Deloney L. Radiology A. Donepudi S.K. Mitchell C.M. Klein S. G. Development and Implementation of a Systems-based Practice Project Requirement for Radiology Residents. Academic Radiology (2008) 15:8 (10401045).

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Carey W.A. Colby C.E. Critical care Educating fellows in practice-based learning and improvement and systems-based practice: The value of quality improvement in clinical practice. Journal of Critical Care (2013) 28:1 (112.e1-112.e5).

Coleman MT, Nasraty S, Primary care Ostapchuk M, Wheeler S, Looney S, Rhodes S. Introducing practicebased learning and improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf. 2003;29(5):238-47.

AIM

Complete QI Assessment CurricuContent Type Instruction Type Type lum?

To describe a PDSA model design/ implementation of SBP project requirement for radiology residents to meet ACGME requirements and the goal of training residents to identify and solve stems problems that compromise their ability to provide the most efficient and effective patient care To describe the PDSA, DMAIC development of a QI-based fellowship curriculum to teach and assess PBLI and SBP in a critical care environment

Lectures (minimal) Evaluation of Yes and experiential PDSA/SBP via projects projects using QIPAT7 (QI proposal assessment tool)

Didactic þ Direct Yes experiential evaluation of (participation in portfolios institutional during Quality and monthly Safety meetings Committee) with fellowship director and during Quality and Safety meetings A curriculum to PDSA, run charts, Monthly one-hour Feedback Yes address histograms, sessions meetings and selected and tests of incorporated evaluation of ACGME change into preexisting QI projects competencies didactics by incorporating PBLI activities into the routine clinical work of family

Target Learners

Time Allocation

Costs

Outcomes/ Satisfaction/ Assessment

Second- and 2 years third-year residents (12 total)

Time, buy-in from faculty

QIPAT and evaluations by residents; high level of satisfaction

3 fellows this far

1 year

Time, buy-in from faculty

Improved patient outcomes (decreased CLABSIs, etc) and fellow satisfaction as well as presentation at national meetings

All residents

6-month cycles

Time, buy-in from faculty

Evaluations by residents/ faculty/staff overwhelmingly positive

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Cosby KS, Croskerry P. ER Patient safety: a curriculum for teaching patient safety in emergency medicine. Acad Emerg Med. 2003;10(1):69-78. Daniel DM, Casey DE Jr, Primary care Levine JL, et al. Taking a unified approach to teaching and implementing quality improvements across multiple residency programs: the Atlantic Health Experience. Acad Med. 2009;84 (12):1788-1795.

Diaz VA, Carek PJ, Primary care Dickerson LM, Steyer TE. Teaching quality improvement in a primary care residency. Jt Comm J Qual Patient Saf. 2010 Oct;36 (10):454-60.

Djuricich AM, Ciccarelli Primary care M, Swigonski NL. A continuous quality improvement curriculum for residents: addressing core competency, improving systems. Acad Med. 2004 Oct;79(10 Suppl):S65-7. Kim CS, Lukela MP, Primary care Parekh VI, Mangrulkar RS, Del Valle J, Spahlinger DA, Billi JE. Teaching internal medicine residents quality improvement and patient safety: a lean thinking approach.

medicine residents To propose a variety of approaches for teaching patient safety to ER residents To describe the planning process, the structure, and the effect of the use of a QI collaborative to effectively teach QI to residents in a multihospital, multiprogram system To describe a longitudinal scholarly activity curriculum that has been fully integrated into a communitybased universityaffiliated family medicine program To describe the development, implementation, and evaluation of a residency continuous QI curriculum

Patient safety/ error prevention (Swiss cheese model; RCA)

Didactics, small groups, videos, and reading materials

None described

No

All residents

Not described

Time, buy-in from faculty

Not described

PDSA

Didactic þ experiential

Monthly progress reports via face-to-face meetings

Yes

Selected resident champions

6 months

Time, buy-in from faculty

Improvement in patient outcomes (medication reconciliation) and resident awareness of active and ongoing QI

Generic overview Didactics, small of QI principles groups, web-based modules, and experiential modules

PDSA

Evaluation by Yes faculty QI experts and presentation at the departmental level

Didactic sessions, Projects graded Yes brainstorming using small groups, standardized and QI projects tool; pretests and posttests for residents

To outline an Lean thinking and Monthly interactive Feedback Yes educational adverse event small group meetings and curriculum to analysis sessions and evaluation of provide team projects QI projects foundational lead by faculty via knowledge in QI expert committee of QI and PS to QI experts all trainees

All second- and Annual curriculum Time, buy-in third-year cycle from faculty residents

Over 6 years, 21 total QI projects completed with multiple national presentations and publications

Resident on ambulatory rotations

Increased resident knowledge of QI based on posttest evaluations

One-month ambulatory block

Time, buy-in from faculty

Time, buy-in All first- and Monthly small from faculty second-year groups plus residents time to develop projects

Not described

11

12

TABLE A1 (continued)

Title

Specialty

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Am J Med Qual. 2010 May-Jun;25(3):211-7. Krajewski K. Radiology Siewert B. Yam S. Kressel H.Y. Kruskal J.B. A Quality Assurance Elective for RadiologyResidents. Academic Radiology (2007) 14:2 (239245). Primary care Kravet SJ, Howell E, Wright SM. Morbidity and mortality conference, grand rounds, and the ACGME0 s core competencies. J Gen Intern Med. 2006;21 (11):1192-4. Ogrinc G, Headrick L, Primary care Morrison L, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med 2004;19 (5Pt 2):496500. Oyler J, Vinci L, Arora V, Primary care Johnson J. Teaching internal medicine residents quality improvement techniques using the ABIM0 s practice improvement modules. J Gen Intern Med. 2008 Jul;23 (7):927-30. Peters A, Kimura J, Ladden Primary Care M, March E, Moore G. A self-instructional model to teach systemsbased practice and practice-based learning and improvement. J Gen Intern Med 2008;23 (7):931-6.

AIM

Complete QI Assessment CurricuContent Type Instruction Type Type lum?

To describe the QI methods and Didactic teaching, Residents Yes creation of risk self-learning via deliver QA/ elective that management online modules, QI lecture at familiarizes and practical grand rounds residents with experience along with the principles direct and practice feedback of quality sessions with assurance QI expert To describe how PBLI/PS Lecture-based at None No to use the grand rounds described ACGME core competencies as a structure for discussing conferences at M&M To describe the PBLI with PDSA design, implementation, and evaluation of a 4-week PBLI elective

Didactic sessions þ QIKAT QI projects

Yes

To describe the PDSA and test of Didactics þ web- Evaluation of Yes implementachange based American projects by tion of a Board of Internal QI longitudinal Medicine (ABIM) champions/ quality Clinical faculty assessment Preventive members as and Services Practice well as improvement Improvement successful curriculum Module (CPS completion of (QAIC) PIM) online modules To describe a Generic QI Mostly web-based Knowledge test Yes 4-week methodology modules with and selfclinical some didactics assessment elective that teaches residents SBP and PBLI with one week devoted specifically to QI

Target Learners

Time Allocation

Elective; one One-month resident at a elective time

Costs

Time, buy-in from faculty

Passive Preparation/case Time, buy-in involvement review by from faculty by all faculty for residents M&M who attend M&M

11 internal medicine residents

One-month Time, buy-in elective (halffrom faculty days) integrated into ambulatory rotation

Second-year residents

Two one-month blocks during ambulatory month

Optional elective

4-week blocks

Outcomes/ Satisfaction/ Assessment

Multiple QI projects implemented, presented, and published

Not described

Improved patient care via QI projects, improved QI knowledge (QIKAT), and overall satisfaction with elective $25/resident for Improved patient the ABIM PIM care and modules overall resident satisfaction

Time, buy-in from faculty

Increased resident knowledge and self-assessment of PBLI and SBP

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Reznek MA, Digiovine B, GME Kromrei H, Levine D, Wiese-Rometsch W, Schreiber M. Quality Education and Safe Systems Training (QuESST): Development and Assessment of a Comprehensive CrossDisciplinary Resident Quality and Patient Safety Curriculum. J Grad Med Educ. 2010 Jun;2(2):222-7. Schleyer AM, Best JA, GME McIntyre LK, Ehrmantraut R, Calver P, Goss JR. Improving Resident Engagement in Quality Improvement and Patient Safety Initiatives at the Bedside: The Advocate for Clinical Education (ACE). Am J Med Qual. 2012 Aug 20.

Shekter I, Nevo I, GME Fitzpatrick M, EverettThomas R, Sanko JS, Birnbach DJ. Creating a common patient safety denominator: the intern’s course. J Grad Med Educ. 2009;1 (2):269-272. Singh R, Naughton B, Primary care Taylor JS, et al. A comprehensive collaborative patient safety residency curriculum to address the ACGME core competencies. Med Educ. 2005;39 (12):1195-1204.

To describe the Quality and development Patient safety of a formal, standardized crossdisciplinary quality and patient safety curriculum for first-year residents

4-hour didactic session

Preintervention Yes and postintervention assessment

First-year residents

16 total hours of Time, buy-in teaching; time from faculty to prepare

Improved knowledge and satisfaction

(1) To describe Quality and Bedside evaluation Preintervention Yes self-reported Patient safety and teaching by survey practice QI RN followed by patterns for direct prespecified observation QI and PS and behaviors reporting of (Medicine outcomes and Surgery residents and attending surgeons) and (2) to describe the design and implementation of physiciancentered QI initiative of education and feedback at the bedside To describe an Communication, Lecture, interactive Direct Yes innovative teamwork, workshop, and evaluation patient safety patient web-based course for handoffs, hand didactics interns related hygiene, and to prevention prevention of of medical errors errors

Selected resident teams (not all encompassing)

4 months

0.5 FTE (RN)

Improved knowledge and satisfaction

All interns (multidisciplinary)

Three 3-hour courses

Time, buy-in from faculty

Postcourse surveys (95% found it to be beneficial)

To design and Patient safety and Workshops, Assessment of Yes implement a error didactics, small resident new patient prevention group exercises, performance safety and longitudinal throughout curriculum for QI project curricular family activities and medicine OSCE residents

46 family medicine residents

During previously Time, buy-in scheduled from faculty didactics

Improved patient outcomes and increased resident knowledge

13

14

TABLE A1 (continued)

Title

Specialty

Tomolo AM, Lawrence Primary care RH, Aron DC. A case study of translating ACGME practice-based learning and improvement requirements into reality: systems quality improvement projects as the key component to a comprehensive curriculum. Qual Saf Health Care. 2009;18 (3):217-224.

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Tomolo AM, Lawrence Primary care RH, Watts B, Augustine S, Aron DC, Singh MK. Pilot study evaluating a practice-based learning and improvement curriculum focusing on the development of system-level quality improvement skills. J Grad Med Educ. 2011 Mar;3(1):49-58.

Varkey P, Karlapudi S, GME Rose S, Nelson R, Warner M. A systems approach for implementing practicebased learning and improvement and systems-based practice in graduate medical education. Acad Med. 2009;84(3):335-9. Varkey P, Reller MK, Smith GME A, Ponto J, Osborn M. An experiential interdisciplinary quality improvement educational initiative.

AIM

Complete QI Assessment CurricuContent Type Instruction Type Type lum?

To summarize QI principles and Small groups, the tools (PDSA, didactics, implementarun charts, projects, and tion of PBLI Pareto charts, presentations curriculum to etc) address components of the ACGME requirement and to evaluate its effect on clinical practice To describe a Generic QI Four 2-hour curriculum methodology didactics that addresses (run charts, gaps in tests of change, content and aim statements, experiential etc) learning activities through didactics and participation in systemslevel QI projects that focus on making changes in health care processes Description of SBP and PBLI Didactic sessions an institutionwide curriculum to facilitate teaching and assessment of PBLI and SBP competencies among all residents Describes and PDSA, QI, and Lectures/didactics interprofespatient safety sional QI educational pilot

None described

No

Target Learners 87 internal medicine residents during rotation at the VAMC

Time Allocation

Costs

One afternoon/ Time, buy-in week x4 weeks from faculty

Outcomes/ Satisfaction/ Assessment Resident feedback/ evaluation of curriculum overall positive

Systems quality Yes improvement training and assessment tool (SQI TAT)

Mandatory for One 1/2 day a internal week for medicine 4 weeks residents on their ambulatory block

Time, buy-in from faculty

Increased knowledge and comfort levels with QI and ability to develop a QI project

Surveys of PDs Yes and residents

All residents

Time, buy-in from faculty

Not described

QiKAT

Elective rotation 4 weeks

Time, buy-in from faculty

QiKAT scores improved; residents satisfied

Yes

Regularly scheduled didactics þ 3-hour PD workshop

Journal of Surgical Education  Volume ]/Number ]  ] 2014

Am J Med Qual. 2006;21(5):317-322. Vinci LM, Oyler J, Johnson Primary care JK, Arora VM. Effect of a quality improvement curriculum on resident knowledge and skills in improvement. Qual Saf Health Care. 2010 Aug;19(4): 351-4. Voss JD, May NB, Primary care Schorling JB, et al. Changing conversations: teaching safety and quality in residency training. Acad Med. 2008;83 (11):1080-1087.

Weigel C, Suen W, Gupte Primary care G. Using Lean Methodology to Teach Quality Improvement to Internal Medicine Residents at a Safety Net Hospital. Am J Med Qual. 2013 Feb 4. Weingart SN, Tess A, Primary care Driver J, Aronson MD, Sands K. Creating a quality improvement elective for medical house officers. J Gen Intern Med. 2004;19 (8):861–867.

To assess the PDSA, etc Didactics, webQIKAT before Yes effectiveness based modules, and after of the quality and group curriculum assessment projects and improvement curriculum (QAIC) on resident knowledge and skill in QI To describe an Patient safety, Didactics/lectures Direct Yes experiential RCA and QI projects evaluation/ curriculum to observation teach SBP and of patient PBLI safety project competencies with faculty that use feedback quality and patient safety as an educational thread To develop a QI Lean methods Didactics and Direct Yes curriculum (process projects evaluation of using Lean mapping, value projects methodology vs. non-value for IM added residents activities) To report the QI basics (tests of Didactic þ format and change, RCA, experiential content of a etc) QI elective for medicine residents

None described

No

PGY2 residents Weekly 90Time, buy-in during minute sessions from faculty ambulatory blocks

Increased resident knowledge in QI

All residents

4 total weekly 3-hour seminars during ambulatory block

Improved patient outcomes; resident satisfaction

2nd and 3rd year residents

Planning time but Time, buy-in otherwise from faculty during previously scheduled didactics

Increased knowledge and comfort levels with QI and ability to develop a QI project

Second- and third-year residents

20 hours/week  Time, buy-in 3 weeks from faculty

No formal evaluation

Time, buy-in from faculty

15

TABLE A2. Summary of Nonsurgical QI Consensus Statements and Opinion Pieces (n ¼ 7) Title Huang GC, Newman LR, Tess AV, Schwartzstein RM. Teaching patient safety: conference proceedings and consensus statements of the Millennium Conference 2009. Teach Learn Med. 2011;23(2):172-8. Kirch D.G. Boysen P.G. Changing the culture in medical education to teach patient safety. Health Affairs (2010) 29:9 (1600-1604). Varkey P et al. A patient safety curriculum for graduate medical education: results from a needs assessment of educators and patient safety experts. Am J Med Qual. 2009;24(3):214-21. Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012 Jan;46(1):107-19. Tess AV, Yang JJ, Smith CC, Fawcett CM, Bates CK, Reynolds EE. Combining clinical microsystems and an experiential quality improvement curriculum to improve residency education in internal medicine. Acad Med. 2009 Mar;84(3):326-34. Morrison LJ, Headrick LA. Teaching residents about practice-based learning and improvement. Jt Comm J Qual Patient Saf. 2008 Aug;34(8):453-9. Chase SM, Miller WL, Shaw E, Looney A, Crabtree BF. Meeting the challenge of practice quality improvement: a study of seven family medicine residency training practices. Acad Med. 2011 Dec;86(12):1583-9.

Aim To develop and present concrete approaches to teach patient safety in undergraduate and graduate medical education To explore 5 factors critical to transforming the culture for patient safety and to reflect on one real-world example at UNC SOM Needs assessment (8 PDs, 10 PS experts, and 9 experts in education technology) reported to help describe a framework for the development of a PS curricula in GME To describe current efforts to teach/engage trainees in PS and QI, summarize progress to date, and to list recommendations for next steps to integrate PS and QI in medical education To describe how training core faculty in patient safety and QI will lead to increased numbers of residents participating in QI projects, improve resident engagement in QI work, improve patient outcomes, and improve the overall educational experience for residents To examine the efficacy of two different PBLI educational initiatives To explore key characteristics shaping the relative success or failure of QI efforts in 7 primary care and family medicine training sites

TABLE A3. Summary of Preexisting Systematic Literature Reviews (n ¼ 5) Title Boonyasai RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, Bass EB. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA. 2007;298(9):1023-1037. Ogrinc “A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review” Academic Medicine 2003 Patow “Residents’ Engagement in Quality Improvement: A Systematic Review of the Literature” Academic Medicine 2009 Windish “Methodological Rigor of Quality Improvement Curricula for Physician Trainees: A Systematic Review and Recommendations for Change” Academic Medicine 2009 Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010 Sep;85(9):1425-39.

16

Aim SR to determine the effectiveness of QI curricula and to determine whether teaching methods influence effectiveness (reviews curricula for UME/GME as well as RNs and MDs in practice) To create a framework for teaching knowledge and skills of PBLI to medical students/residents based on proven and effective strategies (review of articles 1996-2001) To identify QI initiatives in which there was active engagement of residents (1987-2008) To systematically determine whether published QI curricula for trainees adhere to QI guidelines and meet standards for study quality in medical education research (1980-2008) To systematically review published QI and PS curricula for medical students or residents or both to determine (1) educational content and teaching methods, (2) assess learning outcomes achieved, and (3) identify factors promoting or hindering implementation Journal of Surgical Education  Volume ]/Number ]  ] 2014

APPENDIX Details of QI curriculum blueprints see Tables A1-A3.

11. Brewster LP, Risucci DA, Joehl RJ, et al. Management

of adverse surgical events: a structured education module for residents. Am J Surg. 2005;190:687-690. 12. Canal DF, Torbeck L, Djuricich AM. Practice-based

REFERENCES 1. Accreditation Council for Graduate Medical Education.

Program Requirements for Graduate Medical Education in General Surgery. Available at: /http://www.acgme.org/ acgmeweb/Portals/0/PFAssets/ProgramRequirements/ 440_general_surgery_01012008_07012012.pdfS; 2012 Accessed 01.05.12. 2. GME Commission. Graduate medical education

financing: focusing on educational priorities. In: Report to the Congress: aligning incentives in medicare; 2010:103-125. 3. Accreditation Council for Graduate Medical Education.

Clinical Learning Environment Review (CLER) Program. Available at: /http://www.acgme-nas.org/cler. htmlS Accessed 15.04.13. 4. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next

GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051-1056. 5. Sellers MM, Reinke CE, Kreider S, et al. American

College of Surgeons NSQIP: quality in-training initiative pilot study. J Am Coll Surg. 2013;216(3):420-427. 6. Sellers M, Reinke C, Kreider S, et al. American

College of Surgeons national surgical quality improvement program’s quality in training initiative pilot study. J Surg Res. 2013;179(2):231. 7. Sakran JV, Hoffman RL, Ko C, Kelz RR. The ACS

NSQIp quality in-training initiative: educating residents to ensure the future of optimal surgical care. Bull Am Coll Surg. 2013;98:30-35. 8. Kelz RR, Sellers MM, Reinke CE, Medbery RL,

Morris J, Ko C. Quality in-training initiative-a solution to the need for education in quality improvement: results from a survey of program directors. J Am Coll Surg. 2013;217:1126-1132 [e5]. 9. Kohn LT, Corrigan JM, Donaldson MS. To Err is Hu-

man: Building a Safer Health System. Washington, DC: Institute of Medicine, National Academy Press; 2000. 10. Kern DE, Thomas PA, Hughes MT. Curriculum

learning and improvement: a curriculum in continuous quality improvement for surgery residents. Arch Surg. 2007;142:479-482 [discussion 82-3]. 13. Darosa DA, Pugh CM. Error training: missing link in

surgical education. Surgery. 2012;151:139-145. 14. Kauffmann RM, Landman MP, Shelton J, et al. The

use of a multidisciplinary morbidity and mortality conference to incorporate ACGME general competencies. J Surg Educ. 2011;68:303-308. 15. Mery CM, Greenberg JA, Patel A, Jaik NP. Teaching

and assessing the ACGME competencies in surgical residency. Bull Am Coll Surg. 2008;93:39-47. 16. Morales CS, Kontonicolas F, Volpe AA, Saldinger PF,

Fukumoto R. Performance improvement: getting an early start. J Surg Educ. 2012;69:774-779. 17. O’Connor ES, Mahvi DM, Foley EF, Lund D,

McDonald R. Developing a practice-based learning and improvement curriculum for an academic general surgery residency. J Am Coll Surg. 2010;210:411-417. 18. Rosenfeld JC. Using the morbidity and mortality

conference to teach and assess the ACGME general competencies. Curr Surg. 2005;62:664-669. 19. Sachdeva AK. Surgical education to improve the quality

of patient care: the role of practice-based learning and improvement. J Gastrointest Surg. 2007;11:1379-1383. 20. Sachdeva AK, Blair PG. Educating surgery residents

in patient safety. Surg Clin North Am. 2004;84: 1669-1698 [xii]. 21. Sachdeva AK, Philibert I, Leach DC, et al. Patient

safety curriculum for surgical residency programs: results of a national consensus conference. Surgery. 2007;141:427-441. 22. Welling R, Grannan K, Boberg J, Pierce-Boggs K,

Engel A. Graduate medical education as the driver for quality improvement and patient safety: a national initiative of independent academic medical centers. J Surg Educ. 2009;66:336-339. 23. Buchmann RF, Deloney LA, Donepudi SK, Mitchell

CM, Klein SG. Development and implementation of a systems-based practice project requirement for radiology residents. Acad Radiol. 2008;15:1040-1045.

Development for Medical Education: A Six-Step Approach. Baltimore, MD: Johns Hopkins University Press; 2010.

24. Carey WA, Colby CE. Educating fellows in practice-

Journal of Surgical Education  Volume ]/Number ]  ] 2014

17

based learning and improvement and systems-based

practice: the value of quality improvement in clinical practice. J Crit Care. 2013;28(112):e1-e5.

resident quality and patient safety curriculum. J Grad Med Educ. 2010;2:222-227.

25. Coleman MT, Nasraty S, Ostapchuk M, Wheeler S,

37. Schleyer AM, Best JA, McIntyre LK, Ehrmantraut R,

Looney S, Rhodes S. Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf. 2003;29:238-247.

Calver P, Goss JR. Improving resident engagement in quality improvement and patient safety initiatives at the bedside: the Advocate for Clinical Education (ACE). Am J Med Qual. 2013;28:243-249.

26. Cosby KS, Croskerry P. Patient safety: a curriculum

38. Shekhter I, Nevo I, Fitzpatrick M, Everett-Thomas R,

for teaching patient safety in emergency medicine. Acad Emerg Med. 2003;10:69-78.

Sanko JS, Birnbach DJ. Creating a common patient safety denominator: the interns’ course. J Grad Med Educ. 2009;1:269-272.

27. Daniel DM, Casey DE Jr., Levine JL, et al. Taking a

unified approach to teaching and implementing quality improvements across multiple residency programs: the Atlantic Health experience. Acad Emerg Med. 2009;84:1788-1795.

39. Singh R, Naughton B, Taylor JS, et al. A compre-

28. Diaz VA, Carek PJ, Dickerson LM, Steyer TE. Teaching

40. Tomolo AM, Lawrence RH, Aron DC. A case study of

quality improvement in a primary care residency. Jt Comm J Qual Patient Saf. 2010;36:454-460.

translating ACGME practice-based learning and improvement requirements into reality: systems quality improvement projects as the key component to a comprehensive curriculum. Qual Saf Health Care. 2009;18:217-224.

29. Djuricich AM, Ciccarelli M, Swigonski NL. A con-

tinuous quality improvement curriculum for residents: addressing core competency, improving systems. Acad Med. 2004;79:S65-S67.

hensive collaborative patient safety residency curriculum to address the ACGME core competencies. Med Educ. 2005;39:1195-1204.

41. Tomolo AM, Lawrence RH, Watts B, Augustine S,

internal medicine residents quality improvement and patient safety: a lean thinking approach. Am J Med Qual. 2010;25:211-217.

Aron DC, Singh MK. Pilot study evaluating a practicebased learning and improvement curriculum focusing on the development of system-level quality improvement skills. J Grad Med Educ. 2011;3:49-58.

31. Krajewski K, Siewert B, Yam S, Kressel HY, Kruskal

42. Varkey P, Karlapudi S, Rose S, Nelson R, Warner M.

30. Kim CS, Lukela MP, Parekh VI, et al. Teaching

JB. A quality assurance elective for radiology residents. Acad Radiol. 2007;14:239-245. 32. Kravet SJ, Howell E, Wright SM. Morbidity and

mortality conference, grand rounds, and the ACGME’s core competencies. J Gen Intern Med. 2006;21:1192-1194. 33. Ogrinc G, Headrick LA, Morrison LJ, Foster T.

Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19:496-500. 34. Oyler J, Vinci L, Arora V, Johnson J. Teaching

internal medicine residents quality improvement techniques using the ABIM’s practice improvement modules. J Gen Intern Med. 2008;23:927-930. 35. Peters AS, Kimura J, Ladden MD, March E, Moore

GT. A self-instructional model to teach systems-based practice and practice-based learning and improvement. J Gen Intern Med. 2008;23:931-936. 36. Reznek MA, Digiovine B, Kromrei H, Levine D,

Wiese-Rometsch W, Schreiber M. Quality education and safe systems training (QuESST): development and assessment of a comprehensive cross-disciplinary 18

A systems approach for implementing practice-based learning and improvement and systems-based practice in graduate medical education. Acad Med. 2009;84: 335-339. 43. Varkey P, Reller MK, Smith A, Ponto J, Osborn M.

An experiential interdisciplinary quality improvement education initiative. Am J Med Qual. 2006;21: 317-322. 44. Vinci LM, Oyler J, Johnson JK, Arora VM. Effect of a

quality improvement curriculum on resident knowledge and skills in improvement. Qual Saf Health Care. 2010;19:351-354. 45. Voss JD, May NB, Schorling JB, et al. Changing

conversations: teaching safety and quality in residency training. Acad Med. 2008;83:1080-1087. 46. Weigel C, Suen W, Gupte G. Using lean methodology

to teach quality improvement to internal medicine residents at a safety net hospital. Am J Med Qual. 2013;28:392-399. 47. Weingart SN, Tess A, Driver J, Aronson MD, Sands

K. Creating a quality improvement elective for medical house officers. J Gen Intern Med. 2004;19:861-867. Journal of Surgical Education  Volume ]/Number ]  ] 2014

48. Chase SM, Miller WL, Shaw E, Looney A, Crabtree

BF. Meeting the challenge of practice quality improvement: a study of seven family medicine residency training practices. Acad Med. 2011;86:1583-1589. 49. Kirch DG, Boysen PG. Changing the culture in

medical education to teach patient safety. Health Aff (Millwood). 2010;29:1600-1604. 50. Morrison LJ, Headrick LA. Teaching residents about

practice-based learning and improvement. Jt Comm J Qual Patient Saf. 2008;34:453-459. 51. Tess AV, Yang JJ, Smith CC, Fawcett CM, Bates CK,

review and recommendations for change. Acad Med. 2009;84:1677-1692. 57. Wong BM, Etchells EE, Kuper A, Levinson W,

Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85:1425-1439. 58. Huang GC, Newman LR, Tess AV, Schwartzstein

RM. Teaching patient safety: conference proceedings and consensus statements of the Millennium Conference 2009. Teaching and learning in medicine. 2011;23:172-178.

Reynolds EE. Combining clinical microsystems and an experiential quality improvement curriculum to improve residency education in internal medicine. Acad Med. 2009;84:326-334.

59. Leenstra JL, Beckman TJ, Reed DA, et al. Validation of a

52. Wong BM, Levinson W, Shojania KG. Quality

safety curriculum for graduate medical education: results from a needs assessment of educators and patient safety experts. Am J Med Qual. 2009;24:214-221.

improvement in medical education: current state and future directions. Med Educ. 2012;46:107-119. 53. Boonyasai

RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, Bass EB. Effectiveness of teaching quality improvement to clinicians: a systematic review. J Am Med Assoc. 2007;298:1023-1037.

54. Ogrinc G, Headrick LA, Mutha S, Coleman MT,

O’Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med. 2003;78:748-756. 55. Patow CA, Karpovich K, Riesenberg LA, et al. Resi-

dents’ engagement in quality improvement: a systematic review of the literature. Acad Med. 2009;84:1757-1764. 56. Windish DM, Reed DA, Boonyasai RT, Chakraborti

method for assessing resident physicians’ quality improvement proposals. J Gen Intern Med. 2007;22:1330-1334. 60. Varkey P, Karlapudi S, Rose S, Swensen S. A patient

61. Quality

Assessment and Improvement Curriculum (QAIC) Toolkit. Department of Medicine Quality Improvement Initiatives. The University of Chicago Medical Center. Available at: /http://medqi.bsd.uchi cago.eduS Accessed online May 2013.

62. Surgical Council on Resident Education (SCORE). Res-

ident Portal. Available at: /http://www.surgicalcore. org/S Accessed March 2013.

63. Institute of Healthcare Improvement (IHI). Available at:

/http://www.ihi.orgS Accessed March 2013.

64. American College of Physicians Center for Quality. Avail

able at: /http://www.acponline.org/running_practice/ quality_improvement/S Accessed May 2013.

C, Bass EB. Methodological rigor of quality improvement curricula for physician trainees: a systematic

65. University Healthsystem Consortium (UHC). Available

Journal of Surgical Education  Volume ]/Number ]  ] 2014

19

at: /www.uhc.eduS Accessed online March 2013.