- Email: [email protected]
The American Board of Emergency Medicine Maintenance of Certification Summit
The Journal of Emergency Medicine, Vol. 49, No. 5, pp. 722–728, 2015 Copyright Ó 2015 Elsevier Inc. Printed in the USA. All rights reserved 0736-4679/$ - see front matter
http://dx.doi.org/10.1016/j.jemermed.2015.06.058
Education THE AMERICAN BOARD OF EMERGENCY MEDICINE MAINTENANCE OF CERTIFICATION SUMMIT Francis L. Counselman, MD,* Michael L. Carius, MD,† Terry Kowalenko, MD,‡ Nicole Battaglioli, MD,§ Cherri Hobgood, MD,k Andy S. Jagoda, MD,{ Elise Lovell, MD,††‡‡ Lillian Oshva, MD,§§ Anant Patel, DO,kk Philip Shayne, MD,{{ Jeffrey A. Tabas, MD,*** and Earl J. Reisdorff, MD**** *Department of Emergency Medicine, Eastern Virginia Medical School, Norfolk, Virginia, †Department of Emergency Medicine, Norwalk Hospital, Norwalk, Connecticut, ‡Department of Emergency Medicine, Oakland University William Beaumont School of Medicine, Royal Oak, Michigan, §Department of Emergency Medicine, York Hospital, York, Pennsylvania, kDepartment of Emergency Medicine, Indiana University School of Medicine, {Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai School of Medicine, New York, New York, ††Department of Emergency Medicine, University of Illinois, Chicago, Illinois, ‡‡Department of Emergency Medicine, Advocate Christ Medical Center, Oak Lawn, Illinois, §§Department of Emergency Medicine, CTSI Scientific Review Committee (SRC), New York University, New York, New York, kkJohn Peter Smith Health Network, Fort Worth, Texas, {{Department of Emergency Medicine, Education Program, Emory University School of Medicine, Atlanta, Georgia, ***Department of Clinical Emergency Medicine, University of California, San Francisco School of Medicine, San Francisco, California, and ****American Board of Emergency Medicine Reprint Address: Earl J. Reisdorff, MD, American Board of Emergency Medicine, 3000 Coolidge Road, East Lansing, MI 48823
, Abstract—Background: The American Board of Emergency Medicine (ABEM) convened a summit of stakeholders in Emergency Medicine (EM) to critically review the ABEM Maintenance of Certification (MOC) Program. Objective: The newly introduced American Board of Medical Specialties (ABMS) 2015 MOC Standards require that the ABMS Member Boards, including ABEM, ‘‘engage in continual quality monitoring and improvement of its Program for MOC .’’ ABEM sought to have the EM community participate in the quality improvement process. Discussion: A review of the ABMS philosophy of MOC and requirements for MOC were presented, followed by an exposition of the ABEM MOC Program. Roundtable
discussions included strengths of the program and opportunities for improvement; defining, teaching, and assessing professionalism; identifying and filling competency gaps; and enhancing relevancy and adding value to the ABEM MOC Program. Conclusions: Several suggestions to improve the ABEM MOC Program were discussed. ABEM will consider these recommendations when developing its next revision of the ABEM MOC Program. Ó 2015 Elsevier Inc.
Conflicts of Interest: Francis L. Counselman, MD, Michael L. Carius, MD, and Terry Kowalenko, MD, are Directors for the American Board of Emergency Medicine (ABEM). Earl J. Reisdorff, MD is employed by the ABEM. The opinions expressed in this paper are endorsed by the American Academy of Emergency Medicine (AAEM), AAEM/Resident and
Student Association, ABEM, American College of Emergency Physicians, and the Emergency Medicine Residents’ Association. As an accrediting body, the Resident Review Committee for Emergency Medicine and the Accreditation Council for Graduate Medical Education do not take stances on other organizations’ positions.
, Keywords—maintenance of certification; quality improvement; American Board of Emergency Medicine (ABEM)
RECEIVED: 2 March 2015; FINAL SUBMISSION RECEIVED: 9 June 2015; ACCEPTED: 23 June 2015 722
ABEM MOC Summit
723
INTRODUCTION On October 14–15, 2014, the American Board of Emergency Medicine (ABEM) convened the ABEM Maintenance of Certification (MOC) Summit, composed of representatives from the major emergency medicine stakeholder organizations (Table 1). Letters and a preliminary agenda were sent to leaders (e.g., President, Chief Executive Officer) of each organization, inviting them to participate and to send a representative to the meeting. The goals of the Summit included disseminating information about the American Board of Medical Specialties (ABMS) and the ABEM MOC Program and soliciting feedback from the emergency medicine community. Specifically, ABEM wished to gather ideas about how to improve the ABEM MOC Program.
in medical practice, some form of continuous professional development is necessary to meet a physician’s educational needs and public expectations. The ABEM MOC Program assures the public that the participating emergency physician is engaged in a rigorous continuous professional development process that adheres to a national, specialty-specific standard. The ABMS approved the MOC 2015 Standards in January 2014. The Standards require that the ABMS Member Boards, including ABEM, ‘‘engage in continual quality monitoring and improvement of its Program for MOC .’’ (1). The ABMS aspires to create MOC programs that are trusted by the public and believed in by the profession. A key to realizing this vision is to create MOC programs that are highly relevant to clinicians. History of the ABEM MOC Program
The ABMS and MOC The ABMS is a federation of 24 medical specialty certification boards, representing 37 specialties and 123 subspecialties. The ABMS believes that if the medical profession acts in benevolence to the public, society will continue to grant the medical profession substantial autonomy to determine educational standards, selfassess, and self-regulate; MOC is one manifestation of that public covenant. Structured professional learning takes place during the 4 years students spend in medical school and the 3–6 years physicians spend in residency. MOC supports the physician throughout the 30–40 years of a subsequent medical career. Given the dynamic changes that occur
Since its inception in 1979, ABEM has always required recertification with a time-limited, 10-year certificate. In 2004, this recertification requirement transitioned into Emergency Medicine Continuous Certification, now called the ABEM MOC Program. The four components of the ABEM MOC Program are outlined in Table 2. Since its beginning in 2004, the ABEM MOC Program has undergone multiple refinements, largely based on feedback from diplomates. Originally, there were 16–20 readings for each Lifelong Learning and SelfAssessment (LLSA) test; this number was reduced to 10–15 readings. The number of test questions correspondingly declined from 30–40 questions to 20–30.
Table 1. Participating Organizations and Representatives Organization American Academy of Emergency Medicine American Academy of Emergency Medicine, Resident and Student Association Association of Academic Chairs of Emergency Medicine American Board of Emergency Medicine
American Board of Medical Specialties American College of Emergency Physicians Council of Emergency Medicine Residency Directors Emergency Medicine Residents’ Association Residency Review Committee for Emergency Medicine Society for Academic Emergency Medicine
Representative Lillian Oshva, MD, Member, Education Committee, American Academy of Emergency Medicine Nicole Battaglioli, MD, Chair, Membership Committee Andy S. Jagoda, MD, President, Association of Academic Chairs of Emergency Medicine Francis L. Counselman, MD, President Michael L. Carius, MD, Secretary-Treasurer, and Chair, Maintenance of Certification Committee Terry Kowalenko, MD, Member-at-Large, and Vice-Chair, Maintenance of Certification Committee Earl J. Reisdorff, MD, Executive Director Mira Irons, MD, Vice President for Academic Affairs (presenter only) Jeffrey A. Tabas, MD, Chair-elect, Education Committee, American College of Emergency Physicians Elise Lovell, MD, Member-at-Large, Council of Emergency Medicine Residency Directors Anant Patel, DO, Speaker of the Council, Emergency Medicine Residents’ Association Philip Shayne, MD, Chair, Residency Review Committee for Emergency Medicine, Accreditation Council for Graduate Medical Education Cherri Hobgood, MD, Past President
724
F. L. Counselman et al.
Table 2. ABEM MOC Program Requirements Component Part 1: Professional Standing
Part 2: Lifelong Learning and Self-Assessment (LLSA)
Part 3: Cognitive Expertise (Continuous Certification Examination—ConCertÔ) Part 4: Assessment of Practice Performance (APP)
Description Physicians must hold a current, active, valid, full, unrestricted, and unqualified license in the United States, a US territory, or Canada. Every medical license held by the physician must meet the criterion above. LLSA activities are composed of an annual set of 10–15 readings followed by an open-book, online test over the readings. A Patient Safety LLSA is required in the first 5 years of a certification cycle. An annual average of 25 AMA PRA Category 1Ô credits must be completed. Of these, eight must be self-assessment (i.e., involving a pretest or posttest). ConCert is a secure, high-stakes, proctored examination of 205 single, best-answer items. The first component is the completion and attestation of a Practice Improvement (PI) activity, typically a quality improvement process. The second component is the participation in a communicationprofessionalism (CP) activity, typically a patient satisfaction survey.
Requirement Cycle This requirement is continuous. Failure to meet this requirement can result in decertification at any time.
Four LLSAs are required in the first 5-year cycle, one of which must be the Patient Safety LLSA. Four LLSAs are required in the second 5-year cycle. CME activity must be reported every 5 years.
The ConCert examination must be taken and passed in the second 5-year certification cycle. One PI activity and one CP activity are required in the first 5-year cycle. One PI activity and one CP activity are required in the second 5-year cycle.
ABEM = American Board of Emergency Medicine; MOC = maintenance of certification; CME = continuing medical education; AMA PRA = American Medical Association Physician’s Recognition Award. More complete descriptions are available on the ABEM website.
The score for passing the posttest was changed from 90% to 85%. The Assessment of Practice Performance (APP) Practice Improvement (PI) component originally required the physician to have 10 patients in an initial sample, and 10 patients in a second sample. This requirement would be difficult for many physicians to meet for certain conditions such as sepsis and acute myocardial infarction. Physicians can now use a lower number of patients and group or department data for high-acuity, lowfrequency conditions. DISCUSSION Attitudes about ABEM MOC There is a high rate of recertification through the ABEM MOC Program, which might reflect general acceptance of the requirements as well as other compelling external factors that lead to maintaining board certification. Of the 2455 diplomates with 2013 certification expiration dates, 1877 (76%) renewed certification in or before 2013. This is consistent with historical trends, including during the years prior to the institution of MOC in 2004. There tends to be an annual rate of about 4% of diplomates who regain certification each year. So, in 2013 there was an approximately 80% net retention of certification for physicians in the terminal year of certification. Of the total diplo-
mate pool at the end of the 2013 (31,154 diplomates), only 593 certifications (<2% of all certifications) lapsed. Summit participants viewed the ABEM MOC Program as being highly relevant. The purpose of the program is easy to understand. It attempts to address changes in the science of emergency medicine. It provides a mechanism to identify physicians who have competencies that have substantially diminished. The MOC Program emphasizes medical knowledge, cognitive skills, clinical skills, and patient safety. Certification and MOC were thought to be increasing in importance, especially because practice opportunities for noncertified physicians have been declining (2). Summit participants thought that MOC requirements are clearly communicated. The MOC Web site, which shows diplomate-specific MOC requirements, was favorably reviewed. The physician’s MOC Requirements and Status page brings clarity to the system. It was the general consensus that ABEM achieves a balance between having a sufficiently rigorous program while avoiding onerous requirements. In addition, the responsiveness of ABEM to diplomate feedback was described favorably; specifically, the manner in which the LLSA had evolved. Regarding the cost of MOC, there are fees associated with LLSA activities and the Continuous Certification (ConCertÔ) Examination. The annualized costs of
ABEM MOC Summit
MOC are $265 (excluding the cost of the optional continuing medical education [CME] activity), which are at the 50th percentile compared to all other ABMS Member Boards. The cost of MOC is <0.1% of the average diplomate’s annual compensation (3). Of note, in 2013, board-certified emergency physicians received, on average, an additional $34,800 in total compensation compared to non-board-certified physicians (3). Attitudes about Components of MOC Professionalism. The ABEM MOC Program has a ‘‘professional standing’’ requirement based on medical licensure (Table 3). The ABMS MOC 2015 Standards require that ABEM define professional expectations, and teach and assess professionalism. Participants explored the complex and enigmatic nature of defining medical professionalism. Measuring professionalism is difficult in any circumstance. Medical students and residents are in a supervised environment where faculty and program directors can assess ethical and professional behavior and provide feedback. However, once a physician is practicing independently, evaluating professionalism in the workplace is challenging. Assessing professionalism might be accomplished by requesting an attestation by the department director that the physician is ‘‘practicing medicine to acceptable standards of professionalism’’ or by a self-examination and attestation by the diplomate. A next step for ABEM is to identify those attributes that characterize professionalism and define professional behaviors. Lifelong learning and self assessment (LLSA). The primary goal of LLSA activities is to promote continuous learning by diplomates. Diplomates are required to pass a series of LLSA tests, one of which must be the new Patient Safety LLSA. They must also report completing an established number of CME credits. Each LLSA test includes an optional CME activity. A majority of physicians who complete the LLSA activity opt for CME credit, and all of these diplomates have been surveyed. Respondents report that the LLSA
725
activity is relevant to clinical practice: 70% reported that the readings were very relevant to significantly relevant; 28% thought the articles had some relevancy; and 2% thought there was little to no relevancy. Further, 92% of respondents thought the information in the articles would, to some degree, change their practices (4). In 2013, 23,576 diplomates were expected to complete an LLSA activity. There were 22,904 diplomates (97%) who passed LLSA tests. Of diplomates who passed the ConCert examination in 2013, only five did not renew their certification as a consequence of not meeting LLSA requirements. Some participants thought the bar for passing LLSA tests was too low, and was vulnerable to physicians ‘‘gaming the system.’’ Participants thought that the additional $30 fee for the LLSA CME activity was a good value. ABEM was encouraged to promote the LLSA as a CME activity by making more LLSA activities available, including subspecialty LLSAs. The value of the LLSA could also be enhanced by making the readings freely available for download from the ABEM Web site. It was thought that the ABEM MOC Program was effective at delivering new content to practitioners, but there could be an opportunity to make this more contemporary. There is a roughly 18-month lag between the time that readings are selected for inclusion for an LLSA and the time when diplomates have access to the LLSA activity. Assessment of cognitive expertise: the ConCert examination. Summit participants acknowledged that society highly values the requirement of a high-stakes, secure examination to ensure that a physician is ‘‘keeping up’’ with advances in a specialty, but thought that the ConCert Examination was the least relevant component of the ABEM MOC Program. Summit participants noted that some questions on the examination could seem obtuse and focused on minutiae. On occasion, questions seemed to have more than one correct answer, and the wording of certain items could be more straightforward. Survey data of physicians taking the 2014 ConCert Examination showed that 82% of respondents thought that all or most questions were relevant to physicians’
Table 3. Proceeding Summary and Recommendations
In general, key Emergency Medicine stakeholder organizations support the ABEM MOC Program The ABEM MOC Program is relevant to the majority of emergency physicians The ABEM MOC Web site and physician Personal Page simplify a complex set of requirements ABEM should continue to develop evidence supporting MOC Diplomates should be informed about the process that is used to score the examination. This would include a description of field testing processes that are used to ensure fairness and address test questions that seem to be esoteric or require further editing ABEM should continue to enhance the relevancy of all components of MOC ABEM should more effectively communicate the costs and value of MOC ABEM should define its standards for professionalism and increase awareness about these professionalism standards ABEM should develop MOC processes that better address competency gaps
ABEM = American Board of Emergency Medicine; MOC = maintenance of certification.
726
practices; 18% reported that most of the questions were not relevant (5). Although the overall relevancy of the items was regarded as good, there is an opportunity to enhance relevancy. ABEM notes that there are field test items on the ConCert, and that those questions might be the ones that test takers find confusing. Field test items are not scored; only ‘‘active items’’ are scored after they have demonstrated a testing performance profile that meets ABEM standards, including difficulty and discrimination. This profile is initially determined through field testing, and reassessed with every use on an examination. ConCert test takers can write comments about any items. All test-taker comments are collated and reviewed by ABEM staff and ConCert Exam editors, as well as the Chair of the Test Administration Committee, who is a clinically active emergency physician. No item is scored until all candidate comments are read and a satisfactory testing profile is obtained. ABEM could enhance the effectiveness of feedback to ConCert test takers, especially in terms of providing timely responses that are specific and actionable. Currently, ABEM provides a key word score report to physicians on items that are missed. The degree to which a physician who passes the ConCert will develop a performance improvement plan based on the examination’s score report is uncertain. Another consideration is that ABEM must balance the security of examination items against providing more detailed information about individual questions that are missed. It was acknowledged that significant learning occurs during the process of preparing for the ConCert Examination. An ABEM survey of those taking the 2014 ConCert revealed that 98% of diplomates prepare, in some fashion, for the examination (5). This preparation was thought to be more instructive than the score reports they receive from ABEM after the examination. A potential indicator that the ConCert Examination is clinically relevant is that physician performance on the examination is maintained over decades of practice (6). This would be unlikely if the content of the examination was not relevant to the clinical practice of Emergency Medicine. The pass rate over the last 5 years (2010– 2014) has ranged from 95% to 97%. Summit participants stressed that the ConCert Examination is an insufficient surrogate for assessing the composite skills needed to be a competent emergency physician. Although the examination measures judgment, diagnostic reasoning, synthesis of clinical data, and medical knowledge, it does not measure other competencies, such as communication skills and procedural skills. The secure examination experience at Pearson VUE testing centers was characterized as ‘‘easy, professional, and benign.’’ The amount of time allotted for testing was described as sufficient. There was an individual
F. L. Counselman et al.
concern that biometric palm scanning could be perceived as a personal invasion. However, a survey of 2014 ConCert test takers showed that only 1% thought the Pearson VUE processes were objectionable. There is an opportunity to better explain the cost and value of the ConCert Examination. It was thought that understanding the annualized cost of MOC might create greater acceptance of that cost. Despite the large payout every 10 years for the ConCert Examination, participants voiced no interest in annualizing fees. There was a discussion about the ConCert Examination and the degree to which it should be summative or formative. ABEM currently designs the ConCert Examination to be a summative assessment. As a high-stakes process, the physician can lose certification as a consequence of failing the examination. The notion was raised that a less comprehensive, more frequent testing format (e.g., every 2 years) might have advantages. Summit participants also explored the concept of a modular examination (a test in which physicians can select specific content areas). This might be potentially applicable for physicians working in emergency departments (EDs) that do not treat children. Arguments against a modular examination included that 1) on occasion, children are cared for in general (adult) EDs; 2) emergency physicians tend to be generalists; and 3) emergency physicians do not select the patients they see. Furthermore, certification is intended to be portable—ABEM certification is a national standard for the entire specialty. Assessment of practice performance (APP). The relevance of the APP PI requirement is determined by the physician or local ED. An inventory of 2013 APP PI attestations shows the top five quality activities to be: percutaneous coronary intervention (door-to-balloon times), door-to-doctor times, throughput time measures, aspirin for acute myocardial infarction, and sepsis care. These five activities accounted for 54% of all reported APP PI quality improvement activities (7). Approximately 5894 diplomates could have been expected to complete APP activities in 2013. There were 9348 (159% of required) attestations for PI activities, and 8137 (138%) attestations for the Communication/ Professionalism requirement. Although ABEM has aligned APP requirements with physician clinical and quality activities, Summit participants challenged APP PI activities as being ‘‘checkbox’’ requirements. Quality improvement activities and monitoring are ubiquitous in EDs. ABEM’s choice to align the APP PI requirements with existing activities that emergency physicians are performing might make ABEM’s approach to this requirement feel perfunctory.
ABEM MOC Summit
There were divergent opinions about whether APP activities—specifically, the PI component—should have a greater focus on individual competency assessment, or on a systems-based, department-wide assessment. Currently, the ABEM MOC Program allows both individual and department-based activities. Further, ABEM joined the ABMS Multi-specialty Portfolio Project, which provides APP PI credit for institutional and multidisciplinary quality improvement activities (8). Additional Suggestions Using MOC to identify competency gaps. Although gap identification already occurs to some degree within the ABEM MOC Program, it is on a fairly broad basis, not an individual physician basis. Further, ABEM’s examination processes can only measure selected competencies, not every competency required to be an emergency physician. Approaching gap identification in a broad fashion is supported by the assumption that emergency physicians are, for the most part, generalists throughout their careers. As major and important new advances are introduced into clinical practice, ABEM MOC might have a unique opportunity to introduce them to the practicing physician. Examples include bedside ultrasonography or video laryngoscopy. It was suggested that The Joint Commission’s Ongoing Professional Practice Evaluation (OPPE) and Focused Professional Practice Evaluation (FPPE) are designed to find gaps in physician performance. One idea worthy of further exploration was integrating elements of OPPE/FPPE into the ABEM MOC Program. Further, ABEM could be more intentional about first identifying a gap and then finding an LLSA reading to fill that gap. Participants discussed the possibility of providing feedback that would assist individual physicians identify knowledge gaps. This presents several challenges, one of which is that the ConCert Examination would need to contain many more items. The current examination ‘‘blueprint’’ requires a limited number of questions in several content areas, and the examination design uses a ‘‘survey examination’’ approach, in which selected items serve as a proxy for the ‘‘universe of knowledge’’ within the specialty. In its current form, the number of questions in some areas on the ConCert Examination are insufficient to accurately determine individual gaps. Physicians have limited proficiency with selfassessment (9). In fact, for physicians with the greatest competency gaps, self-awareness might have limited impact. Despite the inability of physicians to effectively self-assess, ABEM might still wish to develop an effective self-assessment inventory. Adding value and enhancing relevancy. ABEM monitors the relevancy of its MOC Program, and Summit partic-
727
ipants encouraged ABEM to continue doing so. The Longitudinal Study of Emergency Physicians could be better used to monitor physician perceptions of the ABEM MOC Program. Relevancy could also be further enhanced by integrating emerging issues in a more timely way. Balancing a more immediate presentation of information with maintaining standards and avoiding practices and trends that might quickly fade will be challenging. It was cautioned that the ‘‘brand’’ of ABEM will be diminished if ABEM presents poorly vetted information. Resident awareness. There is an opportunity to bring information about the career-long commitment of MOC to physicians in training. The degree to which residents need to know the details of the ABEM MOC Program is limited during their period of training, but the program’s structure can be shared. CONCLUSIONS Summit participants supported the convening of the MOC Summit, with the resultant community-based conversation through which the ABEM MOC Program could be improved. Also, having the ABMS participate was thought to be valuable. The representatives from the Emergency Medicine Residents’ Association and the Resident and Student Association of the American Academy of Emergency Medicine thought there was an opportunity to more effectively share information with residents. Participation by the two resident representatives was especially appreciated. ABEM encouraged all participants to share the information from the Summit with their sponsoring organizations. Although there were several suggestions for improvement (Table 3), there was substantial support for the ABEM MOC Program from Summit participants. General suggestions for improvement included developing greater evidence that MOC improves physician performance, patient outcomes, and patient safety. Finally, it was recommended that ABEM share even more information about MOC costs, relevance, and the operational details of the program. Acknowledgment—The authors wish to thank Ms. Frances M. Spring for her assistance in the preparation of this manuscript.
REFERENCES 1. American Board of Medical Specialties (ABMS). The ABMS MOC 2015 Standards. Available at: http://www.abms.org/media/1109/ standards-for-the-abms-program-for-moc-final.pdf. Accessed July 10, 2015.
728 2. Katz B. Top 10 best, worst places for emergency physicians to seek jobs in 2014–2015. ACEP Now, September 12, 2014. Available at: http://www.acepnow.com/article/top-10-best-worst-places-emergencyphysicians-seek-jobs-2014-2015/. Accessed July 5, 2015. 3. American College of Emergency Physicians (ACEP). 2013 National emergency medicine salary survey: clinical results. Emergency medicine compensation & benefit surveys. Dallas: American College of Emergency Physicians and Daniel Stern and Associates; 2013. 4. Jones JH, Smith-Coggins R, Meredith JM, Korte RC, Reisdorff EJ, Russ CM. Lifelong learning and self-assessment is relevant to emergency physicians. J Emerg Med 2013;45:935–41. 5. Marco CA, Wahl RP, Counselman FL, Heller BN, Harvey AL, Joldersma KB, Kowalenko T, Reisdorff EJ. The American Board of Emergency Medicine ConCert examination: Emergency physicians’ perceptions of clinical relevance and benefits. Acad Emerg Med 2015. Accepted for publication.
F. L. Counselman et al. 6. Marco CA, Counselman FL, Korte RC, Russ CM, Whitley C, Reisdorff EJ. Emergency physicians maintain performance on the American Board of Emergency Medicine continuous certification (ConCert) examination. Acad Emerg Med 2014;21: 532–7. 7. Kowalenko T, Carius ML, Korte RC, Miller MC, Reisdorff EJ. Emergency department quality improvement activity: an inventory from the American Board of Emergency Medicine maintenance of certification program. Acad Emerg Med 2015;22: 367–72. 8. American Board of Medical Specialties (ABMS). Multi-specialty MOC portfolio approval program. Available at: http://mocport folioprogram.org. Accessed November 15, 2014. 9. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–102.
http://dx.doi.org/10.1016/j.jemermed.2015.06.058
Education THE AMERICAN BOARD OF EMERGENCY MEDICINE MAINTENANCE OF CERTIFICATION SUMMIT Francis L. Counselman, MD,* Michael L. Carius, MD,† Terry Kowalenko, MD,‡ Nicole Battaglioli, MD,§ Cherri Hobgood, MD,k Andy S. Jagoda, MD,{ Elise Lovell, MD,††‡‡ Lillian Oshva, MD,§§ Anant Patel, DO,kk Philip Shayne, MD,{{ Jeffrey A. Tabas, MD,*** and Earl J. Reisdorff, MD**** *Department of Emergency Medicine, Eastern Virginia Medical School, Norfolk, Virginia, †Department of Emergency Medicine, Norwalk Hospital, Norwalk, Connecticut, ‡Department of Emergency Medicine, Oakland University William Beaumont School of Medicine, Royal Oak, Michigan, §Department of Emergency Medicine, York Hospital, York, Pennsylvania, kDepartment of Emergency Medicine, Indiana University School of Medicine, {Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai School of Medicine, New York, New York, ††Department of Emergency Medicine, University of Illinois, Chicago, Illinois, ‡‡Department of Emergency Medicine, Advocate Christ Medical Center, Oak Lawn, Illinois, §§Department of Emergency Medicine, CTSI Scientific Review Committee (SRC), New York University, New York, New York, kkJohn Peter Smith Health Network, Fort Worth, Texas, {{Department of Emergency Medicine, Education Program, Emory University School of Medicine, Atlanta, Georgia, ***Department of Clinical Emergency Medicine, University of California, San Francisco School of Medicine, San Francisco, California, and ****American Board of Emergency Medicine Reprint Address: Earl J. Reisdorff, MD, American Board of Emergency Medicine, 3000 Coolidge Road, East Lansing, MI 48823
, Abstract—Background: The American Board of Emergency Medicine (ABEM) convened a summit of stakeholders in Emergency Medicine (EM) to critically review the ABEM Maintenance of Certification (MOC) Program. Objective: The newly introduced American Board of Medical Specialties (ABMS) 2015 MOC Standards require that the ABMS Member Boards, including ABEM, ‘‘engage in continual quality monitoring and improvement of its Program for MOC .’’ ABEM sought to have the EM community participate in the quality improvement process. Discussion: A review of the ABMS philosophy of MOC and requirements for MOC were presented, followed by an exposition of the ABEM MOC Program. Roundtable
discussions included strengths of the program and opportunities for improvement; defining, teaching, and assessing professionalism; identifying and filling competency gaps; and enhancing relevancy and adding value to the ABEM MOC Program. Conclusions: Several suggestions to improve the ABEM MOC Program were discussed. ABEM will consider these recommendations when developing its next revision of the ABEM MOC Program. Ó 2015 Elsevier Inc.
Conflicts of Interest: Francis L. Counselman, MD, Michael L. Carius, MD, and Terry Kowalenko, MD, are Directors for the American Board of Emergency Medicine (ABEM). Earl J. Reisdorff, MD is employed by the ABEM. The opinions expressed in this paper are endorsed by the American Academy of Emergency Medicine (AAEM), AAEM/Resident and
Student Association, ABEM, American College of Emergency Physicians, and the Emergency Medicine Residents’ Association. As an accrediting body, the Resident Review Committee for Emergency Medicine and the Accreditation Council for Graduate Medical Education do not take stances on other organizations’ positions.
, Keywords—maintenance of certification; quality improvement; American Board of Emergency Medicine (ABEM)
RECEIVED: 2 March 2015; FINAL SUBMISSION RECEIVED: 9 June 2015; ACCEPTED: 23 June 2015 722
ABEM MOC Summit
723
INTRODUCTION On October 14–15, 2014, the American Board of Emergency Medicine (ABEM) convened the ABEM Maintenance of Certification (MOC) Summit, composed of representatives from the major emergency medicine stakeholder organizations (Table 1). Letters and a preliminary agenda were sent to leaders (e.g., President, Chief Executive Officer) of each organization, inviting them to participate and to send a representative to the meeting. The goals of the Summit included disseminating information about the American Board of Medical Specialties (ABMS) and the ABEM MOC Program and soliciting feedback from the emergency medicine community. Specifically, ABEM wished to gather ideas about how to improve the ABEM MOC Program.
in medical practice, some form of continuous professional development is necessary to meet a physician’s educational needs and public expectations. The ABEM MOC Program assures the public that the participating emergency physician is engaged in a rigorous continuous professional development process that adheres to a national, specialty-specific standard. The ABMS approved the MOC 2015 Standards in January 2014. The Standards require that the ABMS Member Boards, including ABEM, ‘‘engage in continual quality monitoring and improvement of its Program for MOC .’’ (1). The ABMS aspires to create MOC programs that are trusted by the public and believed in by the profession. A key to realizing this vision is to create MOC programs that are highly relevant to clinicians. History of the ABEM MOC Program
The ABMS and MOC The ABMS is a federation of 24 medical specialty certification boards, representing 37 specialties and 123 subspecialties. The ABMS believes that if the medical profession acts in benevolence to the public, society will continue to grant the medical profession substantial autonomy to determine educational standards, selfassess, and self-regulate; MOC is one manifestation of that public covenant. Structured professional learning takes place during the 4 years students spend in medical school and the 3–6 years physicians spend in residency. MOC supports the physician throughout the 30–40 years of a subsequent medical career. Given the dynamic changes that occur
Since its inception in 1979, ABEM has always required recertification with a time-limited, 10-year certificate. In 2004, this recertification requirement transitioned into Emergency Medicine Continuous Certification, now called the ABEM MOC Program. The four components of the ABEM MOC Program are outlined in Table 2. Since its beginning in 2004, the ABEM MOC Program has undergone multiple refinements, largely based on feedback from diplomates. Originally, there were 16–20 readings for each Lifelong Learning and SelfAssessment (LLSA) test; this number was reduced to 10–15 readings. The number of test questions correspondingly declined from 30–40 questions to 20–30.
Table 1. Participating Organizations and Representatives Organization American Academy of Emergency Medicine American Academy of Emergency Medicine, Resident and Student Association Association of Academic Chairs of Emergency Medicine American Board of Emergency Medicine
American Board of Medical Specialties American College of Emergency Physicians Council of Emergency Medicine Residency Directors Emergency Medicine Residents’ Association Residency Review Committee for Emergency Medicine Society for Academic Emergency Medicine
Representative Lillian Oshva, MD, Member, Education Committee, American Academy of Emergency Medicine Nicole Battaglioli, MD, Chair, Membership Committee Andy S. Jagoda, MD, President, Association of Academic Chairs of Emergency Medicine Francis L. Counselman, MD, President Michael L. Carius, MD, Secretary-Treasurer, and Chair, Maintenance of Certification Committee Terry Kowalenko, MD, Member-at-Large, and Vice-Chair, Maintenance of Certification Committee Earl J. Reisdorff, MD, Executive Director Mira Irons, MD, Vice President for Academic Affairs (presenter only) Jeffrey A. Tabas, MD, Chair-elect, Education Committee, American College of Emergency Physicians Elise Lovell, MD, Member-at-Large, Council of Emergency Medicine Residency Directors Anant Patel, DO, Speaker of the Council, Emergency Medicine Residents’ Association Philip Shayne, MD, Chair, Residency Review Committee for Emergency Medicine, Accreditation Council for Graduate Medical Education Cherri Hobgood, MD, Past President
724
F. L. Counselman et al.
Table 2. ABEM MOC Program Requirements Component Part 1: Professional Standing
Part 2: Lifelong Learning and Self-Assessment (LLSA)
Part 3: Cognitive Expertise (Continuous Certification Examination—ConCertÔ) Part 4: Assessment of Practice Performance (APP)
Description Physicians must hold a current, active, valid, full, unrestricted, and unqualified license in the United States, a US territory, or Canada. Every medical license held by the physician must meet the criterion above. LLSA activities are composed of an annual set of 10–15 readings followed by an open-book, online test over the readings. A Patient Safety LLSA is required in the first 5 years of a certification cycle. An annual average of 25 AMA PRA Category 1Ô credits must be completed. Of these, eight must be self-assessment (i.e., involving a pretest or posttest). ConCert is a secure, high-stakes, proctored examination of 205 single, best-answer items. The first component is the completion and attestation of a Practice Improvement (PI) activity, typically a quality improvement process. The second component is the participation in a communicationprofessionalism (CP) activity, typically a patient satisfaction survey.
Requirement Cycle This requirement is continuous. Failure to meet this requirement can result in decertification at any time.
Four LLSAs are required in the first 5-year cycle, one of which must be the Patient Safety LLSA. Four LLSAs are required in the second 5-year cycle. CME activity must be reported every 5 years.
The ConCert examination must be taken and passed in the second 5-year certification cycle. One PI activity and one CP activity are required in the first 5-year cycle. One PI activity and one CP activity are required in the second 5-year cycle.
ABEM = American Board of Emergency Medicine; MOC = maintenance of certification; CME = continuing medical education; AMA PRA = American Medical Association Physician’s Recognition Award. More complete descriptions are available on the ABEM website.
The score for passing the posttest was changed from 90% to 85%. The Assessment of Practice Performance (APP) Practice Improvement (PI) component originally required the physician to have 10 patients in an initial sample, and 10 patients in a second sample. This requirement would be difficult for many physicians to meet for certain conditions such as sepsis and acute myocardial infarction. Physicians can now use a lower number of patients and group or department data for high-acuity, lowfrequency conditions. DISCUSSION Attitudes about ABEM MOC There is a high rate of recertification through the ABEM MOC Program, which might reflect general acceptance of the requirements as well as other compelling external factors that lead to maintaining board certification. Of the 2455 diplomates with 2013 certification expiration dates, 1877 (76%) renewed certification in or before 2013. This is consistent with historical trends, including during the years prior to the institution of MOC in 2004. There tends to be an annual rate of about 4% of diplomates who regain certification each year. So, in 2013 there was an approximately 80% net retention of certification for physicians in the terminal year of certification. Of the total diplo-
mate pool at the end of the 2013 (31,154 diplomates), only 593 certifications (<2% of all certifications) lapsed. Summit participants viewed the ABEM MOC Program as being highly relevant. The purpose of the program is easy to understand. It attempts to address changes in the science of emergency medicine. It provides a mechanism to identify physicians who have competencies that have substantially diminished. The MOC Program emphasizes medical knowledge, cognitive skills, clinical skills, and patient safety. Certification and MOC were thought to be increasing in importance, especially because practice opportunities for noncertified physicians have been declining (2). Summit participants thought that MOC requirements are clearly communicated. The MOC Web site, which shows diplomate-specific MOC requirements, was favorably reviewed. The physician’s MOC Requirements and Status page brings clarity to the system. It was the general consensus that ABEM achieves a balance between having a sufficiently rigorous program while avoiding onerous requirements. In addition, the responsiveness of ABEM to diplomate feedback was described favorably; specifically, the manner in which the LLSA had evolved. Regarding the cost of MOC, there are fees associated with LLSA activities and the Continuous Certification (ConCertÔ) Examination. The annualized costs of
ABEM MOC Summit
MOC are $265 (excluding the cost of the optional continuing medical education [CME] activity), which are at the 50th percentile compared to all other ABMS Member Boards. The cost of MOC is <0.1% of the average diplomate’s annual compensation (3). Of note, in 2013, board-certified emergency physicians received, on average, an additional $34,800 in total compensation compared to non-board-certified physicians (3). Attitudes about Components of MOC Professionalism. The ABEM MOC Program has a ‘‘professional standing’’ requirement based on medical licensure (Table 3). The ABMS MOC 2015 Standards require that ABEM define professional expectations, and teach and assess professionalism. Participants explored the complex and enigmatic nature of defining medical professionalism. Measuring professionalism is difficult in any circumstance. Medical students and residents are in a supervised environment where faculty and program directors can assess ethical and professional behavior and provide feedback. However, once a physician is practicing independently, evaluating professionalism in the workplace is challenging. Assessing professionalism might be accomplished by requesting an attestation by the department director that the physician is ‘‘practicing medicine to acceptable standards of professionalism’’ or by a self-examination and attestation by the diplomate. A next step for ABEM is to identify those attributes that characterize professionalism and define professional behaviors. Lifelong learning and self assessment (LLSA). The primary goal of LLSA activities is to promote continuous learning by diplomates. Diplomates are required to pass a series of LLSA tests, one of which must be the new Patient Safety LLSA. They must also report completing an established number of CME credits. Each LLSA test includes an optional CME activity. A majority of physicians who complete the LLSA activity opt for CME credit, and all of these diplomates have been surveyed. Respondents report that the LLSA
725
activity is relevant to clinical practice: 70% reported that the readings were very relevant to significantly relevant; 28% thought the articles had some relevancy; and 2% thought there was little to no relevancy. Further, 92% of respondents thought the information in the articles would, to some degree, change their practices (4). In 2013, 23,576 diplomates were expected to complete an LLSA activity. There were 22,904 diplomates (97%) who passed LLSA tests. Of diplomates who passed the ConCert examination in 2013, only five did not renew their certification as a consequence of not meeting LLSA requirements. Some participants thought the bar for passing LLSA tests was too low, and was vulnerable to physicians ‘‘gaming the system.’’ Participants thought that the additional $30 fee for the LLSA CME activity was a good value. ABEM was encouraged to promote the LLSA as a CME activity by making more LLSA activities available, including subspecialty LLSAs. The value of the LLSA could also be enhanced by making the readings freely available for download from the ABEM Web site. It was thought that the ABEM MOC Program was effective at delivering new content to practitioners, but there could be an opportunity to make this more contemporary. There is a roughly 18-month lag between the time that readings are selected for inclusion for an LLSA and the time when diplomates have access to the LLSA activity. Assessment of cognitive expertise: the ConCert examination. Summit participants acknowledged that society highly values the requirement of a high-stakes, secure examination to ensure that a physician is ‘‘keeping up’’ with advances in a specialty, but thought that the ConCert Examination was the least relevant component of the ABEM MOC Program. Summit participants noted that some questions on the examination could seem obtuse and focused on minutiae. On occasion, questions seemed to have more than one correct answer, and the wording of certain items could be more straightforward. Survey data of physicians taking the 2014 ConCert Examination showed that 82% of respondents thought that all or most questions were relevant to physicians’
Table 3. Proceeding Summary and Recommendations
In general, key Emergency Medicine stakeholder organizations support the ABEM MOC Program The ABEM MOC Program is relevant to the majority of emergency physicians The ABEM MOC Web site and physician Personal Page simplify a complex set of requirements ABEM should continue to develop evidence supporting MOC Diplomates should be informed about the process that is used to score the examination. This would include a description of field testing processes that are used to ensure fairness and address test questions that seem to be esoteric or require further editing ABEM should continue to enhance the relevancy of all components of MOC ABEM should more effectively communicate the costs and value of MOC ABEM should define its standards for professionalism and increase awareness about these professionalism standards ABEM should develop MOC processes that better address competency gaps
ABEM = American Board of Emergency Medicine; MOC = maintenance of certification.
726
practices; 18% reported that most of the questions were not relevant (5). Although the overall relevancy of the items was regarded as good, there is an opportunity to enhance relevancy. ABEM notes that there are field test items on the ConCert, and that those questions might be the ones that test takers find confusing. Field test items are not scored; only ‘‘active items’’ are scored after they have demonstrated a testing performance profile that meets ABEM standards, including difficulty and discrimination. This profile is initially determined through field testing, and reassessed with every use on an examination. ConCert test takers can write comments about any items. All test-taker comments are collated and reviewed by ABEM staff and ConCert Exam editors, as well as the Chair of the Test Administration Committee, who is a clinically active emergency physician. No item is scored until all candidate comments are read and a satisfactory testing profile is obtained. ABEM could enhance the effectiveness of feedback to ConCert test takers, especially in terms of providing timely responses that are specific and actionable. Currently, ABEM provides a key word score report to physicians on items that are missed. The degree to which a physician who passes the ConCert will develop a performance improvement plan based on the examination’s score report is uncertain. Another consideration is that ABEM must balance the security of examination items against providing more detailed information about individual questions that are missed. It was acknowledged that significant learning occurs during the process of preparing for the ConCert Examination. An ABEM survey of those taking the 2014 ConCert revealed that 98% of diplomates prepare, in some fashion, for the examination (5). This preparation was thought to be more instructive than the score reports they receive from ABEM after the examination. A potential indicator that the ConCert Examination is clinically relevant is that physician performance on the examination is maintained over decades of practice (6). This would be unlikely if the content of the examination was not relevant to the clinical practice of Emergency Medicine. The pass rate over the last 5 years (2010– 2014) has ranged from 95% to 97%. Summit participants stressed that the ConCert Examination is an insufficient surrogate for assessing the composite skills needed to be a competent emergency physician. Although the examination measures judgment, diagnostic reasoning, synthesis of clinical data, and medical knowledge, it does not measure other competencies, such as communication skills and procedural skills. The secure examination experience at Pearson VUE testing centers was characterized as ‘‘easy, professional, and benign.’’ The amount of time allotted for testing was described as sufficient. There was an individual
F. L. Counselman et al.
concern that biometric palm scanning could be perceived as a personal invasion. However, a survey of 2014 ConCert test takers showed that only 1% thought the Pearson VUE processes were objectionable. There is an opportunity to better explain the cost and value of the ConCert Examination. It was thought that understanding the annualized cost of MOC might create greater acceptance of that cost. Despite the large payout every 10 years for the ConCert Examination, participants voiced no interest in annualizing fees. There was a discussion about the ConCert Examination and the degree to which it should be summative or formative. ABEM currently designs the ConCert Examination to be a summative assessment. As a high-stakes process, the physician can lose certification as a consequence of failing the examination. The notion was raised that a less comprehensive, more frequent testing format (e.g., every 2 years) might have advantages. Summit participants also explored the concept of a modular examination (a test in which physicians can select specific content areas). This might be potentially applicable for physicians working in emergency departments (EDs) that do not treat children. Arguments against a modular examination included that 1) on occasion, children are cared for in general (adult) EDs; 2) emergency physicians tend to be generalists; and 3) emergency physicians do not select the patients they see. Furthermore, certification is intended to be portable—ABEM certification is a national standard for the entire specialty. Assessment of practice performance (APP). The relevance of the APP PI requirement is determined by the physician or local ED. An inventory of 2013 APP PI attestations shows the top five quality activities to be: percutaneous coronary intervention (door-to-balloon times), door-to-doctor times, throughput time measures, aspirin for acute myocardial infarction, and sepsis care. These five activities accounted for 54% of all reported APP PI quality improvement activities (7). Approximately 5894 diplomates could have been expected to complete APP activities in 2013. There were 9348 (159% of required) attestations for PI activities, and 8137 (138%) attestations for the Communication/ Professionalism requirement. Although ABEM has aligned APP requirements with physician clinical and quality activities, Summit participants challenged APP PI activities as being ‘‘checkbox’’ requirements. Quality improvement activities and monitoring are ubiquitous in EDs. ABEM’s choice to align the APP PI requirements with existing activities that emergency physicians are performing might make ABEM’s approach to this requirement feel perfunctory.
ABEM MOC Summit
There were divergent opinions about whether APP activities—specifically, the PI component—should have a greater focus on individual competency assessment, or on a systems-based, department-wide assessment. Currently, the ABEM MOC Program allows both individual and department-based activities. Further, ABEM joined the ABMS Multi-specialty Portfolio Project, which provides APP PI credit for institutional and multidisciplinary quality improvement activities (8). Additional Suggestions Using MOC to identify competency gaps. Although gap identification already occurs to some degree within the ABEM MOC Program, it is on a fairly broad basis, not an individual physician basis. Further, ABEM’s examination processes can only measure selected competencies, not every competency required to be an emergency physician. Approaching gap identification in a broad fashion is supported by the assumption that emergency physicians are, for the most part, generalists throughout their careers. As major and important new advances are introduced into clinical practice, ABEM MOC might have a unique opportunity to introduce them to the practicing physician. Examples include bedside ultrasonography or video laryngoscopy. It was suggested that The Joint Commission’s Ongoing Professional Practice Evaluation (OPPE) and Focused Professional Practice Evaluation (FPPE) are designed to find gaps in physician performance. One idea worthy of further exploration was integrating elements of OPPE/FPPE into the ABEM MOC Program. Further, ABEM could be more intentional about first identifying a gap and then finding an LLSA reading to fill that gap. Participants discussed the possibility of providing feedback that would assist individual physicians identify knowledge gaps. This presents several challenges, one of which is that the ConCert Examination would need to contain many more items. The current examination ‘‘blueprint’’ requires a limited number of questions in several content areas, and the examination design uses a ‘‘survey examination’’ approach, in which selected items serve as a proxy for the ‘‘universe of knowledge’’ within the specialty. In its current form, the number of questions in some areas on the ConCert Examination are insufficient to accurately determine individual gaps. Physicians have limited proficiency with selfassessment (9). In fact, for physicians with the greatest competency gaps, self-awareness might have limited impact. Despite the inability of physicians to effectively self-assess, ABEM might still wish to develop an effective self-assessment inventory. Adding value and enhancing relevancy. ABEM monitors the relevancy of its MOC Program, and Summit partic-
727
ipants encouraged ABEM to continue doing so. The Longitudinal Study of Emergency Physicians could be better used to monitor physician perceptions of the ABEM MOC Program. Relevancy could also be further enhanced by integrating emerging issues in a more timely way. Balancing a more immediate presentation of information with maintaining standards and avoiding practices and trends that might quickly fade will be challenging. It was cautioned that the ‘‘brand’’ of ABEM will be diminished if ABEM presents poorly vetted information. Resident awareness. There is an opportunity to bring information about the career-long commitment of MOC to physicians in training. The degree to which residents need to know the details of the ABEM MOC Program is limited during their period of training, but the program’s structure can be shared. CONCLUSIONS Summit participants supported the convening of the MOC Summit, with the resultant community-based conversation through which the ABEM MOC Program could be improved. Also, having the ABMS participate was thought to be valuable. The representatives from the Emergency Medicine Residents’ Association and the Resident and Student Association of the American Academy of Emergency Medicine thought there was an opportunity to more effectively share information with residents. Participation by the two resident representatives was especially appreciated. ABEM encouraged all participants to share the information from the Summit with their sponsoring organizations. Although there were several suggestions for improvement (Table 3), there was substantial support for the ABEM MOC Program from Summit participants. General suggestions for improvement included developing greater evidence that MOC improves physician performance, patient outcomes, and patient safety. Finally, it was recommended that ABEM share even more information about MOC costs, relevance, and the operational details of the program. Acknowledgment—The authors wish to thank Ms. Frances M. Spring for her assistance in the preparation of this manuscript.
REFERENCES 1. American Board of Medical Specialties (ABMS). The ABMS MOC 2015 Standards. Available at: http://www.abms.org/media/1109/ standards-for-the-abms-program-for-moc-final.pdf. Accessed July 10, 2015.
728 2. Katz B. Top 10 best, worst places for emergency physicians to seek jobs in 2014–2015. ACEP Now, September 12, 2014. Available at: http://www.acepnow.com/article/top-10-best-worst-places-emergencyphysicians-seek-jobs-2014-2015/. Accessed July 5, 2015. 3. American College of Emergency Physicians (ACEP). 2013 National emergency medicine salary survey: clinical results. Emergency medicine compensation & benefit surveys. Dallas: American College of Emergency Physicians and Daniel Stern and Associates; 2013. 4. Jones JH, Smith-Coggins R, Meredith JM, Korte RC, Reisdorff EJ, Russ CM. Lifelong learning and self-assessment is relevant to emergency physicians. J Emerg Med 2013;45:935–41. 5. Marco CA, Wahl RP, Counselman FL, Heller BN, Harvey AL, Joldersma KB, Kowalenko T, Reisdorff EJ. The American Board of Emergency Medicine ConCert examination: Emergency physicians’ perceptions of clinical relevance and benefits. Acad Emerg Med 2015. Accepted for publication.
F. L. Counselman et al. 6. Marco CA, Counselman FL, Korte RC, Russ CM, Whitley C, Reisdorff EJ. Emergency physicians maintain performance on the American Board of Emergency Medicine continuous certification (ConCert) examination. Acad Emerg Med 2014;21: 532–7. 7. Kowalenko T, Carius ML, Korte RC, Miller MC, Reisdorff EJ. Emergency department quality improvement activity: an inventory from the American Board of Emergency Medicine maintenance of certification program. Acad Emerg Med 2015;22: 367–72. 8. American Board of Medical Specialties (ABMS). Multi-specialty MOC portfolio approval program. Available at: http://mocport folioprogram.org. Accessed November 15, 2014. 9. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–102.