Predictors of performance on the Maintenance of Certification in Anesthesiology Program® (MOCA®) examination

Journal of Clinical Anesthesia (2014) xx, xxx–xxx

Original contribution

Predictors of performance on the Maintenance of Certification in Anesthesiology Program® (MOCA®) examination☆ Huaping Sun Ph.D. (Manager of Psychometrics and Research)a,⁎, Deborah J. Culley M.D. (Associate Professor of Anesthesiology, Director)a,b , Cynthia A. Lien M.D. (Professor of Anesthesiology, Director)a,c , Diana L. Kitchener B.S. (Manager of Maintenance of Certification Programs)a , Ann E. Harman Ph.D. (Chief Assessment Officer)a , David O. Warner M.D. (Professor of Anesthesiology, Director)a,d a

The American Board of Anesthesiology, Raleigh, NC, 27609 Department of Anesthesiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, 02115 c Department of Anesthesiology, Weill Cornell Medical College, New York, NY, 10065 d Department of Anesthesiology, Mayo Clinic, Rochester, MN, 55905 b

Received 8 April 2014; revised 7 August 2014; accepted 8 August 2014

Keywords: The American Board of Anesthesiology; The Maintenance of Certification in Anesthesiology Program (MOCA); MOCA examination; Regression analysis

Abstract Study objective: The aim of this study was to determine the independent factors associated with performance on the Maintenance of Certification in Anesthesiology Program (MOCA) examination. Design: Cross-sectional study. Setting: The American Board of Anesthesiology, Raleigh, NC. Subjects: The American Board of Anesthesiology (ABA) diplomates who were certified between 2000 and 2006 and had taken the MOCA examination at least once by July 2013. Measurements: MOCA examination score for the first attempt. Main results: Independent positive predictors for MOCA examination score in multiple regression analysis included passing the ABA Part 1 and Part 2 certification examinations on the first attempt and male sex, whereas negative predictors included history of action(s) taken against any medical license, taking the examination later in the MOCA cycle and older age at primary certification. Conclusions: Several factors in addition to performance on the written examination for primary certification (Part 1 Examination) are independently associated with performance on the MOCA examination. Because many of these factors are not modifiable, those diplomates who possess unfavorable risk factors should pay special attention to engaging in continuing learning to prepare for the MOCA examination. © 2014 Elsevier Inc. All rights reserved.

☆

Supported by the American Board of Anesthesiology, Raleigh, NC, USA. ⁎ Corresponding author at: The American Board of Anesthesiology, 4208 Six Forks Road, Suite 1500, Raleigh, NC, 27609. Tel.: + 1 919 745 2268; fax: + 1 919 745 2201. E-mail address: [email protected] (H. Sun). http://dx.doi.org/10.1016/j.jclinane.2014.08.007 0952-8180/© 2014 Elsevier Inc. All rights reserved.

1. Introduction Since 2000, all certificates issued by the American Board of Anesthesiology, Inc.® (ABA, Raleigh, NC) expire 10 years after the year that primary certification was awarded unless the

2 diplomate satisfactorily completes the requirements of the Maintenance of Certification in Anesthesiology Program (MOCA). The goal of MOCA is to help anesthesiologists continue to meet the core competencies of patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice throughout their career. MOCA requirements are designed to provide opportunity for professional growth and improvement as well as assessments in these competencies through participation in the following four domains: part 1, Professional Standing (maintaining medical licensure); part 2, Lifelong Learning and Self-Assessment (participation in continuing medical education); part 3, Cognitive Expertise (a secure written examination); and part 4, Practice Performance Assessment and Improvement (including a formative simulation experience and/or a case evaluation). The MOCA examination is currently utilized to meet the part 3 Cognitive Expertise requirement of MOCA. This examination has been administered since 2005 in secure testing centers (Pearson VUE) and currently consists of 200 multiple choice questions. The examination is offered biannually and can be taken as early as the seventh year of the 10-year MOCA cycle, thus allowing every diplomate as many as 8 attempts to pass the examination before his or her certificate expires. In its current form, the examination assesses diplomates' medical knowledge and clinical judgment in the field of anesthesiology, and, although similar to the written examination required for primary certification (designated as the Part 1 Examination), the topics emphasized are those specifically related to patient care. A satisfactory performance on this examination is necessary to satisfy the MOCA part 3 requirement. To the extent that cognitive expertise is a stable construct over time, and considering evidence that past written medical examination scores strongly predict future written medical examination performance [1–4], performance on the Part 1 Examination would be expected to correlate with performance on the MOCA examination. Other factors that affect examinee performance on certification examinations include gender [2,5,6], age [7,8], medical school country [5,9], and interval of time since residency [7,8]. The aim of this study was to determine the independent factors associated with performance on the MOCA examination. We hypothesized that performance on the MOCA examination would be primarily associated with performance on the Part 1 Examination, but that several other candidate factors, including gender, age, medical school country, and the time elapsed from primary certification to taking the MOCA examination, would also affect MOCA examination performance.

2. Materials and methods The Mayo Clinic Institutional Review Board determined that this project was exempt from review.

H. Sun et al.

2.1. Data source We obtained data from ABA records on the diplomates who achieved primary certification between 2000 and 2006 and had taken the MOCA examination at least once as of July 2013. We selected this cohort because ABA diplomates enrolled in MOCA may take this examination between year 7 and year 10 of their MOCA cycle; time-limited certificates were first introduced in 2000, and those who achieved primary certification after 2006 were not yet eligible to take the MOCA examination at the time that this study was conducted. The score on the first attempt MOCA examination was the outcome of interest and was recorded for each individual. Examination results are reported as a standardized scale score with a mean of 250 and a standard deviation of 50 for the calibration group (ie, the American medical school graduates who took the MOCA examination the first time under standard condition). Likewise, data on first attempt Part 1 Examination outcome (pass/fail) and first attempt Part 2 Examination (oral examination for the primary certification) outcome (pass/fail) were recorded. Part 1 Examination and Part 2 Examination actual standardized scores were recorded in the ABA records beginning in 2002. Finally, we collected data on each individual's age at the time of primary certification, sex, medical school country (United States vs international), years between completion of an Accreditation Council for Graduate Medical Education–accredited residency program and board certification, year in the diplomate's MOCA cycle that the examination was first attempted, history of substance use disorder, and history of a Disciplinary Action Notification Service (DANS) alert. The DANS alert is a service of the Federation of State Medical Boards that provides information on any actions regarding the diplomate's medical license to all American Board of Medical Specialties® (ABMS®) member boards [10].

3. Statistical analysis Descriptive statistics were calculated for the outcome variable and the predictor variables. Bivariate correlations between first time Part 1 score and first time MOCA score, and between first time Part 2 score and first time MOCA score were examined for those who took their first time Part 1 Examination in or after 2002. Simple regressions were conducted to identify the potential significant predictors for first attempt MOCA examination score. Those variables that were significant in the simple regressions were subjected to a multiple regression analysis, with the forward stepwise method, to determine factors that were predictive of first attempt MOCA examination score. The forward stepwise method starts with no variable in the model, adds each variable that has the most predictive power of the outcome variable and improves the model the most (ie, the greatest R2 change comparing the model with and without the current variable). This stepwise process for including variables

3 continues until none of the remaining variables will improve the model R2. In a secondary analysis, we calculated the MOCA examination pass rates for all diplomates who obtained primary certification between 2000 and 2006 and had taken the MOCA examination as of July 2013, and compared pass/fail outcomes based on the number of attempts on the MOCA examination. All the statistical analyses were performed using SPSS Version 20.0 (IBM Corporation, Armonk, NY). A P value of less than .05 was considered statistically significant.

4. Results Seven thousand ninety-two physicians received primary certification in anesthesiology between 2000 and 2006. Among these diplomates, 4666 had attempted the MOCA examination at least once as of July 2013. Five diplomates were excluded from the analysis because of missing data, leaving 4661 in the regression analysis. Most of the diplomates were male, US medical school graduates, certified at or before age 36 years, certified within the first year after completion of residency training, and took the MOCA examination for the first time at year 7 or year 8 of their MOCA cycle (Table 1). In contrast, a small percentage had history of a DANS alert indicative of action(s) against their medical license or a reported substance use disorder (Table 1). The pass rates for these 4661 physicians certified between 2000 and 2006 on first attempts on the Part 1 and Part 2 examinations were 78% and 77%, respectively (Table 1). However, 95% of them passed the MOCA examination on the first attempt (Table 1). Among those who did not pass the MOCA examination the first time, the likelihood of success decreased with each subsequent attempt (Table 2). In bivariate correlations (ie, Pearson correlations) with scores expressed as continuous variables, first time MOCA score was correlated with both first time Part 1 score (r = .61, P b .001, n = 1799) and first time Part 2 score (r = .26, P b .001, n = 1799). Simple regressions showed that first attempt Part 1 examination outcome (pass/fail), first attempt Part 2 examination outcome (pass/fail), sex, age at primary certification (years older than 28), the year in the diplomate's MOCA cycle that the examination was first attempted (each year waited after year 7 in the 10-year MOCA cycle), history of a DANS alert, and history of substance use disorder were significant predictors of the MOCA examination score the first time that it was taken (Table 3). The significant predictors in the simple regression were then subjected to a multiple regression analysis. Positive predictors of MOCA examination score in the multiple regression analysis were, in the order of diminishing predictive power, passing Part 1 Examination on the first attempt, passing Part 2 Examination on the first attempt, and being male (Table 3). Holding other predictors constant, those who passed Part 1 Examination on the first attempt scored on average 35.6 points higher on their first MOCA

Table 1 Characteristics of the ABA diplomates from 2000 to 2006 who have attempted MOCA examination Characteristics Sex Male Female Medical school country American International Age at primary certification 35 or younger 36-40 41-45 46-50 51 or older Years between residency graduation and primary certification 1 2 3 4 5 6+ First MOCA attempt year within the MOCA cycle 7th year 8th year 9th year 10th year 11th-13rd year Ever received a DANS alert Yes No Presence of substance use disorder Yes No First attempt Part 1 result Pass Fail First attempt Part 2 result Pass Fail First attempt MOCA result Pass Fail

No.

%

3449 74 1212 26 2772 59 1889 41 2331 50 1335 29 629 13 270 6 96 2

2523 54 1054 23 482 10 234 5 121 3 247 5 1394 1180 1228 788 71

30 25 26 17 2

118 3 4543 97 46 1 4615 99 3641 78 1020 22 3606 77 1055 23 4428 95 233 5

examination attempt than those who failed Part 1 on the first try. Similarly, those who passed Part 2 Examination on the first attempt scored 11.4 points higher on their first MOCA examination attempt than those who failed Part 2 on the first try. Male diplomates scored 9.0 points higher than female diplomates on their first MOCA examination attempt. Negative predictors of MOCA examination score in the multiple regression included older age at primary certification, delay in taking the MOCA examination, and history of a DANS alert (Table 3). Score on the first MOCA examination attempt was 0.7 points lower for each year over age 28 years at the time of primary certification. Similarly, diplomates scored 4.6 points lower for each year they waited beyond

4

H. Sun et al. Table 2

Pass rates on the MOCA examination based on the number of attempts MOCA exam result

No. of attempts

1 2 3 ≥4

Total

Count % within Count % within Count % within Count % within Count

1 attempt 2 attempts 3 attempts ≥ 4 attempts

their first year of eligibility to take the MOCA examination (typically year 7). Finally, MOCA scores were 14.5 points lower on the first attempt in those with a DANS alert vs those without. The number of years between completion of an Accreditation Council for Graduate Medical Education–accredited residency program and primary certification, and a history of substance use disorder were significant in the simple regression but not in the multiple regression. First-attempt MOCA examination scores did not differ between US and international medical graduates. Accordingly, the predictive model for first-attempt MOCA examination score is 210.65 + 35.60 × first attempt Part 1 pass + 11.37 × first attempt Part 2 pass + 9.02 × male − 14.45 × history of a DANS alert − 4.64 × year(s) after year 7 of the MOCA cycle − 0.67 × year(s) older than 28 at primary certification. On the basis of this model, for example, the first time MOCA score for a female who failed the Part 1 and Part 2 on her first attempt, had history of a DANS alert, attempted MOCA examination at year 10 in her MOCA cycle (3 years later than year 7), and got her primary certification at 38

Table 3

Pass

Fail

4428 95.0% 107 66.9% 20 54.1% 5 26.3% 4560

233 5.0% 53 33.1% 17 45.9% 14 73.7% 317

4661 100.0% 160 100.0% 37 100.0% 19 100.0% 4877

years old (10 years older than age 28), is expected to be 176. The six predictors in the model accounted for 15.9% of the variance of first attempt MOCA examination score.

5. Discussion Although passing Part 1 Examination on the first attempt was predictive of MOCA examination performance, several other factors were independently associated with MOCA examination performance. Primary certification examination outcomes were the strongest predictors of MOCA examination score; passing Part 1 Examination on the first attempt was associated with a 35.6 point higher MOCA examination score. These data are consistent with previous studies demonstrating that good performance on one medical content examination is a strong predictor of performance on subsequent medical content examinations. For example, the ABA has previously reported that prior performance on the ABA/American Society of Anesthesiologists In-Training Examination is a good

The results of simple regressions and multiple regression with first time MOCA examination score as the outcome variable

Predictor variables

Simple regression

Multiple regression

Unstandardized coefficients P (95% CI)

Unstandardized coefficients P (95% CI)

First time Part 1 outcome 42.19 First time Part 2 outcome 20.49 Sex 8.94 US vs international medical school a − 2.24 Years between residency graduation and primary certification a − 4.50 Age at primary certification (year older than 28) b − 1.80 Each year waited for first attempt of MOCA exam in the − 6.09 MOCA cycle c History of a DANS alert − 26.55 History of substance use disorder a − 17.26 a

Total

(38.89 to 45.49) (17.06 to 23.92) (5.63 to 12.25) (− 5.20 to 0.73) (− 5.09 to − 3.90) (− 2.07 to − 1.53) (− 7.35 to − 4.83)

b .001 b .001 b .001 .131 b .001 b .001 b .001

35.60 (32.25 to 38.95) 11.37 (8.05 to 14.69) 9.02 (6.02 to 12.01)

b .001 b .001 b .001

− 0.67 (− 0.94 to − 0.41) − 4.64 (− 5.79 to − 3.48)

b .001 b .001

(− 35.79 to − 17.31) b .001 − 14.45 (− 22.84 to − 6.05) (− 31.99 to − 2.54) .019

.001

This variable is not significant in the multiple regression and thus not included in the final regression model. Because the youngest age when becoming ABA certified in this cohort is 28, age 28 is used as the reference age for the variable of age at primary certification. c Because the earliest time that the diplomates are allowed to take the MOCA examination in the 10-year MOCA cycle is year 7, year 7 is used as the reference year for the variable of the year in the diplomates' MOCA cycle that the examination was first attempted. b

5 predictor of success on the ABA Part 1 Examination [2]. Data from the American Board of Internal Medicine (ABIM) and the American Board of Surgery have also demonstrated a positive association between scores on the primary certification and maintenance of certification (MOC) examinations [5]. This correlation is not limited to ABMS member board examinations because performance on the Medical College Admission Test scores strongly correlate with United States Medical Licensing Examination scores [11], and United States Medical Licensing Examination scores predict performance on ABMS member board in-training and certifying examinations [1,12,13]. In essence, this study confirms that good written test takers remain good written test takers over time. Of interest, a first time pass on Part 2 Examination was independently associated with MOCA examination performance. The oral examination is designed to evaluate domains in addition to medical knowledge, including clinical judgment, adaptability, and other domains relevant to clinical practice. The MOCA examination is constructed to focus on areas relevant to clinical practice, and we speculate that the additional predictive power provided by Part 2 performance may reflect this emphasis. The association between higher MOCA examination score and male sex supported previous findings in other ABMS member boards (e.g., ABIM, American Board of Surgery, American Board of Family Medicine [ABFM]) that male sex was a positive predictor of higher MOC examination score [5,9]. The findings regarding how sex affects performance on certification examinations are mixed. Men performed slightly better than women on the ABIM-certifying examination [6], and men outperformed women in the MOC examinations among internists and surgeons [5]. There was no difference between men and women's performance on the ABA's Part 1 Examination, but female anesthesiologists were slightly more likely than male anesthesiologists to pass the ABA's Part 2 Examination [2]. The cause of why male anesthesiologists outperformed female anesthesiologists in the MOCA examination is unclear, but we speculate that a higher rate of part-time employment among female anesthesiologists and/or female's greater responsibility for a growing family in the early stage of her career may contribute. The timing of the MOCA examination within the MOCA cycle is a variable controlled by the diplomate and may have a bearing on a successful outcome. Why those who took the examination earlier in the MOCA cycle performed better is unknown; it may speak more to the characteristics of the individuals, such as self-selecting for those who have better adapted to the concept of lifelong learning, than any intrinsic benefit of early testing. It is worth noting that there is no significant correlation between first time Part 1 score and timing of the diplomate to take the MOCA examination the first time. Given that those taking the examination earlier in the 10-year MOCA cycle have more opportunities to take the examination if they fail without a lapse in certification, diplomates may consider taking their examination at their earliest opportunity. Interestingly, although a shorter interval

after graduation from residency was positively correlated with ABIM recertification examination score [7,8], seasoned family physicians without certification gaps outperformed recent graduates on the ABFM certification examination [9]. Some of the variables have no or little association with performance on the MOCA examination. For example, in contrast to the ABA's previously published predictors of performance on the Part 1 Examination [2] and data on the ABIM MOC examination [14], we identified no difference between the performances of graduates of US vs international medical schools. Although advancing age was a statistically significant negative predictor of MOCA performance, the effect was minor. The ABIM also found lower scores on the internal medicine MOC examination among older internists [5,14]. Nevertheless the ABFM found that those family physicians without a lapse in their certification increased their scores each 7-year cycle they took the recertification examination until they were in their mid-50s [9]. Whether advancing age reflects poor baseline test scores resulting in older age at entry to medical school and completion of residency or poorer performance as one ages cannot be determined in this study. In contrast, history of a DANS alert, indicating medical license actions, was associated with decreased MOCA examination score. The ABIM has previously reported an association between lower primary certification examination score and professionalism rating on the ABIM annual Resident's Evaluation Summary with a higher risk of disciplinary action by state medical boards [15]. The stress associated with a DANS alert may impair performance, or alternatively that poor examination performance may identify an unrecognized risk factor for professionalism issues. A history of substance use disorder is associated with lower ABA Part 1 examination score for primary certification [2], but was not found to be independently associated with MOCA examination performance in this study with other predictors in the regression model, although this may be explained by the relatively low number of diplomates with this condition. In conclusion, although performance on the ABA Part 1 Examination for primary certification was associated with performance on the MOCA examination taken at least 7 years later, several other factors, including performance on the ABA Part 2 Examination for primary certification, sex, age, a history of action(s) against medical license(s), and timing of MOCA examination attempt, were independently associated with MOCA examination performance. Because many of these factors are not modifiable, those diplomates who possess unfavorable risk factors should pay special attention to engaging in continuing learning to prepare for the MOCA examination.

References [1] Myles TD, Henderson RC. Medical licensure examination scores: relationship to obstetrics and gynecology examination scores. Obstet Gynecol 2002;100(5 Pt. 1):955-8 [Epub 2002/11/09. PubMed PMID: 12423860].

6 [2] McClintock JC, Gravlee GP. Predicting success on the certification examinations of the American Board of Anesthesiology. Anesthesiology 2010;112(1):212-9. http://dx.doi.org/10.1097/ALN.0b013e3181c62e2f [Epub 2009/12/17. PubMed PMID: 20010419]. [3] Klein GR, Austin MS, Randolph S, Sharkey PF, Hilibrand AS. Passing the boards: can USMLE and Orthopaedic In-Training Examination Scores predict passage of the ABOS Part-I examination? J Bone Joint Surg 2004;86(5):1092-5. [4] Rollins LK, Martindale JR, Edmond M, Manser T, Scheld WM. Predicting pass rates on the American Board of Internal Medicine certifying examination. J Gen Intern Med 1998;13(6):414-6 [Epub 1998/ 07/21. PubMed PMID: 9669571; PubMed Central PMCID: PMC1496976]. [5] Lipner R, Song H, Biester T, Rhodes R. Factors that influence general internists' and surgeons' performance on maintenance of certification exams. Acad Med 2011;86(1):53-8. [6] Day SC, Norcini JJ, Shea JA, Benson JA Jr. Gender differences in the clinical competence of residents in internal medicine. J Gen Intern Med 1989;4(4):309-12. [7] Norcini JJ, Lipner RS, Benson JA Jr, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Ann Intern Med 1985;102(3):385-9 [Epub 1985/03/01. PubMed PMID: 3970474]. [8] Meskauskas JA, Webster GD. The American Board of Internal Medicine recertification examination: process and results. Ann Intern Med 1975; 82(4):577-81 [Epub 1975/04/01. PubMed PMID: 1119774]. [9] O'Neill TR, Puffer JC. Maintenance of certification and its association with the clinical knowledge of family physicians. Acad Med 2013;88(6): 780-7.

H. Sun et al. [10] Warner DO, Berge K, Sun H, Harman A, Hanson A, Schroeder DR. Substance use disorder among anesthesiology residents, 1975-2009. JAMA 2013;310(21):2289-96. http://dx.doi.org/10.1001/jama.2013. 281954. [11] Zhao X, Oppler S, Dunleavy D, Kroopnick M. Validity of four approaches of using repeaters' MCAT scores in medical school admissions to predict USMLE Step 1 total scores. Acad Med 2010;85(10 Suppl.):S64-7. http:// dx.doi.org/10.1097/ACM.0b013e3181ed38fc [Epub 2010/10/15. PubMed PMID: 20881707]. [12] de Virgilio C, Yaghoubian A, Kaji A, Collins JC, Deveney K, Dolich M, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg 2010;145(9):852-6. [13] Thundiyil JG, Modica RF, Silvestri S, Papa L. Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents? J Emerg Med 2010;38(1):65-9. http://dx.doi.org/10.1016/j.jemermed.2008.04.010 [Epub 2008/10/28. PubMed PMID: 18950974]. [14] Reed DA, West CP, Holmboe ES, Halvorsen AJ, Lipner RS, Jacobs C, et al. Relationship of electronic medical knowledge resource use and practice characteristics with Internal Medicine Maintenance of Certification Examination scores. J Gen Intern Med 2012;27(8):917-23. http:// dx.doi.org/10.1007/s11606-012-2017-5 [PubMed PMID: 22374410; PubMed Central PMCID: PMC3403143]. [15] Papadakis MA, Arnold GK, Blank LL, Holmboe ES, Lipner RS. Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards. Ann Intern Med 2008;148(11):869-76. http://dx.doi.org/10.7326/0003-4819-148-11200806030-00009.