- Email: [email protected]
Predictors of performance on the National Board of Medical Examiners obstetrics and gynecology subject examination
Graduate Education Predictors of performance on the National Board of Medical Examiners obstetrics and gynecology subject examination Alicia Armstrong, MD, Chimene Dahl, MD, and William Haffner, MD Objective: To examine the correlation among the National Board of Medical Examiners subject examination, the US Medical Licensure Examination step 1 scores, and grade point average and to determine if poor performance could be predicted by the US Medical Licensure Examination step 1 score and grade point average. Methods: The subject examination scores of 148 third-year medical students were compared with their US Medical Licensure Examination step 1 scores and grade point averages. Scores below the 20th percentile were defined as poor performance. Results: There was a significant correlation between the US Medical Licensure Examination score and the subject examination score (r 5 .6, P < .001). The correlation with grade point average was also significant (r 5 .57, P < .001). Conclusion: The US Medical Licensure Examination step 1 performance, grade point average, and performance on the subject examination are all correlated. However, use of the US Medical Licensure Examination step 1 scores and grade point averages to identify all students at risk was associated with a high false-positive rate at our institution. (Obstet Gynecol 1998;91:1021–2.)
During the obstetrics-gynecology clerkship, grades are based on performance in three areas: 1) noncognitive skills, 2) cognitive skills, and 3) results of the National Board of Medical Examiners subject examination. The first objective of this study was to examine the correlaFrom the Uniformed Services University of the Health Sciences, Department of Obstetrics and Gynecology, Bethesda, Maryland. The opinions or assertions contained herein are the private views of the authors and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense.
VOL. 91, NO. 6, JUNE 1998
tion between the National Board of Medical Examiners subject examination and the US Medical Licensure Examination step 1 score. Our second objective was to determine if marginal performance could be predicted by the step 1 score and grade point average.
Materials and Methods The obstetrics-gynecology subject examination scores and US Medical Licensure Examination step 1 examination scores of 148 third-year medical students from the class of 1996 were obtained from the Registrar’s Office at the Uniformed Services University of the Health Sciences. The associations between scores were examined by Pearson’s correlation coefficient. Means (6 standard deviation [SD]) were compared using the two sample t test. In an effort to identify students who had deficiencies in their fund of knowledge, the faculty identified scores below the 20th percentile as a reasonable indication of poor performance. A receiver operating characteristic curve was used to describe the association of US Medical Licensure Examination scores with performance below the 20th percentile on the National Board of Medical Examiners subject examination. Multivariate analysis included multiple linear regression. Data were analyzed using SPSS 7.5 for Windows (SPSS Inc. Chicago, IL).
Results The mean (6SD) US Medical Licensure Examination step 1 score of the students who scored below the 20th percentile was significantly lower than that of the students scoring above the 20th percentile (189.4 6 7.8 versus 204.3 6 16.6; P 5 .003). The subject exam score correlated significantly with the US Medical Licensure Examination score (r 5 .6, P , .001) and the grade point average (r 5 .57, P , .001). Increases in specificity were associated with a decrease in sensitivity, as illustrated by the receiver operating characteristic curve (Figure 1). Using multiple linear regression, grade point average (coefficient 5 50.2, standard error 5 18.6, P 5 .008) and US Medical Licensure Examination score (coefficient 5 2.1, standard error 5 0.6, P , .001) were significant predictors of National Board of Medical Examiners examination scores (R2 5 .38, P , .001) (Figure 2).
Discussion To ensure objective evaluation of the student’s fund of knowledge in the clinical subject area, many institutions use an external standardized test to assess educational achievement. Paul Kelley of the National Board of Medical Examiners estimates that approximately 49,000 of the 72,000 subject examinations used by medical
0029-7844/98/$19.00 PII S0029-7844(98)00090-8
1021
Figure 1. Receiver operating characteristic curve. Specificity of individual step 1 exam scores as a predictor of marginal performance on the National Board of Medical Examiners subject examination is shown on the x axis. Sensitivity of individual step 1 exam scores as a predictor of marginal performance on the National Board subject examination is shown on the y axis.
schools were in the clinical subject areas.1 An advantage of using the National Board of Medical Examiners examination score is that these examinations provide highly reliable measures.2 The subject examination scores also have been valid predictors of performance on the National Board of Medical Examiners part II examination. The correlation coefficient for the subject exam in obstetrics-gynecology and the obstetricsgynecology section of National Board examination was 0.562, which is statistically significant (P , .01).3 In this investigation we found that the US Medical Licensure Examination score and grade point average were significantly associated with the National Board of
Medical Examiners subject examination score. The mean US Medical Licensure Examination score of the students scoring below the 20th percentile was also significantly lower than that of the students scoring at or above the 20th percentile. To identify the students who received a National Board of Medical Examiners subject examination score below the 20th percentile, however, a large number of students with scores above the 20th percentile were identified. We have examined ways in which to use the information gathered in this investigation. The practice at our university was to offer intensive tutorial assistance for students after failure of the National Board of Medical Examiners subject examination. After reviewing the data obtained in this investigation, we elected to organize a self-study tutorial program. Using this approach, we will not commit resources to a laborintensive process that in many cases may not be necessary. Prior studies in the literature have indicated that students’ self-assessments tend to be relatively accurate.4 Therefore, we have students self-identify themselves by requesting the study materials. In conclusion, we found a correlation between performance on the step 1 exam, grade point averages, and the National Board of Medical Examiners subject examination. There are, however, some limitations to the use of the US Medical Licensure Examination and grade point average as the sole predictors of performance on the end-of-clerkship subject examination score.
References 1. Hoffman KI. The USMLE: The NBME subject examinations, and assessment of individual academic achievement. Acad Med 1993; 68:740 –7. 2. Kappelman MM. The impact of external examinations on medical education program and students. J Med Educ 1983;58:300 – 8. 3. Spellacy WN, Dockery JL. A comparison of medical student performance on the obstetrics and gynecology National Board Part II examination and a comparable examination given during the clerkship. J Reprod Med 1980;24:76 – 8. 4. Herbert WNP, McGaghie WC, Drogmueller W, Riddle MH, Maxwell KL. Student evaluation in obstetrics and gynecology selfversus departmental assessment. Obstet Gynecol 1990;76:458 – 61.
Address reprint requests to:
Alicia Y. Armstrong, MD Department of Obstetrics and Gynecology Walter Reed Army Medical Center Washington, DC 20307
Figure 2. Correlation among individual US Medical Licensure Examination step 1 scores, grade point averages, and the National Board of Medical Examiners subject examination.
1022 Armstrong et al
Predictors of Test Performance
Received October 23, 1997. Received in revised form February 2, 1998. Accepted February 19, 1998. Published by Elsevier Science Inc.
Obstetrics & Gynecology
During the obstetrics-gynecology clerkship, grades are based on performance in three areas: 1) noncognitive skills, 2) cognitive skills, and 3) results of the National Board of Medical Examiners subject examination. The first objective of this study was to examine the correlaFrom the Uniformed Services University of the Health Sciences, Department of Obstetrics and Gynecology, Bethesda, Maryland. The opinions or assertions contained herein are the private views of the authors and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense.
VOL. 91, NO. 6, JUNE 1998
tion between the National Board of Medical Examiners subject examination and the US Medical Licensure Examination step 1 score. Our second objective was to determine if marginal performance could be predicted by the step 1 score and grade point average.
Materials and Methods The obstetrics-gynecology subject examination scores and US Medical Licensure Examination step 1 examination scores of 148 third-year medical students from the class of 1996 were obtained from the Registrar’s Office at the Uniformed Services University of the Health Sciences. The associations between scores were examined by Pearson’s correlation coefficient. Means (6 standard deviation [SD]) were compared using the two sample t test. In an effort to identify students who had deficiencies in their fund of knowledge, the faculty identified scores below the 20th percentile as a reasonable indication of poor performance. A receiver operating characteristic curve was used to describe the association of US Medical Licensure Examination scores with performance below the 20th percentile on the National Board of Medical Examiners subject examination. Multivariate analysis included multiple linear regression. Data were analyzed using SPSS 7.5 for Windows (SPSS Inc. Chicago, IL).
Results The mean (6SD) US Medical Licensure Examination step 1 score of the students who scored below the 20th percentile was significantly lower than that of the students scoring above the 20th percentile (189.4 6 7.8 versus 204.3 6 16.6; P 5 .003). The subject exam score correlated significantly with the US Medical Licensure Examination score (r 5 .6, P , .001) and the grade point average (r 5 .57, P , .001). Increases in specificity were associated with a decrease in sensitivity, as illustrated by the receiver operating characteristic curve (Figure 1). Using multiple linear regression, grade point average (coefficient 5 50.2, standard error 5 18.6, P 5 .008) and US Medical Licensure Examination score (coefficient 5 2.1, standard error 5 0.6, P , .001) were significant predictors of National Board of Medical Examiners examination scores (R2 5 .38, P , .001) (Figure 2).
Discussion To ensure objective evaluation of the student’s fund of knowledge in the clinical subject area, many institutions use an external standardized test to assess educational achievement. Paul Kelley of the National Board of Medical Examiners estimates that approximately 49,000 of the 72,000 subject examinations used by medical
0029-7844/98/$19.00 PII S0029-7844(98)00090-8
1021
Figure 1. Receiver operating characteristic curve. Specificity of individual step 1 exam scores as a predictor of marginal performance on the National Board of Medical Examiners subject examination is shown on the x axis. Sensitivity of individual step 1 exam scores as a predictor of marginal performance on the National Board subject examination is shown on the y axis.
schools were in the clinical subject areas.1 An advantage of using the National Board of Medical Examiners examination score is that these examinations provide highly reliable measures.2 The subject examination scores also have been valid predictors of performance on the National Board of Medical Examiners part II examination. The correlation coefficient for the subject exam in obstetrics-gynecology and the obstetricsgynecology section of National Board examination was 0.562, which is statistically significant (P , .01).3 In this investigation we found that the US Medical Licensure Examination score and grade point average were significantly associated with the National Board of
Medical Examiners subject examination score. The mean US Medical Licensure Examination score of the students scoring below the 20th percentile was also significantly lower than that of the students scoring at or above the 20th percentile. To identify the students who received a National Board of Medical Examiners subject examination score below the 20th percentile, however, a large number of students with scores above the 20th percentile were identified. We have examined ways in which to use the information gathered in this investigation. The practice at our university was to offer intensive tutorial assistance for students after failure of the National Board of Medical Examiners subject examination. After reviewing the data obtained in this investigation, we elected to organize a self-study tutorial program. Using this approach, we will not commit resources to a laborintensive process that in many cases may not be necessary. Prior studies in the literature have indicated that students’ self-assessments tend to be relatively accurate.4 Therefore, we have students self-identify themselves by requesting the study materials. In conclusion, we found a correlation between performance on the step 1 exam, grade point averages, and the National Board of Medical Examiners subject examination. There are, however, some limitations to the use of the US Medical Licensure Examination and grade point average as the sole predictors of performance on the end-of-clerkship subject examination score.
References 1. Hoffman KI. The USMLE: The NBME subject examinations, and assessment of individual academic achievement. Acad Med 1993; 68:740 –7. 2. Kappelman MM. The impact of external examinations on medical education program and students. J Med Educ 1983;58:300 – 8. 3. Spellacy WN, Dockery JL. A comparison of medical student performance on the obstetrics and gynecology National Board Part II examination and a comparable examination given during the clerkship. J Reprod Med 1980;24:76 – 8. 4. Herbert WNP, McGaghie WC, Drogmueller W, Riddle MH, Maxwell KL. Student evaluation in obstetrics and gynecology selfversus departmental assessment. Obstet Gynecol 1990;76:458 – 61.
Address reprint requests to:
Alicia Y. Armstrong, MD Department of Obstetrics and Gynecology Walter Reed Army Medical Center Washington, DC 20307
Figure 2. Correlation among individual US Medical Licensure Examination step 1 scores, grade point averages, and the National Board of Medical Examiners subject examination.
1022 Armstrong et al
Predictors of Test Performance
Received October 23, 1997. Received in revised form February 2, 1998. Accepted February 19, 1998. Published by Elsevier Science Inc.
Obstetrics & Gynecology